wireprototypes: clarify documentation of getbundle argument types It seems like it was a mix of what the Python code would see and what was sent over the wire. I've tried to clarify both the type seen in Python and how it's transmitted. Differential Revision: https://phab.mercurial-scm.org/D6871

merge with stable

py3: don't double-convert "opts" to bytes The "opts" are already converted to bytes at the beginning of the function. Doing it twice results in a crash, which makes test-uncommit.t fail. The extra call was added recently, in ff1ff2aae132 (uncommit: add support to modify the commit message and date, 2019-09-07). test-uncommit.t passes again after this patch. Differential Revision: https://phab.mercurial-scm.org/D6864

py3: byte-prefix sanitisation regexes in phabricator.py So it doesn't die with "TypeError: cannot use a string pattern on a bytes-like object". Differential Revision: https://phab.mercurial-scm.org/D6863

py3: pass a bytestring into querydrev instead of a string that'll TypeError This was a regression I introduced in c19d259fd6ad. When the string gets to the memoryview in _tokenize under py3 it'll die. Differential Revision: https://phab.mercurial-scm.org/D6869

py3: add test demonstrating TypeError when phabsending skips unchanged commits Skipping can currently only happen with `--no-amend`, so this isn't a usual configuration. Differential Revision: https://phab.mercurial-scm.org/D6868

tests: clean up built binaries after running test-fuzz-targets.t Most users won't notice a change here because they won't have the fuzzer infra, but by good fortune my workstation has the required bits and keeps leaving the fuzzer binaries around. Differential Revision: https://phab.mercurial-scm.org/D6862

fastannotate: remove support for flock() locking We've seen enough weirdness in CI with flock for remotefilelog that I'm now of the opinion we should just stop using flock() everywhere until someone has a concrete need for the extra performance *and* a way to only use it when safe (even if that's just default-to-off.) Differential Revision: https://phab.mercurial-scm.org/D6861

remotefilelog: remove dead code for using flock() for locking Differential Revision: https://phab.mercurial-scm.org/D6860

narrow: add option for automatically removing unused includes It's been a somewhat common request among our users to have Mercurial automatically pick includes to remove. This patch adds an option for that: `hg tracked --auto-remove-includes`. I'm not sure if this is the right name and semantics for it. Perhaps the feature should also add excludes of large subdirectories even if other files in the include are needed? Narrow clones are experimental, so we can change the name and/or semantics later if necessary. Differential Revision: https://phab.mercurial-scm.org/D6848

narrow: don't hexify paths and double-hexify known nodes on wire (BC) It isn't obvious, but wireprototypes.encodelist() is meant only for binary nodeids. So when we used it for encoding hex nodeids and paths, the encoded result was surprising and hard to read. This patch changes the encoding to make the list of paths a comma-separated list and the list of common nodes to be a encodelist()-encoded list of binary nodeids (so the result is just singly-hexified nodeids). This is clearly a breaking change, but the feature is experimental and we're not aware of anyone running a server using this command yet. Differential Revision: https://phab.mercurial-scm.org/D6851

remotefilelog: replace repack lock to solve race condition 2c74337e6483 reduced the probability of race-conditions when starting background repack and prefetch and we saw the difference in our CI instance with all failures disappearing except one where one call to waitonrepack seems to returns too early. I'm not sure what exactly goes wrong but I realized that while the prefetch operation uses a standard Mercurial lock, the repack operation is using a custom lock based on `fcntl.flock` on available platforms. As `extutil.flock` fallback on traditional Mercurial locks on other platforms and the tests are stable on my laptop, our CI environment and GCC112, I'm sending this patch to standardize the behavior across environments. Differential Revision: https://phab.mercurial-scm.org/D6844

perf: add a --stats argument to perfhelper-pathcopies The arguments will display some statisting about the distribution of the value we measure.

perf: add a --stats argument to perfhelper-mergecopies The arguments will display some statistics about the distribution of the value we measure.

archive: add XZ support if built with Python 3

rust-cpython: add sanity check to PySharedState::decrease_leak_count() If decrease_leak_count() were called unnecessarily, there must be a serious bug. It's better to not silently ignore such cases.

tests: stabilize test-fix.t on Windows `pwd` prints /tmp/... style paths, not C:\... needed for $TESTTMP to be substituted. In the final test, for whatever reason, Windows was missing EOL in the files and printing: [wdir] changedlines: printf: warning: ignoring excess arguments, starting with 'printf' even though it was trying to run: printf "Line ranges:\n"; printf "2 through 2\n"; I tried wrapping both :command and :linerange in `sh -c "..."`, and while that fixed the missing EOL, it missed the "2 through 2" output. Differential Revision: https://phab.mercurial-scm.org/D6852

zstandard: vendor python-zstandard 0.12 The upstream source distribution from PyPI was extracted. Unwanted files were removed. The clang-format ignore list was updated to reflect the new source of files. test-repo-compengines.t was updated to reflect a change in behavior of the zstd library. The project contains a vendored copy of zstandard 1.4.3. The old version was 1.3.8. This should result in some minor performance wins. # no-check-commit because 3rd party code has different style guidelines Differential Revision: https://phab.mercurial-scm.org/D6858

uncommit: enable support for adding a note This comes from the evolve extension's version of uncommit. The logic was already in place, and appears to be the last of the trivial things that can be enabled. Should these note options (including on amend) be marked advanced to keep the help text clutter level down? Differential Revision: https://phab.mercurial-scm.org/D6857

amend: enable support for using the secret phase This comes from the evolve extension's version of amend. The logic was already in place, and appears to be the last of the trivial things that can be enabled. Differential Revision: https://phab.mercurial-scm.org/D6856

amend: enable support for closing the branch This comes from the evolve extension's version of amend. The logic was already in place. Differential Revision: https://phab.mercurial-scm.org/D6855

amend: prevent '\n' in the note string This comes from the evolve function. I'm not sure why this check was missing in core, since it was present when the length check was added to evolve. I didn't flag this as BC because 530b7361e3a9 mentioned this argument wasn't added to the release notes due to no display capability, and that hasn't changed AFAIK. Differential Revision: https://phab.mercurial-scm.org/D6854

amend: add option to update to the current user This is also from the evolve extension's version of amend. A side effect of this refactoring is for uncommit to support `rewrite.update-timestamp`. Differential Revision: https://phab.mercurial-scm.org/D6853

bundle2: fix an off-by-one in debug message of number of parts Differential Revision: https://phab.mercurial-scm.org/D6850

tests: move a config write to top of file since it applies to all tests I'm about to add another test that depends on this config. Differential Revision: https://phab.mercurial-scm.org/D6849

idirstate: group private methods and attrs that are in the interface This makes it a little more obvious at a glance what work is left. Fortunately there's not a whole lot left. I suspect the ignore logic is going to be the tricky bit. Differential Revision: https://phab.mercurial-scm.org/D6839

idirstate: remove now non-public _map attribute Differential Revision: https://phab.mercurial-scm.org/D6838

interfaces: introduce an interface for dirstate implementations As usual with adding interface definitions, this describes the way things are, not the way we'd like things to be. There are some clear problems in the interface right now (eg ._map leaks in a few places), but I have plans to clean those up. There are also many missing docstrings, but again, we'll make a second pass to clean that up. Differential Revision: https://phab.mercurial-scm.org/D6836

cleanup: fix leakage of dirstate._map to client code We already had proper accessors for most of the behavior of dirstate._map that callers cared about exposed in the actual dirstate class as public methods. Sigh. There are two remaining privacy violations in the codebase after this change: 1) In the perf extension, which I suspect has to stick around because it's really testing the dirstate implementation directly 2) In largefiles, where we deal with standins and mutating status. Looking at this, I _strongly_ suspect a formal dirstate interface would allow this to work via composition instead of inheritance and monkeypatching. Fortunately, such wins are a part of my motivation for this work. I anticipate we'll come back to this in due time. Differential Revision: https://phab.mercurial-scm.org/D6837

exchange: convert bookmark nodes from hex to bin ASAP Differential Revision: https://phab.mercurial-scm.org/D6831

exchange: avoid unnecessary conversion of bookmark nodes to hex (API) Differential Revision: https://phab.mercurial-scm.org/D6830

highlight: fix encoding issues to enable Py3 compatibility This commit fixes various encoding issues with the `highlight` extension to enable compatibility with Python 3. Python `.encode()` and `.decode()` requires the target encoding to be passed as a `str`, so the value of `mercurial.encoding.encoding` must be converted before passing to the function. Pygments also assumes the `str` type for values it works with, so we must perform conversions before and after receiving values from its APIs. After applying this patch, `test-highlight.t` passes under Python 3. We add it to `python3-whitelist` as well. Tested with Pygments 2.4.2. Differential Revision: https://phab.mercurial-scm.org/D6832

hgweb: add a `message` attribute to `hgweb.common.ErrorResponse` This fixes a Python 3 bug where hgweb assumes an Exception subclass will have a `.message` attribute after running `Exception.__init__`.[1] The Python 3 way to get this info would be `e.args[0]`, but adding a new named attribute is more ergonomic in my view. [1] https://www.mercurial-scm.org/repo/hg-committed/file/6ccf539aec71/mercurial/hgweb/hgwebdir_mod.py#l459 Differential Revision: https://phab.mercurial-scm.org/D6840

uncommit: make -D/--date and -U/--user mutually exclusive This is how amend and graft work (but not MQ). I'm not sure why this didn't work for me when I first tried it. Differential Revision: https://phab.mercurial-scm.org/D6842

uncommit: drop the hyphen from --current-user and --current-date I didn't pay enough attention to these long forms- graft, amend and MQ already use the old style naming. It's probably more important to be consistent than modern. The hypenated style came from evolve. Yuya mentioned this naming discrepancy in 4145fd3569c3, but it didn't attract any discussion[1]. There's also a bit of inconsistency in that the default parameter for `currentdate` is `False` for graft, and `None` for the rest. [1] https://www.mercurial-scm.org/pipermail/mercurial-devel/2019-January/126767.html Differential Revision: https://phab.mercurial-scm.org/D6841

hgweb: fix websub regex flag syntax on Python 3 The `websub` config section for hgweb is broken under Python 3 when using regex flags syntax (ie the optional `i` in the example from `hg help config.websub`: patternname = s/SEARCH_REGEX/REPLACE_EXPRESSION/[i] Flags are pulled out of the specified byte-string using a regular expression, and uppercased. The flags are then iterated over and passed to the `re` module using `re.__dict__[item]`, to get the object attribute of the same name from the `re` module. So on Python 2 if the `il` flags are passed, this transition looks like: `'il'` -> `'IL'` -> `'I'` -> `re.__dict__['I']` -> `re.I` However on Python 3, these are bytes objects. When we iterate over a bytes object in Python 3, instead of getting the individual characters in the string as string objects of length one, we get the integer \ value corresponding to that byte. So the same transition looks like: `b'il'` -> `b'IL'` -> `73` -> `re.__dict__[73]` -> `KeyError` This commit fixes the type mismatch by converting the bytes to a system string before iterating over each element to pass to `re`. The transition will now look like: `b'il'` -> `u'IL'` -> `u'I'` -> `re.__dict__[u'I']` -> `re.I` In addition we expand `test-websub.t` to cover the regex flag case (for both the `websub` section and `interhg`). Differential Revision: https://phab.mercurial-scm.org/D6788

merge with stable

relnotes: we now require `sh` to support $(command) syntax to run test suite For example, Solaris before version 11 had /bin/sh pointing to the old Bourne Shell (which doesn't support $(command) syntax). Differential Revision: https://phab.mercurial-scm.org/D6833

doc: fix up confusing doc comment Differential Revision: https://phab.mercurial-scm.org/D6829

strip: fix bug with treemanifests and unordered linkrevs This is the treemanifest version of f45f7390c1c5 (strip: calculate list of extra nodes to save and pass it to changegroupsubset, 2008-01-19). Differential Revision: https://phab.mercurial-scm.org/D6795

repair: extract a helper for generating all manifest revlogs We'll need to walk the manifest revlogs also to figure out which manifests have linkrevs out of order (for fixing the bug shown in the previous patch). By the way, perhaps it would be more efficient in many cases to find only the relevant directory manifest revlogs based on the files instead of walking the entire store, but that can be changed later. Differential Revision: https://phab.mercurial-scm.org/D6794

tests: show broken strip with treemanifests and unordered linkrevs This is the treemanifest version of issue764. Differential Revision: https://phab.mercurial-scm.org/D6793

tests: split out manifest case from test-strip-cross.t The manifest case was added on after the other cases, in d67cfe0d4714 (test-strip-cross: test handling of linkrev crosses in the manifest, 2008-01-20). I think it's easier to read and modify if it's separated. Differential Revision: https://phab.mercurial-scm.org/D6792

tests: don't log manifest-file in test-strip-cross.t I'm confident that the file is there only to help produce a certain manifest log; there is nothing special about the file's filelog itself. (And there already is a `hg debugindex --manifest` call higher up in the file that shows the crossed linkrevs.) Differential Revision: https://phab.mercurial-scm.org/D6791

tests: use positive revision numbers in test-strip-cross.t It took me a long time to realize that '-r -1' was pulling revision -1 because I didn't even notice the minus sign until I tried changing the revision number. The order of arguments doesn't matter, so I changed from decreasing order to increasing while at it. Differential Revision: https://phab.mercurial-scm.org/D6790

automation: implement "publish-windows-artifacts" command The new command and associated functionality can be used to automate the publishing of Windows release artifacts. It supports uploading wheels to PyPI (using twine) and copying the artifacts to mercurial-scm.org and updating the latest.dat file to advertise them via the website. I ran `automation.py publish-windows-artifacts 5.1.1` and it appeared to "just work." But the real test will be to do this on the next release... Differential Revision: https://phab.mercurial-scm.org/D6786

automation: upgrade to latest packages in requirements.txt Let's stay modern. Differential Revision: https://phab.mercurial-scm.org/D6785

localrepo: push manifestlog and changelog construction code into store This feels substantially more appropriate, as the store is actually the layer with knowledge of how to handle this storage. I didn't move the caching decorators for now because that's going to require some more involved work, and this unblocks my current experimentation. Differential Revision: https://phab.mercurial-scm.org/D6732

notify: add option for deterministic message-id generation Copied from email by durin42: > Pierre-Yves asked offline why I asked a new option for this and why it > is not the default. Message-Id is supposed to be unique world-wide and > while it is desirable to have a deterministic mechanism for them for > creating follow-up emails, it needs organisational control for ensuring > the uniqueness. Differential Revision: https://phab.mercurial-scm.org/D6824

uncommit: add options to update to the current user or current date These are also from the evolve extension's version of uncommit. I tried adding validation that both forms of user or date can't be specified at the same time, but that fails because these show up in `opts` with a None value whether or not the option was given on the command line. Presumably that means the conditional in `resolvecommitoptions` could be simplified. But this is how both evolve and MQ handle it. Differential Revision: https://phab.mercurial-scm.org/D6828

uncommit: add support to modify the commit message and date Currently, the evolve extension's version of this command supports it. Differential Revision: https://phab.mercurial-scm.org/D6827

run-tests: add a dedicated 'isoptional' function This is clearer than repeated manual call to to 'endswith'. (This is a gratuitous cleanup that I made while investigating a bug).

run-tests: remove the artificial indentation

run-tests: extract a `process_out_line` from the main function The main function doing line comparison is quite complex. Slicing it in smaller piece should clarify it. To avoid a huge diff, the code is kept at the same indentation. We'll re-indent in the next changesets. (This is a gratuitous cleanup that I made while investigating a bug).

run-tests: extract a `process_cmd_line` from the main function The main function doing line comparison is quite complex. Slicing it in smaller piece should clarify it. (This is a gratuitous cleanup that I made while investigating a bug).

changegroup: move message about added changes to transaction summary Before that, applying multiple changegroups in the same transaction issued the message multiple time. This result in a confusing output: adding changesets adding manifests adding file changes added 32768 changesets with 60829 changes to 2668 files adding changesets adding manifests adding file changes added 8192 changesets with 16885 changes to 1553 files adding changesets adding manifests adding file changes added 1020 changesets with 1799 changes to 536 files adding changesets adding manifests ... Instead, we now only issue the message once at the end of the transaction, summing up all added changesets, changes and files. The line is identical, but happens sightly later in the output. There are other suboptimal behavior around issue multiple changegroup (eg: progress bar). We'll cover them later. This impact of lot of test as one would expect, but a two pass check show they are just the order change we expected. To deal with "under the hood" bundle application by internal code, we had to take a slightly hacky move. We could clean that up with a more official way to enter "under the hood" section, however I want to keep this series simple to get it landed. This kind of change have a very high bit rot rate since it impact a lot of test output.

sshserver: flush stream after command dispatch I am not sure why this is not working as expected, but without this client might not see some important output. Without this patch moving some output at transaction closing time makes it disapear for ssh client in various sitaution.

narrow: rely on setting `quiet` mode instead of `pushbuffer` The `quiet` approach is what `shelve` uses and give the same result. This will help us to add less code in future changesets.

transaction: issue "new obsmarkers" message at the end of the transaction Instead of making bundle2 code responsible for this, it seems better to have it handled and the transaction level. First, it means the message will be more consistently printed. Second it means we won't spam the message over and over if the data arrive in multiple piece. Third, we are planning to move other similar message at the same level (for the same reason) so having them all at the same location will help us to control the order they are displayed.

debugobsolete: also issue the "new obsmarkers" messsage We are going to improve the way this message is issued in the core codebase. This will make it appears for `hg debugobsolete` too. Since this seems like a good idea, we make the output change in a previous changesets to clarify the next changeset.

split: use literal syntax to build a set of one element

rust-cpython: leverage py_shared_iterator::from_inner() where appropriate

rust-cpython: remove Option<_> from interface of py_shared_iterator It's the implementation detail of the py_shared_iterator that the leaked reference is kept in Option<_> so that it can be dropped early.

rust-cpython: rename py_shared_iterator_impl to py_shared_iterator It's a public interface now.

rust-cpython: replace dyn Iterator<..> of mapping with concrete type See the previous commit for why. The docstring is moved accordingly.

rust-cpython: replace dyn Iterator<..> of sequence with concrete type We wouldn't care the cost of the dynamic dispatch, but I feel a concrete type helps understanding error messages.

rust-dirstate: provide CopyMapIter and StateMapIter types They will be used in the declaration of Python iterator types.

rust-dirstate: specify concrete return type of DirsMultiset::iter() This allows us to put a returned iterator in a struct. We could implement DirsMultisetIter(hash_map::Keys<..>) struct to hide the implementation detail, but I think type alias is good enough for us.

discovery: replace "heads" by "changesets" in a output note (BC) When using `hg push --rev X`, the subset considered by discovery is only `::X`. In addition, `X` can be any local revisions not just local heads. As a result the message "all local heads known locally" can be misleading. We replace it with the more accurate "all local changesets known remotely". The message appears when in verbose not, so this is stricly speaking a BC breakage. I am not sure this would be a real issue in practice...

py3: drop incorrect fsencode(findexe(...)) in procutil I recently added the bad call, thinking that findexe() was a standard library function returning a string result, but it's actually our own function returning bytes. Thanks to Yuya for noticing. Differential Revision: https://phab.mercurial-scm.org/D6826

flagprocessors: small code update to clarify parameters 'raw' is really a third mode, not a small variant. Differential Revision: https://phab.mercurial-scm.org/D6807

flagprocessors: deprecate _processflags People should use the specialized version instead now. Differential Revision: https://phab.mercurial-scm.org/D6806

simplestorerepo: stop using `_processflags` directly We now use the specialized versions. Differential Revision: https://phab.mercurial-scm.org/D6805

revlog: stop using `_processflags` directly We now use the specialized versions. Differential Revision: https://phab.mercurial-scm.org/D6804

flagprocessors: use _processflagsraw in easy cases When there are no ambiguity regarding the `raw` value, we can simply use the new method. Differential Revision: https://phab.mercurial-scm.org/D6803

flagprocessors: use _processflagsread in simple cases When there are no ambiguity regarding the `raw` value, we can simply use the new method. Differential Revision: https://phab.mercurial-scm.org/D6802

flagprocessors: use _processflagswrite for write operation There are no ambiguity for 'write' operation so it is simple to replace. Differential Revision: https://phab.mercurial-scm.org/D6801

flagprocessors: introduce specialized functions This make the call site clearer and the open the way to more diverse return types. For now, the same old code is still in use under the hood. Differential Revision: https://phab.mercurial-scm.org/D6800

flagutil: use it in simplestorerepo This remove the other code duplication of the `_processflags` code. To be honest, this code looks very dead since it is not run by either developer nor buildbot. However, we update the code to make the task of reviving this less daunting. Differential Revision: https://phab.mercurial-scm.org/D6799

flagutil: make the error class used by the mixin configurable One of the code duplication use a different error class. So let's make it possible to do so. Differential Revision: https://phab.mercurial-scm.org/D6798

flagutil: use the new mixin use in remotefilelog This remove one of the code duplication. Hooray \o/. Differential Revision: https://phab.mercurial-scm.org/D6797

flagutil: introduce a flagprocessorsmixin class To avoid code duplication, we will provide a simple "ready to use" mixin that carry the appropriate logic. First we use it in standard revlog, we'll remove code duplication in later changesets. Differential Revision: https://phab.mercurial-scm.org/D6796

check-code: allow command substitution with $(command) Both `command` and $(command) are specified by POSIX. The latter nests better. I don't see why we shouldn't allow both (or only the latter). Differential Revision: https://phab.mercurial-scm.org/D6789

run-tests: rename `lcmd` variable to `line_cmd` This is clearer and more in line with some other variable names. (This is a gratuitous cleanup that I made while investigating a bug).

run-tests: rename `lout` variable to `out_line` This is clearer and more in line with some other variable names. (This is a gratuitous cleanup that I made while investigating a bug).

run-tests: clarify "l" variable as "out_rawline" More explicit variable name never hurt. (This is a gratuitous cleanup that I made while investigating a bug).

run-tests: use symbolic constant instead of arbitrary number line matching (This is a gratuitous cleanup that I made while investigating a bug).

rustfilepatterns: shorter code for concatenating slices Differential Revision: https://phab.mercurial-scm.org/D6765

match: simplify the regexps created for glob patterns For legibility of the resulting regexes, although it may help with performance as well. Differential Revision: https://phab.mercurial-scm.org/D6764

rustfilepatterns: refactor the pattern of removing a prefix from a &[u8] Differential Revision: https://phab.mercurial-scm.org/D6766

tests: show the pattern generated for a relative glob Differential Revision: https://phab.mercurial-scm.org/D6763

copies: remove existing copy info from the changeset on amend (BC) When amending a changeset with copy information in the changeset and the new changeset doesn't have any copy information (or similar for "filesadded" and "filesremoved"), we shouldn't keep it. A drawback of this is that we now unconditionally remove these four entries from the extras, breaking any extensions that happened to write entries with the same names (which seems very unlikely). I think I'd heard that there was list of blacklisted keys that would be removed from the extras when a commit is rewritten, but I couldn't find that. It would make sense to add the keys mentioned above there instead of the custom filtering I've added in this patch. Differential Revision: https://phab.mercurial-scm.org/D6752

tests: show invalid copies when turning off copies-in-changeset If you turn on copies in changesets and write a commit with a copy, then turn it off and amend the commit while undoing the copy, the invalid copy information will remain. The read path doesn't crash in invalid copy data, but it seems better to not produce the invalid data. Differential Revision: https://phab.mercurial-scm.org/D6751

context: filter out invalid copies from workingctx.p[12]copies() workingctx normally gets its lists of modified, added, removed files etc. based on the dirstate status. Its constructor also accepts a "changes" argument to override the status from the dirstate. This is used for partial commits. If a "changed" argument was passed, we should clearly also filter out copies to those paths, which I had previously missed. This patch adds that filtering and fixes the bugs demonstrated in the previous patch. Differential Revision: https://phab.mercurial-scm.org/D6750

tests: demonstrate crash when committing subset of copies to changeset When writing copy metadata to the changeset and not committing all copies in the dirstate, we get a ProgrammingError. This commit adds two tests showing how to trigger this bug. Differential Revision: https://phab.mercurial-scm.org/D6749

bdiff-torture: fix pyflakes warning reporting undefined name 'inst' Looks like I got a latest version of pyflakes somehow or it's running on py3 and it spotted this. Differential Revision: https://phab.mercurial-scm.org/D6757

split: handle partial commit of renames when doing split or record (issue5723) When using split or record, using either interface (text or curses), selecting portions of the file to be committed/recorded did not work; the entire file was treated as having been selected. This was because the logic for handling partial application of the patches relies on knowing what files are "new with modifications" and it doesn't treat "rename destination" as "new". There was a complicating issue, however. We're relying on the patch header specifying the copy from/to information, which works as long as the 'copy from' file is there. In the case of renames, however, the 'rename from' file is *not* there, so we need to add it back. Differential Revision: https://phab.mercurial-scm.org/D6768

split: handle partial commit of copies when doing split or record When using split or record, using either interface (text or curses), selecting portions of the file to be committed/recorded did not work; the entire file was treated as having been selected. This appears to be because the logic for handling partial application of the patches relies on knowing what files are "new with modifications", and it doesn't treat "copy destination" as "new". Handling renames correctly is more difficult and will be done in a later patch. Differential Revision: https://phab.mercurial-scm.org/D6767

py3: use pycompat.sysargv[0] for instead of fsencode(sys.argv[0]) Yuya noted in a recent review that fsencode(sys.argv[0]) could be incorrect on Windows. Differential Revision: https://phab.mercurial-scm.org/D6782

httppeer: use context manager when reading temporary bundle to send Differential Revision: https://phab.mercurial-scm.org/D6784

httppeer: use context manager when writing temporary bundle to send Differential Revision: https://phab.mercurial-scm.org/D6783

rust-cpython: mark unsafe functions as such It wasn't trivial to fix leak_immutable() to be safe since we have to allow immutable operations (e.g. iter()) on the leaked reference. So let's mark it unsafe for now. Callers must take care of the returned object to guarantee the memory safety. I'll revisit this later. I think $leaked<T: 'static> could have a function that converts itself into $leaked<U: 'static> with a given FnOnce(&T) -> &U, where T is $inner_struct, and U is $iterator_type for example.

rust-cpython: pair leaked reference with its manager object Still leak_immutable() is unsafe since leak_handle must live longer than the leaked_ref.

rust-cpython: introduce restricted variant of RefCell This should catch invalid borrow_mut() calls. Still the ref-sharing interface is unsafe.

rust-cpython: fix unsafe inner(py).borrow_mut() calls Since self.inner is managed by PySharedState, it must not be borrowed mutably through the RefCell interface. Otherwise, the underlying object could be mutated while a reference is leaked to Python world.

revlog: deprecate the use of `revision(..., raw=True)` We have an official `rawdata` function now.

remotefilelog: reduce probability of race-condition in remotefilelog tests ca1014ad3de4 introduced a new parameter `ensurestart` to speed up remotefilelog background processes start. Unfortunately it seems to have increased the possibility of race-conditions in remotefilelog tests testing those background processes. With `ensurestart=False`, it seems that it's more probable to enter in a race condition with `debugwaitonprefetch` and `debugwaitonrepack` in remotefilelog background tests. Our CI seems to have a high probability of triggering this race condition so make it configurable to ensure tests are stable. Differential Revision: https://phab.mercurial-scm.org/D6772

rust: apply more formatting fixes My cargo fmt updated these lines and they look good.

rust-utils: add normalize_case util to mirror Python one While we still don't handle filenames properly cross-platform, this at least sticks closer to the Python behavior. Differential Revision: https://phab.mercurial-scm.org/D6756

rust: fix warnings about trait objects without dyn being deprecated Differential Revision: https://phab.mercurial-scm.org/D6770

py3: convert hg executable path to bytes in missing case in procutil We (Google) noticed this in our tests when we use chg and a hg wrapper script not called 'hg'. The executable then ended up being a native string, which then failed in chgserver when trying to convert the environment dict to a byte string. Differential Revision: https://phab.mercurial-scm.org/D6775

py3: make statprof's chrome output work With this patch, this command works: python3 hg --profile --config profiling.statformat=chrome st (and it works with s/python3/python2/ as well) Differential Revision: https://phab.mercurial-scm.org/D6777

py3: for statprof's Chrome output, write json to string, then encode to bytes `json.dump(obj, fp)` requires `fp.write()` to accept str output, and since the file pointer we have there only accepts bytes, we need to change to json.dumps() and then encode as utf-8. We have already done the same thing for the json (non-Chrome) format in 4b7eb862692e (py3: encode json output to bytes and use write(), 2018-10-12). Differential Revision: https://phab.mercurial-scm.org/D6781

statprof: use context manager for file when writing flame graph Differential Revision: https://phab.mercurial-scm.org/D6780

statprof: use context manager when reading source from file Differential Revision: https://phab.mercurial-scm.org/D6779

statprof: clarify by naming tuple members while enumerate()'ing Differential Revision: https://phab.mercurial-scm.org/D6778

upgrade: make sure we reclone all revlogs when updating to some format Adding or removing some requirement (eg: sparserevlog), requires to reclone revlog to use the new format. We cannot simply copy the original files. In this case, we issue a warning to proceed with clone every revlogs.

upgrade: add an argument to control changelog upgrade Same as for `--manifest` we can now select more selection.

upgrade: add an argument to control manifest upgrade The argument can be used to only "clone" manifest revlog or clone all of them but this one. The selection will make more sense once we have a `--changelog` flag in the next changesets.

unionrepo: drop the custom `rawdata` implementation We can rely on the main one now.

unionrepo: drop `baserevdiff` It has no caller anymore.

unionrepo: use normal inheritance scheme to call revdiff

unionrepo: fix `revdiff` implementation to use `rawdata` The parent code is using rawdata so we should use it here. Before this change, union repo was probably broken with some flag processors.

unionrepo: get rid of `baserevision` The method is not called anywhere anymore, so we can safely drop it. Some of the comment get moved to `baserevdiff` because we did not got rid of it (yet).

unionrepo: use a lower level overide in unionrepo too The unionrepo class also have a strange `baserevision` hack, let's try to get ride of it too.

bundlerepo: drop the custom `rawdata` implementation We can rely on the main one now.

bundlerepo: drop the `baserevision` hack It is not used anywhere anymore, so we can safely drop it.

bundlerepo: simplify code to take advantage of `_rawtext` In the revlog code, the code getting the raw text is now isolated. We take advantage of this to simplify the bundlerepo code.

merge with stable

rust: run cargo fmt

py3: use pycompat.maplist() in chgserver test-chg.t almost passes on py3 after this patch. Differential Revision: https://phab.mercurial-scm.org/D6771

run-tests: handle --local before --with-hg We no longer support them both together, so this is now safe to do. By checking --local first, we avoid error out about an invalid --with-hg script if --local was also given (we instead tell the user that the options are mutually exclusive). I also had to wrap the 'binpath' we pass to setattr() in _strpath() to keep `python3 run-tests.py -l` working. That change also made `python3 run-tests.py -l --chg` work, which was the reason for this series. Differential Revision: https://phab.mercurial-scm.org/D6760

run-tests: error out on `--local --with-[c]hg` I don't see much reason to allow these combinations. You could use --local and override only one of --with-hg or --with-chg, but I don't see much practical use for that. It would be easy to work around anyway by passing both --with-hg and --with-chg. By erroring out, it makes the code a bit easier to reason about to allow the next few patches. Differential Revision: https://phab.mercurial-scm.org/D6759

contrib: simplify the genosxversion.py command to find the hg libraries I forget what problem I ran into while trying to teach the makefile to use a non-system python. (It might have ben missing hg-evolve and/or keyring, but `check_output()` was raising an error.) This still isn't great because it will return non zero for something like the username not being set, even though we aren't asking for it. But I suppose it's still useful to simplify. Differential Revision: https://phab.mercurial-scm.org/D6753

interfaceutil: move to interfaces/ Now that we have a dedicated folder for interfaces, let's move interfaceutil there. Differential Revision: https://phab.mercurial-scm.org/D6742

interfaces: create a new folder for interfaces and move repository.py in it I was trying to understand current interfaces and write new ones and I realized we need to improve how current interfaces are organised. This creates a dedicated folder for defining interfaces and move `repository.py` which defines all the current interfaces inside it. Differential Revision: https://phab.mercurial-scm.org/D6741

narrow: fix typo "respositories" Differential Revision: https://phab.mercurial-scm.org/D6758

merge with stable

merge: hint about using `hg resolve` for resolving conflicts This was suggested by one of our users at Google. Makes sense to me. Differential Revision: https://phab.mercurial-scm.org/D6755

rust-dirstate: remove test case for DirsMultiset::new(Manifest, Some) It's no longer possible.

rust-dirstate: split DirsMultiset constructor per input type Since skip_state only applies to dirstate, it doesn't make sense to unify these constructors and dispatch by enum.

rust-dirstate: remove excessive clone() of parameter and return value I think pass-by-ref is preferred in general.

rust-dirstate: handle invalid length of p1/p2 parameters It uses match syntax since map_err() failed to deduce the argument type.

rust: simply use TryInto to convert slice to array Since our rust module depends on TryInto, there's no point to avoid using it. While rewriting copy_into_array(), I noticed CPython interface doesn't check the length of the p1/p2 values, which is marked as TODO.

rust-dirstate: use PARENT_SIZE constant where appropriate

rust-dirstate: rename NULL_REVISION to NULL_ID which isn't a revision number

rust-dirstate: remove repetition in array literal

rust-dirstate: remove too abstracted way of getting &[u8]

rust-dirstate: remove unneeded "ref" At 7cae6bc29ff9, .to_owned() was rewritten as .to_owned().to_vec(), which is no longer needed since the filename is a single reference.

rust-parsers: fix unboxing of PyInt on Python 3 Broken at 7cae6bc29ff9.

revlog: split `rawtext` retrieval out of _revisiondata This part is reasonably independent. Having it on its own clarify the code flow and will help code that inherit from revlog to overwrite specific area only.

revlog: avoid caching raw text too early in _revisiondata Without this change, we could cache the rawtext without considering for it validating the cache or not. If the exception raised by the invalid hash were to be caught and the same revision accessed again, the invalid rawtext would be returned.

revlog: add some documentation to `_revisiondata` code

revlog: move `nullid` early return sooner in `_revisiondata` Let us deal with the special case before we start dealing with more generic case.

revlog: stop calling `basetext` `rawtext` in _revisiondata If the cache entry is used as a base test for delta, it is not the rawtext we need. We update the variable name to clarify this.

revlog: assign rawtext earlier in `_revisiondata` Assigning the revision earlier make the code easier to read.

revlog: drop silly `raw` parameter to `rawdata` function This is a leftover from `revision` and does not make sense for `rawdata`

perf: don't depend on pycompat for older Mercurial versions We already define local _sysstr() and _xrange() for compatibility, so we should use those. Also create a similar local _bytestr() corresponding to pycompat.bytestr(). Differential Revision: https://phab.mercurial-scm.org/D6745

perf: don't try to call `util.queue` on Mercurial version before it existed Differential Revision: https://phab.mercurial-scm.org/D6744

perf: handle NameError for `pycompat.foo` when pycompat wasn't imported On old Mercurial versions, we won't have a pycompat variable defined, and then `pycompat.foo` will raise a NameError. Differential Revision: https://phab.mercurial-scm.org/D6743

rawdata: update callers in shallowbundle We update callers incrementally because this help bisecting failures. This was useful during development, so we expect it might be useful again in the future.

rawdata: update callers in storageutils We update callers incrementally because this help bisecting failures. This was useful during development, so we expect it might be useful again in the future.

rawdata: update callers in delta utils We update callers incrementally because this help bisecting failures. This was useful during development, so we expect it might be useful again in the future.

rawdata: update callers in repository We update callers incrementally because this help bisecting failures. This was useful during development, so we expect it might be useful again in the future.

rawdata: update callers in testing/storage.py We update callers incrementally because this help bisecting failures. This was useful during development, so we expect it might be useful again in the future.

rawdata: update callers in test-revlog-raw We update callers incrementally because this help bisecting failures. This was useful during development, so we expect it might be useful again in the future.

rawdata: update callers in lfs' tests We update callers incrementally because this help bisecting failures. This was useful during development, so we expect it might be useful again in the future.

rawdata: update callers in lfs' wrapper We update callers incrementally because this help bisecting failures. This was useful during development, so we expect it might be useful again in the future.

rawdata: update caller in wireprotov2server We update callers incrementally because this help bisecting failures. This was useful during development, so we expect it might be useful again in the future.

rawdata: update callers in debugcommands We update callers incrementally because this help bisecting failures. This was useful during development, so we expect it might be useful again in the future.

rawdata: update callers in sqlitestore We update callers incrementally because this help bisecting failures. This was useful during development, so we expect it might be useful again in the future.

rawdata: update caller in remotefilelog We update callers incrementally because this help bisecting failures. This was useful during development, so we expect it might be useful again in the future.

rawdata: update callers in bundlerepo We update callers incrementally because this help bisecting failures. This was useful during development, so we expect it might be useful again in the future.

rawdata: update callers in context We update callers incrementally because this help bisecting failures. This was useful during development, so we expect it might be useful again in the future.

rawdata: update caller in revlog We update callers incrementally because this help bisecting failures. This was useful during development, so we expect it might be useful again in the future.

setup: fix a sorting issue I noticed in package names Differential Revision: https://phab.mercurial-scm.org/D6733

py3: do not convert rust module/attribute names to bytes policy.import*() functions expect system strings.

transplant: unnest --stop case It should be aligned with --continue.

rust-discovery: use while loop instead of match + break This looks slightly nicer.

rust-discovery: remove useless extern crate

transplant: added support for --stop flag This adds fuctionality for `--stop` flag to `transplant`. A new method `stop` is added to `transplanter` class containing logic to abort transplant. Tests are updated as shown. Differential Revision: https://phab.mercurial-scm.org/D6695

unshelve: abort on using --keep and --interactive together I am working on making interactive mode support `--keep` flag. Until we support the usage of `--interactive` and `--keep` together, let us abort on it. Differential Revision: https://phab.mercurial-scm.org/D6699

config: add experimental argument to the config registrar Until now, there are almost 28 config items which are considered as `experimental` but, not present in the `experimental` section of the registrar. This patch adds an `experimental` argument to the config registrar to mark such config items. Differential Revision: https://phab.mercurial-scm.org/D6728 Differential Revision: https://phab.mercurial-scm.org/D6746

tests: split joint repo/changelog fake into one for each type I'm just not comfortable with fakes that get overloaded for multiple types: too often it gets out of hand and becomes difficult to trace. Differential Revision: https://phab.mercurial-scm.org/D6725

fix: pass line ranges as value instead of callback The callback no longer takes any arguments from the inner function, so we might as well call it sooner and pass the value instead. Note the value still needs to be recomputed every iteration to account for the previous iteration's changes to the file content. Differential Revision: https://phab.mercurial-scm.org/D6727

fix: correctly parse the :metadata subconfig It's being handled as a string instead of a bool, though the thruthiness of the string makes the feature still essentially work. Added a regression test. Differential Revision: https://phab.mercurial-scm.org/D6726

fix: allow tools to use :linerange, but also run if a file is unchanged The definition of "unchanged" here is subtle, because pure deletion diff hunks are ignored. That means this is different from using the --whole flag. This change allows you to configure, for example, a code formatter that: 1. Formats specific line ranges if specified via flags 2. Does not format the entire file when there are no line ranges provided 3. Performs some other kind of formatting regardless of provided line ranges This sounds a little far fetched, but it is meant to address a specific corner case encountered in Google's use of the fix extension. The default behavior is kept because it exists to prevent mistakes that could erase uncommitted changes. Differential Revision: https://phab.mercurial-scm.org/D6723

rust-dirstate: call rust dirstatemap from Python Since Rust-backed Python classes cannot be used as baseclasses (for rust-cpython anyway), we use composition rather than inheritance. This also allows us to keep the IO operations in the Python side, removing (for now) the need to rewrite VFS in Rust, which would be a heavy undertaking. Differential Revision: https://phab.mercurial-scm.org/D6634

rust-dirstate: rust-cpython bridge for dirstatemap This change also showcases the limitations of the `py_shared_ref!` macro. See the previous commit 'rust-dirstate: rust implementation of dirstatemap` for an explanation for the TODOs in the code. Differential Revision: https://phab.mercurial-scm.org/D6633

rust-dirstate: rust implementation of dirstatemap The `dirstatemap` is one of the last building blocks needed to get to a `dirstate.walk` Rust implementation. Disclaimer: This change is part of a big (10) series of patches, all of which started as one big changeset that took a long time to write. This `dirstatemap` implementation is a compromise in terms of complexity both for me and for the reviewers. I chose to submit this patch right now because while it is not perfect, it works and is simple enough (IMHO) to be reviewed. The Python implementation uses a lot of lazy propertycaches, breaks encapsulation and is used as an iterator in a lot of places, all of which dictated the somewhat unidiomatic patterns in this change. Like written in the comments, rewriting this struct to use the typestate pattern might be a good idea, but this is a good first step. Differential Revision: https://phab.mercurial-scm.org/D6632

rust-cpython: add macro for sharing references Following an experiment done by Georges Racinet, we now have a working way of sharing references between Python and Rust. This is needed in many points of the codebase, for example every time we need to expose an iterator to a Rust-backed Python class. In a few words, references are (unsafely) marked as `'static` and coupled with manual reference counting; we are doing manual borrow-checking. This changes introduces two declarative macro to help reduce boilerplate. While it is better than not using macros, they are not perfect. They need to: - Integrate with the garbage collector for container types (not needed as of yet), as stated in the docstring - Allow for leaking multiple attributes at the same time - Inject the `py_shared_state` data attribute in `py_class`-generated structs - Automatically namespace the functions and attributes they generate For at least the last two points, we will need to write a procedural macro instead of a declarative one. While this reference-sharing mechanism is being ironed out I thought it best not to implement it yet. Lastly, and implementation detail renders our Rust-backed Python iterators too strict to be proper drop-in replacements, as will be illustrated in a future patch: if the data structure referenced by a non-depleted iterator is mutated, an `AlreadyBorrowed` exception is raised, whereas Python would allow it, only to raise a `RuntimeError` if `next` is called on said iterator. This will have to be addressed at some point. Differential Revision: https://phab.mercurial-scm.org/D6631

rust-docstrings: add missing module docstrings Differential Revision: https://phab.mercurial-scm.org/D6630

rust-dirstate: improve API of `DirsMultiset` - Use opaque `Iterator` type instead of implementation-specific one from `HashMap` - Make `DirsMultiset` behave like a set both in Rust and from Python Differential Revision: https://phab.mercurial-scm.org/D6690

rust-dirstate: use EntryState enum instead of literals This improves code readability quite a bit, while also adding a layer of safety because we're checking the state byte against the enum. Differential Revision: https://phab.mercurial-scm.org/D6629

rust-parsers: switch to parse/pack_dirstate to mutate-on-loop Both `parse_dirstate` and `pack_dirstate` can operate directly on the data they're passed, which prevents the creation of intermediate data structures, simplifies the function signatures and reduces boilerplate. They are exposed directly to the Python for now, but a later patch will make use of them inside `hg-core`. Differential Revision: https://phab.mercurial-scm.org/D6628

rust-parsers: move parser bindings to their own file and Python module This tidies up the Rust side while simplifying the Python side. Differential Revision: https://phab.mercurial-scm.org/D6627

rust-dirstate: create dirstate submodule in hg-cpython This module will soon hold multiple files, this change is to make the review process easier. Differential Revision: https://phab.mercurial-scm.org/D6626

rust-discovery: using from Python code As previously done in other topics, the Rust version is used if it's been built. The version fully in Rust of the partialdiscovery class has the performance advantage over the Python version (actually using the Rust MissingAncestor) if the undecided set is big enough. Otherwise no sampling occurs, and the discovery is reasonably fast anyway. Note: it's hard to predict the size of the initial undecided set, it can depend on the kind of topological changes between the local and remote graphs. The point of the Rust version is to make the bad cases acceptable. More specifically, the performance advantages are: - faster sampling, especially takefullsample() - much faster addmissings() in almost all cases (see commit message in grandparent of the present changeset) - no conversion cost of the undecided set at the interface between Rust and Python == Measurements with big undecided sets For an extreme example, discovery between mozilla-try and mozilla-unified (over one million undecided revisions, same case as in dbd0fcca6dfc), we get roughly a x2.5/x3 better performance: Growing sample size (5% starting with 200): time goes down from 210 to 72 seconds. Constant sample size of 200: time down from 1853 to 659 seconds. With a sample size computed from number of roots and heads of the undecided set (`respectsize` is `False`), here are perfdiscovery results: Before ! wall 9.358729 comb 9.360000 user 9.310000 sys 0.050000 (median of 50) After ! wall 3.793819 comb 3.790000 user 3.750000 sys 0.040000 (median of 50) In that later case, the sample sizes are routinely in the hundreds of thousands of revisions. While still faster, the Rust iteration in addmissings has less of an advantage than with smaller sample sizes, but one sees addcommons becoming faster, probably a consequence of not having to copy big sets back and forth. This example is not a goal in itself, but it showcases several different areas in which the process can become slow, due to different factors, and how this full Rust version can help. == Measurements with small undecided sets In cases the undecided set is small enough than no sampling occurs, the Rust version has a disadvantage at init if `targetheads` is really big (some time is lost in the translation to Rust data structures), and that is compensated by the faster `addmissings()`. On a private repository with over one million commits, we still get a minor improvement, of 6.8%: Before ! wall 0.593585 comb 0.590000 user 0.550000 sys 0.040000 (median of 50) After ! wall 0.553035 comb 0.550000 user 0.520000 sys 0.030000 (median of 50) What's interesting in that case is the first addinfo() at 180ms for Rust and 233ms for Python+C, mostly due to add_missings and the children cache computation being done in less than 0.2ms on the Rust side vs over 40ms on the Python side. The worst case we have on hand is with mozilla-try, prepared with discovery-helper.sh for 10 heads and depth 10, time goes up 2.2% on the median. In this case `targetheads` is really huge with 165842 server heads. Before ! wall 0.823884 comb 0.810000 user 0.790000 sys 0.020000 (median of 50) After ! wall 0.842607 comb 0.840000 user 0.800000 sys 0.040000 (median of 50) If that would be considered a problem, more adjustments can be made, which are prematurate at this stage: cooking special variants of methods of the inner MissingAncestors object, retrieving local heads directly from Rust to avoid the cost of conversion. Effort would probably be better spent at this point improving the surroundings if needed. Here's another data point with a smaller repository, pypy, where performance is almost identical Before ! wall 0.015121 comb 0.030000 user 0.020000 sys 0.010000 (median of 186) After ! wall 0.015009 comb 0.010000 user 0.010000 sys 0.000000 (median of 184) Differential Revision: https://phab.mercurial-scm.org/D6430

rust-discovery: optimization of add commons/missings for empty arguments These two cases have to be catched early for different reasons. In the case of add_missing_revisions, we don't want to trigger the computation of the undecided set (and the children cache) too early: the later the better. In the case of add_common_revisions, the inner `MissingAncestors` object wouldn't know that all ancestors of its bases have already been removed from the undecided. In principle, that would in itself be a lead for further improvement: this remove_ancestors_from could be more incremental, but the current performance seems to be good enough. Differential Revision: https://phab.mercurial-scm.org/D6429

rust-discovery: using the children cache in add_missing The DAG range computation often needs to get back to very old revisions, and turns out to be disproportionately long, given that the end goal is to remove the descendents of the given missing revisons from the undecided set. The fast iteration capabilities available in the Rust case make it possible to avoid the DAG range entirely, at the cost of precomputing the children cache, and to simply iterate on children of the given missing revisions. This is a case where staying on the same side of the interface between the two languages has clear benefits. On discoveries with initial undecided sets small enough to bypass sampling entirely, the total cost of computing the children cache and the subsequent iteration becomes better than the Python + C counterpart, which relies on reachableroots2. For example, on a repo with more than one million revisions with an initial undecided set of 11 elements, we get these figures: Rust version with simple iteration addcommons: 57.287us first undecided computation: 184.278334ms first children cache computation: 131.056us addmissings iteration: 42.766us first addinfo total: 185.24 ms Python + C version first addcommons: 0.29 ms addcommons 0.21 ms first undecided computation 191.35 ms addmissings 45.75 ms first addinfo total: 237.77 ms On discoveries with large undecided sets, the initial price paid makes the first addinfo slower than the Python + C version, but that's more than compensated by the gain in sampling and subsequent iterations. Here's an extreme example with an undecided set of a million revisions: Rust version: first undecided computation: 293.842629ms first children cache computation: 407.911297ms addmissings iteration: 34.312869ms first addinfo total: 776.02 ms taking initial sample query 2: sampling time: 1318.38 ms query 2; still undecided: 1005013, sample size is: 200 addmissings: 143.062us Python + C version: first undecided computation 298.13 ms addmissings 80.13 ms first addinfo total: 399.62 ms taking initial sample query 2: sampling time: 3957.23 ms query 2; still undecided: 1005013, sample size is: 200 addmissings 52.88 ms Differential Revision: https://phab.mercurial-scm.org/D6428

discovery: new devel.discovery.randomize option By default, this is True, but setting it to False is a uniform way to kill all randomness in integration tests such as test-setdiscovery.t By "uniform" we mean that it can be passed to implementations in other languages, for which the monkey-patching of random.sample would be irrelevant. In the above mentioned test file, we use it right away, replacing the adhoc extension that had the same purpose, and to derandomize a case with many round-trips, that we'll need to behave uniformly in the Rust version. Differential Revision: https://phab.mercurial-scm.org/D6427

rust-discovery: optionally don't randomize at all, for tests As seen from Python, this is a new `randomize` kwarg in init of the discovery object. It replaces random picking by some arbitrary yet deterministic strategy. This is the same as what test-setdiscovery.t does, with the added benefit to be usable both in Python and Rust implementations. Differential Revision: https://phab.mercurial-scm.org/D6426

rust-discovery: exposing sampling to python Differential Revision: https://phab.mercurial-scm.org/D6425

rust-discovery: takefullsample() core implementation take_full_sample() browses the undecided set in both directions: from its roots as well as from its heads. Following what's done on the Python side, we alter update_sample() signature to take a closure returning an iterator: either ParentsIterator or an iterator over the children found in `children_cache`. These constructs should probably be split off in a separate module. This is a first concrete example where a more abstract graph notion (probably a trait) would be useful, as this is nothing but an operation on the reversed DAG. A similar motivation in the context of the discovery process would be to replace the call to dagops::range in `add_missing_revisions()` with a simple iteration over descendents, again an operation on the reversed graph. Differential Revision: https://phab.mercurial-scm.org/D6424

rust-discovery: core implementation for take_quick_sample() This makes in particular `rand` no longer a testing dependency. We keep a seedable random generator on the `PartialDiscovery` object itself, to avoid lengthy initialization. In take_quick_sample() itself, we had to avoid keeping the reference to `self.undecided` to cope with the mutable reference introduced by the the call to `limit_sample`, but it's still manageable without resorting to inner mutability. Sampling being prone to be improved in the mid-term future, testing is minimal, amounting to checking which code path got executed. Differential Revision: https://phab.mercurial-scm.org/D6423

rust-discovery: read the index from a repo passed at init This makes the API of the Rust PartialDiscovery object now the same (or rather a subset) of the Python object, hence easier to control through module policy down the road. Differential Revision: https://phab.mercurial-scm.org/D6517

rust-discovery: accept the new 'respectsize' init arg At this stage, we don't do anything about it: it will be meaningful in sampling methods that aren't implemented yet. Differential Revision: https://phab.mercurial-scm.org/D6516

merge with stable

unshelve: forget unknown files after a partial unshelve This is a follow-up patch to 6957f7b93e03. This allows hg to forget unknown files after a partial unshelve. Differential Revision: https://phab.mercurial-scm.org/D6724

flagutil: move addflagprocessor to the new module (API)

flagutil: move insertflagprocessor to the new module (API)

flagutil: move REVIDX_KNOWN_FLAGS source of truth in flagutil (API) Since REVIDX_KNOWN_FLAGS is "not really a constant" (extension can update it) and python integer,... it needs to be the responsability of a single module and always accessed through the module. We update all the user and move the source of truth in flagutil.

flagutil: move the `flagprocessors` mapping in the new module This module is meant to host most of the flag processing logic. We start with the mapping between flag and processors.

flagutil: create a `mercurial.revlogutils.flagutil` module The flagprocessings logic is duplicated in 2 extra places, and usually in a less robust flavor. This is a maintenance nightmare that I would like to see cleaned up. To do so I am creating a `flagutil` module to move flag processings related code and make it easily reusable by other code.

rawdata: register the method for `ifiledata` The interface have a `revision(..., raw=False)` method so it should get a `rawdata` one. I am not sure why nothing complained about the lack of it earlier.

rawdata: implement the method for `unionrepo` too This is required for all implementations.

rawdata: implement the method for `remotefilelog` too This is needed for all storage implementations.

rawdata: implement `rawdata` for `simplestore` too This is needed for all implementation.

rawdata: forward `rawdata` call on `manifestlog` This needs to be sent to the underlying `revlog` too.

rawdata: implement `rawdata` for `sqlitestore` too This is a different store, it needs it declared.

rawdata: add the method to bundlerevlog The bundlerepo logic has its own `revision` method on its own `revlog` object. We need to "implement" `rawdata` there too.

rawdata: forward the method call on `filelog` object We have a new method, we need to expose it.

rawdata: introduce a `rawdata` method on revlog This method aims at replacing `revision(..., raw=True)` call. The purpose of data returned without and without raw are different enough that having two different method would make sense. This split is motivated by other work aiming at storing data on the side of the main revision of a revlog. Having a cleaner API makes it simpler to add this work. The series following this first changesets is organised as follow: 1) add `rawdata` method everywhere it is useful 2) update all caller 3) implement all `rawdata` method without using `revision` 4) deprecate the `rawdata` parameter of `revision`

revlog: split a `_revisiondata` method to file `revision` job We are about to introduce more public method to access revision data (eg: `rawdata`). revset subclass tend to recursively call `revision` which will create all kind of issue with the coming series. To avoid them we introduce an explicit difference between the internal call and the public all. This will be useful for later work anyway (so the subclass issue is just moving it earlier in the series). I am not sure if the subclass are actually doing something sensible. However, I am certain I don't want to be rabbit holed into figuring it out right now.

continue: added support for transplant This creates a seperate function `continuetransplant()` containing logic for resuming transplant from interrupted state. `continuetransplant()` is then registered as `continuefunc` for state detection API. Results are shown in tests. Differential Revision: https://phab.mercurial-scm.org/D6689

merge with stable

branchmap: explicitly warm+write all subsets of the branchmap caches 'full' claims it will warm all of the caches that are known about, but this was not the case - it did not actually warm the branchmap caches for subsets that we haven't requested, or for subsets that are still considered "valid". By explicitly writing them to disk, we can force the subsets for ex: "served" to be written ("immutable" and "base"), making it cheaper to calculate "served" the next time it needs to be updated. Differential Revision: https://phab.mercurial-scm.org/D6710

changectx: extract explicit computechangesetfilesremoved method from context Right now, the logic around changeset centric removed files data are buried into the "changectx" code. We extract this code in a dedicated method (in the scmutil module) for clarity. This clarity will help to explicitly compute and caches these data in the future.

changectx: extract explicit computechangesetfilesadded method from context Right now, the logic around changeset centric added files data are buried into the "changectx" code. We extract this code in a dedicated method (in the scmutil module) for clarity. This clarity will help to explicitly compute and caches these data in the future.

copies: extract an explicit `computechangesetcopie` method from context Right now, the logic around changeset centric copies data are buried into the "changectx" code. We extract this code in a dedicated method (in the copies module) for clarity. This clarity will help to explicitly compute and caches these data in the future.

config: fix fm.data() handling of defaultvalue This is a follow-up patch to rHG51a2e3102db2. This moves `fm.data()` out of the if block in `commands.config()`. Differential Revision: https://phab.mercurial-scm.org/D6720

config: remove pycompat.bytestr() for defaultvalue This is a follow-up patch to 51a2e3102db2. This removes `pycompat.bytestr` to preserve `None` in `commands.config()`. Differential Revision: https://phab.mercurial-scm.org/D6712

unshelve: clear shelvedstate and _finishunshelve() on partial unshelve On a partial unshelve, `shelvedstate` was not cleared and `_finishunshelve()` was not called. Ideally, these should be called on this case. This patch makes `shelvedstate` to delete after a successful partial unshelve and calls `_finishunshelve()` in the same case. Differential Revision: https://phab.mercurial-scm.org/D6708

unshelve: delete shelvedstate after a successful unshelve --continue `unshelve --continue` was preventing the deletion of `shelvedstate` on a partial `unshelve`. Ideally, `shelvedstate` should be deleted after a successful `unshelve`. Now, the behavior of `unshelve --continue` will be as follows in interactive mode: 1] The user tried to `unshelve` changes interactively but, ran into conflicts. 2] They resolved the conflicts and triggered `unshelve --continue` but, unshelved changes partially. 3] Now, on trying to do `unshelve --continue` again will abort as the last `unshelve` was successful and we are deleting the `shelvedstate`. 4] If they want to unshelve the remaining shelved change, they need to trigger `unshelve` without `--continue`. Differential Revision: https://phab.mercurial-scm.org/D6694

unshelve: handle stripping changesets on interactive mode On interactive mode, changesets on `nodestoremove` should be stripped regardless of the shelve is partial or not. This patch modifies `unshelvecontinue()` to do that. Differential Revision: https://phab.mercurial-scm.org/D6686

byteify-strings: add --version argument This is indispensable for automated tools to detect changes in behavior.

byteify-strings: add space in special comments to silence flake8 error This is done soon enough that nobody has had the time to use this feature yet.

revset: drop argument when it's None getstack's definition is `getstack(repo, rev=None)`, so providing None explicitly is unnecessary. Moreover, when x is not None, it's definitely not a revision but a part of a parsed tree of revset arguments. Differential Revision: https://phab.mercurial-scm.org/D6707

stack: remove unnecessary reverse() predicate Stack already sorts revisions, so no need to do it twice. This change was a part of D2400, which didn't land for other reasons. See also D2399, where this change was suggested. Differential Revision: https://phab.mercurial-scm.org/D6706

automation: increase root volume size on Linux It is close to full in the AMI. I actually ran out of space running tests without increasing the volume size! Differential Revision: https://phab.mercurial-scm.org/D6716

automation: install Rust in Linux environment This will install Rust 1.31.1, 1.34.2, and whatever stable is at the time the install runs. We install 1.31.1 as our minimum supported Rust version (I think that's what we're currently targeting) and 1.34 because that's what Debian 10 is shipping. Differential Revision: https://phab.mercurial-scm.org/D6715

automation: update packages in requirements files We like keeping up to date. The content of the autogenerated files changed slightly because I used a newer version of `pip-compile` than what was used previously. Differential Revision: https://phab.mercurial-scm.org/D6714

automation: install latest Python versions This required bumping the pyenv commit so the new versions are available. Differential Revision: https://phab.mercurial-scm.org/D6713

upgrade: introduce the internal code for revlog cloning selection For now we still clone every single revlogs but all the selection mechanism is now in place in the lower layer. The next changesets will introduce the user interface part of the selection.

upgrade: introduce a _copyrevlog method This function copies a revlog from the old store to the new one, without re-adding all deltas manually. This will eventually save a lot of time when some revlog does not needs any conversions. Code actually using this will be introduced in later changesets.

upgrade: rename `_copyrevlogs` to `_clonerevlogs` The underlying revlog method is named `clone`, keeping the naming consistent seems clearer. This is motivated to clarify the difference with an (upcoming) function that simply copy revlog files as is.

upgrade: walk the source store file only once I don't expect this to have a significant performance impact, but it seems simpler and saner to do the operation only once and to keep the result around.

upgrade: always use full text if "full-add" mode is enable We should not be using a delta since the goal is to perform a full addition from scratch in all cases. Without this patch, `hg debugupgraderepo --optimize re-delta-fulladd --run` can crash.

byteify-strings: fix misalignment with multi-line parenthesis This improves the current fix to also take into account cases where the last line ended on the opening `(`, `[` or `{` and adds a regression test.

byteify-strings: add test for byteify-strings.py This tests the basic features expected from this script, some cases may not be covered yet. A future patch will demonstrate an issue with multi-line `(`, `[` and `{` alignment and propose a fix.

merge with stable

byteify-strings: add cli argument to handle `attr*()` when they are methods Certain code bases have useful utils that wrap the builtin functions, and are called like `util.setattr`.

byteify-strings: simplify default value for `--treat-as-kwargs`

byteify-strings: add --treat-as-kwargs argument to handle kwargs-like objects This argument will help extensions move to Python 3 as keyword arguments should not be byte-prefixed. Most of the time, code bases will call this object `kwargs`, but other conventions exist like `opts`, so it should make sense to allow for custom names. This is a best effort solution that does minimal static checking; cases like `options = [o for o in ('a', 'b', 'c') if kwargs.get(o)]` and other just as complicated will not be detected.

byteify-strings: add helpers to check for item access or method call These helpers will be used in a future patch, split for ease of review.

byteify-strings: add support for ignore comments Our simple token analysis is sometimes not clever enough, we need to be able to turn off our script for parts of the code. This change introduces three special comments: - `#no-py3-transform` to tell `byteify-strings` ignore the next line - `#py3-transform: off` to ignore everything until the end of the file - `#py3-transform: on` to stop ignoring The last two can be particularly useful within Python 2/3 compatibility files.

byteify-strings: handle triple quoted strings if they are not docstrings As with anything in this script, this is a best effort approach. Most of the time, when a triple quoted string is assigned to something, it's not a docstring.

byteify-strings: handle multi-line strings in _ensuresysstr The current implementation did not handle calls like `repo.ui.log("first line" "other line")` correctly.

fix: run fixer tools in the repo root as cwd so they can use the working copy This lets fixer tools do things like find configuration files, with the caveat that they'll only see the version of that file in the working copy, regardless of what revisions are being fixed. Differential Revision: https://phab.mercurial-scm.org/D6440

config: add defaultvalue template keyword This patch tries to fix one of the issues mentioned in issue6014. This adds a new `defaultvalue` template keyword to be used with `hg showconfig` to get the default value of the config item. Differential Revision: https://phab.mercurial-scm.org/D6704

merge with stable

module-policy: update rust extension import to use the new module policy Differential Revision: https://phab.mercurial-scm.org/D6677

transaction: leave unfinished without crashing when not properly released I think the transaction.__del__ is there just as a last resort in case we (or an extension) forgot to release the transaction. When that happens, the repo can (or will on Python 3?) get deleted before the transaction. This leads to a crash in test-devel-warnings.t on Python 3 because we tried to access repo.dirstate, where repo was retried from a weak reference. There's not much we can do here, but let's at least avoid the crash. The user will have run `hg recover` afterwards regardless. Differential Revision: https://phab.mercurial-scm.org/D6664

unshelve: add abort on using continue and interactive together `unshelve --continue --interactive` will not work as expected by the user as the mode of in-progress unshelve is preserved and cannot be overwritten. This patch makes `unshelve` to throw an error on using both `--continue` and `--interactive` together with `unshelve`. Differential Revision: https://phab.mercurial-scm.org/D6703

py3: add one more test to list of passing tests Found by buildbot. Differential Revision: https://phab.mercurial-scm.org/D6702

tests: sort imports in test-bookmarks-corner-case.t test-check-module-imports.t breaks on py3 without this change. Differential Revision: https://phab.mercurial-scm.org/D6701

fix: add some new test cases These cover a couple of behaviors we were testing at Google that weren't covered here before. Differential Revision: https://phab.mercurial-scm.org/D6698

unshelve: store information about interactive mode in shelvedstate This is a follow-up patch to 5162753c4c14. This makes `unshelve` stores the information about interactive mode on conflicts. Differential Revision: https://phab.mercurial-scm.org/D6679

unshelve: create a matcher only if required on creating unshelve ctx Differential Revision: https://phab.mercurial-scm.org/D6687

unshelve: changes how date is set on interactive mode On an interactive unshelve, the remaining changes are shelved again for later. This patch modifies the date of remaining shelved change to the time of interactive shelve. Differential Revision: https://phab.mercurial-scm.org/D6685

unshelve: unify logic around creating an unshelve changeset This is a follow-up patch to 5162753c4c14 on addressing reviews on the commit. This unifies the logic around creating an unshelve changeset. Differential Revision: https://phab.mercurial-scm.org/D6683

fix: ignore fixer tool configurations that are missing patterns This is to prevent a crash under the same circumstances. This is also to avoid data loss due to accidental application of a fixer tool to all files, if the matching logic somehow changed to that effect. Affecting all files until otherwise configured would be dangerous, and not very useful. We shouldn't abort because there may be other fixers, and it may still be useful to run them without having to adjust configuration. A user might not feel confident in changing configs, for example. Differential Revision: https://phab.mercurial-scm.org/D6693

fix: add a test case around the effect of cwd on pattern matching This was not covered by previous tests. It is related to a regression encountered at Google due to misconfiguration of [fix]. Differential Revision: https://phab.mercurial-scm.org/D6692

fix: remove support for :fileset sub-config in favor of :pattern Differential Revision: https://phab.mercurial-scm.org/D6691

fsmonitor: add support for extra `hg debuginstall` data This might make some things easier to debug, and for default bug report templates it'll help collect more data from users all at once. I don't actually need fsmonitor in our bug reports (we don't use it), but this demonstrates the utility of the preceding patches without having to add new things to core. Differential Revision: https://phab.mercurial-scm.org/D6682

debugcommands: add support for extensions adding their own debug info We've had a couple of cases where it'd be handy at Google to add data to `hg debuginstall`'s output. We've kludged around that at various times, but it seems reasonable to let extensions add their own data here so extension maintainers can get useful extra data. Differential Revision: https://phab.mercurial-scm.org/D6681

fsmonitor: refactor watchmanclient.client to accept ui and repo path This will make my next patch simpler. Differential Revision: https://phab.mercurial-scm.org/D6680

Added signature for changeset 181e52f2b62f

Added tag 5.1.2 for changeset 181e52f2b62f

merge: back out changeset a4ca0610c754 (parents order when grafting a merge) Turns out it's not enough to just swap parents, because when we do, there are unexpected bad side effects, such as a tracked file becoming untracked. These side effects need more code to be handled properly, but it's not written yet. Let's back this feature out from stable for now and some day implement it on default instead.

merge: respect parents order when using `graft` on a merge, this time for real See a4ca0610c754. potherp1 is a boolean variable that means "pother is ctx.p1", and parents is naturally [ctx.p1, ctx.p2]. pctx is always removed from parents, so if pctx is parents[0], then we end up using parents[1] as pother. To be true to its name, potherp1 should then be True only when pctx is at parents[1].

phabricator: don't abort if property writing fails during amending Currently if one of the writediffproperty calls fails due to network issues during the amending of commit messages to include the Diff. Rev. line, the transaction is aborted and rolled back. This means that the associations between the commits and the DREVs are lost for any already amended commits because the removal of the local tags isn't covered by the rollback. Differential Revision: https://phab.mercurial-scm.org/D6835

merge: respect parents order when using `graft` on a merge The previous code did not record the index of the replaced parent. It was always using the "graft" destination as `p1`. This could switch parents order in some situation (eg: some of the evolve evolving merge case). Recording and using the information fixes the issue in evolve. We are not aware of core commands calling graft in that fashion, so we could not build a simple test case for it using core commands.

tests: register test-merge-combination.t as small but slow run-tests.py use file size as an heuristic for test run time. The new `test-merge-combination.t` is a small file that do a lot of processing. As a result it tend to be scheduled really late but delay the full test run by a lot. On an example test run, the one-before-last test completed 279s after the start of the run, while `test-merge-combination.t` finished 355s after it. A 76s delay. This delay can be avoided since `test-merge-combination.t` only got started 175s after the start of the run.

test: allow different result for zstd compression (issue6188) test-repo-compengines fails on big-endian due to different file size, but the repo doesn't seem broken, so allow both sizes. Differential Revision: https://phab.mercurial-scm.org/D6787

Added signature for changeset a4e32fd539ab

Added tag 5.1.1 for changeset a4e32fd539ab

python-zstandard: apply big-endian fix (issue6188) This is a port of commit d4baf1f95b811f40773f5df0d8780fb2111ba6f5 from the upstream project to fix python-zstandard on 64-bit big-endian.

exchange: abort on pushing bookmarks pointing to secret changesets (issue6159) Until now, if there is a bookmark points to a changeset which is in secret phase, hg will push the bookmark, but not the changeset referenced by that bookmark. This leaves the server bookmarks in a bad state, because that bookmark now points to a revision that does not exist on the server. This patch makes hg to abort on such cases. Differential Revision: https://phab.mercurial-scm.org/D6731

tests: add test to demonstrate issue6159 Differential Revision: https://phab.mercurial-scm.org/D6740

packaging: add Bullseye, remove Jessie Jessie is oldoldstable now, and Bullseye is going to be the next stable (now testing). We're continuing to support the current oldstable, stable and testing releases. Differential Revision: https://phab.mercurial-scm.org/D6762

packaging: add Cosmic and Disco, remove Trusty and Artful - Trusty was publicly supported until 2019-04-30 - Artful was publicly supported until 2018-07-19 - Cosmic was publicly supported until 2019-07-18 - Disco will be publicly supported until 2020-01 Cosmic is officially out-of-date, but since it still may be in use, and because we didn't add it when it first came out, I think it would be nice to support it until the next time somebody decides to update this list of Ubuntu releases. Differential Revision: https://phab.mercurial-scm.org/D6761

makefile: run Rust tests if cargo is installed While no particular minimum toolchain version is targeted as of yet, this serves as a first step to make more people/machines run the Rust tests.

tests: use `tr -d` and not `tr --delete` as the latter is absent on BSD tr(1) Differential Revision: https://phab.mercurial-scm.org/D6729

fncache: make debugrebuildfncache not fail on broken fncache The code reading the fncache changed in 5.0, to complain if the file is not \n terminated. This makes apparent the fact that the fncache gets corrupted. Make it possible to recover, instead of having `hg debugrebuildfncache` failing by saying `(run hg debugrebuildfncache)`. The corruption itself is most likely due to hg not using fsync in general, and so various bad things can happen. Here, the reported problems happened when running out of disk space. So I suspect that because the fncache is much bigger than the average commit/pull, when running out of disk space, the bulk of the pull may succeed, but the new fncache may get half-written and still renamed into place. Differential Revision: https://phab.mercurial-scm.org/D6722

fncache: show that debugrebuildfncache is partly broken Differential Revision: https://phab.mercurial-scm.org/D6721

test: further fixes to matching for run-tests.py bug The fix in bac24a8a095a did not fix the full issue, because the extra number also eat some of the separator space. Since we are already matching arbitrary number of space, we easily fix the matching.

demandimport: explicitly declare `_session` at the module level The `_session` module level variable is set within a function using the `global` keyword. This confuses my `test-check-pyflakes.t`. Explicitly declaring the variable at the top level solves the issue (and seems absolutely reasonable).

tests: give more room for slowness in test-run-tests.t The test expected any run-test.py run to end in less than 10 seconds. On slower loaded CI machine, this gets slower than that. We give a bit more room to the regexp.

relnotes: copy "next" to "5.1" and clear "next" To avoid merge conflicts, we want to avoid modifying the file on multiple branches in parallel. This patch is therefore meant to be applied to the stable branch and then quickly be merged to default (at least before edits are made to relnotes/next there). Another option would have been to copy the file on the stable branch and to clear it on the default branch. However, that still results in conflicts if the copy is edited on the stable branch (Mercurial would try to apply the changes from the default branch to it). We could also delete the file in one commit and recreate it in another commit. However, Mercurial is quite inconsistent in what it considers a break in history (see test-copies-unrelated.t), so I'd like to avoid that. Differential Revision: https://phab.mercurial-scm.org/D6705

automation: push changes affecting .hgtags When I went to build the 5.1 tag using the in-repo automation, the automatic version calculation failed to deduce the clean 5.1 version string because we had only pushed the changeset corresponding to the 5.1 tag and not the changeset containing the 5.1 tag. So from the perspective of the remote repo, the 5.1 tag didn't exist yet and automatic version deduction failed. This commit changes the `hg push` to also push all changesets affecting the .hgtags file, ensuring the remote has up-to-date tags information. I tested this by creating a local draft changeset with a dummy tag value on a different DAG head and instructed the automation to build a revision that didn't have this change to .hgtags. The tag was successfully pushed and the built package had a version number incorporating that tag. Sending this to stable so the 5.1.1 automation hopefully "just works."

bookmarks: actual fix for race condition deleting bookmark This is a simple but efficient fix to prevent the issue tested in `test-bookmarks-corner-case.t`. It might be worth pursuing a more generic approach where filecache learn to depend on each other, but that would not be suitable for stable. The issue is complicated enough that I documented the race and its current solution as inline comment. See this comment for details on the fix.

strip: access bookmark before getting a reference to changelog Bookmark access might invalidate the current changelog (to make sure both are in a reasonable synchronisation state). So we should grab the reference to changelog after we access bookmark. Otherwise we risk using a dead object for the whole strip process. (note: this dead object business probably requires a new layers of checking)

test: use a more verbose output in the test While debugging some test failure, I released the test never checks if the relevant changesets were preserved. So I am updating the test from `hg parents` usage to `hg log -G` with a special template. This increase the area covered by the test and clarify the test failures.

Added signature for changeset e91930d712e8

Added tag 5.1 for changeset e91930d712e8

automation: execute powershell when connecting For some reason, the ability to execute PS scripts appears to come online after the ability to execute regular command scripts. This is creating race conditions when connecting to instances resulting in our wait_for_winrm() returning before PS is available leading to an exception being thrown in other code. Let's change the client connection code to execute a minimal PS script so we can try to trap the exception in wait_for_winrm().

automation: allow exit code of 1 for `hg push` `hg push` exits 1 for no-ops. No-op pushes should be fine in the context of automation.

curses: do not setlocale() at import time (issue5261) setlocale() can break date formatting/parsing functions because they are locale dependent. We should avoid doing setlocale() as possible. This patch moves setlocale() just before curses.wrapper(), which function is documented to "initialize curses." I don't know the details about the curses initialization, but I *think* this would work as well. Maybe we can extract a curses setup function later. https://www.mercurial-scm.org/pipermail/mercurial-devel/2019-February/128788.html

contrib: install Python 3.8b2 instead of 3.8a2 Let's install the most recent Python 3.8 distribution. Differential Revision: https://phab.mercurial-scm.org/D6674

automation: make Windows base image name configurable Since automation broke in the middle of the 5.0 release cycle, there's a good chance it will break again in the future. While a robust solution might be to search for all available images and choose the newest one, it does seem useful to be able to explicitly choose the name of the image to find and use so users can opt in to using a different image. This commit implements that functionality. Differential Revision: https://phab.mercurial-scm.org/D6673

automation: extract strings to constants Differential Revision: https://phab.mercurial-scm.org/D6672

automation: use newer Windows base image It looks like the old base image disappeared. Let's use a newer image that exists today. Differential Revision: https://phab.mercurial-scm.org/D6671

copies: fix crash on in changeset-centric tracing from commit to itself When we trace copies from a changeset to itself, the "work" queue ends up empty and we hit the "assert False" after it. It was only the last of the three added tests that failed before this patch. That is because the other two cases have fast paths, so _committedforwardcopies() is never reached. Differential Revision: https://phab.mercurial-scm.org/D6675

unshelve: add help text on --interactive in verbose mode This is a follow-up patch to rHG9eace8d6d537. This modifies the help text of unshelve in verbose mode to mention the details about `--interactive` flag. Differential Revision: https://phab.mercurial-scm.org/D6676

amend: stop committing unrequested file reverts (issue6157) Differential Revision: https://phab.mercurial-scm.org/D6667

amend: add a test for a simplified version of issue6157 Differential Revision: https://phab.mercurial-scm.org/D6666

py: error out if a "skip" character was given with non-dict to util.dirs() util.dirs() keeps track of the directories in its input collection. If a "skip" character is given to it, it will assume the input is a dirstate map and it will skip entries that are in the given "skip" state. I think this is used only for skipping removed entries ("r") in the dirtate. The C implementation of util.dirs() errors out if it was given a skip character and a non-dict was passed. The pure implementation simply ignored the request skip state. Let's make it easier to discover bugs here by erroring out in the pure implementation too. Let's also switch to checking for the dict-ness, to make the C implementation (since that's clearly been sufficient for many years). This last change makes test-issue660.t pass on py3 in pure mode, since the old check was for existence of iteritems(), which doesn't exist on py3. Differential Revision: https://phab.mercurial-scm.org/D6669

py3: fix incorrect fix of test-setdiscovery.t in eb27d9eee2cc Both places should have been changed from 185 to 187. Differential Revision: https://phab.mercurial-scm.org/D6668

Added signature for changeset e386b5f4f836

Added tag 5.1rc0 for changeset e386b5f4f836

merge default into stable for 5.1 release

rust-filepatterns: unescape comment character property There were multiple issues in the original implementation: a. the local variable "line" dropped soon after replace_slice() applied b. replace_slice() was noop since br"\#".len() != b"#" This patch uses bytes::Regex::replace_all() since it seems the simplest way to replace bytes of arbitrary length, and I don't think we have to avoid using Regexp here.

rust-filepatterns: use literal b'#' instead of cast

rust-filepatterns: fix type of warnings tuple to (bytes, bytes) Otherwise warn() in match.py would fail if the warning contains non-ASCII character. We might want to add a thin ByteString wrapper around Vec<u8> to implement ToPyObject<ObjectType = PyBytes>, but I'm not sure.

hgignore: add escape syntax test for glob patterns The last example, [\#], is what the rust implementation fails to parse. The other escapes can be removed by regexp engine or _globre().

hgignore: add a few more weird patterns to test case

hgignore: update \-escape test to reflect actual behavior "\\<char>" is not an escape character but "\\" + <char>.

py3: add a b'' prefix in tests/test-convert-identity.t Differential Revision: https://phab.mercurial-scm.org/D6662

lookup: don't use "00changelog.i@None" when lookup of prefix fails We were shadowing the "node" variable, so we always passed None to the LookupError instead of the node we meant to pass. (This showed up in py3 tests since py3 doesn't like to format None using "%s".) Differential Revision: https://phab.mercurial-scm.org/D6661

py3: fix test-setdiscovery.t on Python 3 by conditionalizing two lines I'm not clear why this behaves very slightly differently on Python 3, but I'm also not concerned about it. Differential Revision: https://phab.mercurial-scm.org/D6658

commands: removed part of description from abort and continue The description for registration of new `continuefunc` or `abortfunc` is removed as it is not required from user perspective. Differential Revision: https://phab.mercurial-scm.org/D6660

tests: glob over some timing numbers in test-shelve.t The Windows bot is slow enough that it was 2s in the first hunk. Differential Revision: https://phab.mercurial-scm.org/D6663

py3: another passing test Differential Revision: https://phab.mercurial-scm.org/D6656

cleanup: remove redundant import For some reason the import checker only caught this on py3. Differential Revision: https://phab.mercurial-scm.org/D6657

shelve: modify help text on --interactive We now have `unshelve --interactive` after rHG5162753c4c14. So, the help text on `shelve --interactive` suggesting that it only works for `shelve` can be removed. Differential Revision: https://phab.mercurial-scm.org/D6654

unshelve: mark unshelve interactive as experimental This is a follow-up patch to rHG5162753c4c14. We have the logic for interactive unshelve under `_rebaserestorecommit()`. So, we might get conflicts even if there are conflicting changes other than selected changes by the user. We should mark unshelve `--interactive` as `EXPERIMENTAL` until we solve this issue. Differential Revision: https://phab.mercurial-scm.org/D6653

commit: improve the files field of changelog for merges Currently, the files list of merge commits repeats all the deletions (either actual deletions, or files that got renamed) that happened between base and p2 of the merge. If p2 is the main branch, the list can easily be much bigger than the change being merged. This results in various problems worth improving: - changelog is bigger than necessary - `hg log directory` lists many unrelated merge commits, and `hg log -v -r commit` frequently fills multiple screens worth of files - it possibly slows down adjustlinkrev, by forcing it to read more manifests, and that function can certainly be a bottleneck - the server side of pulls can waste a lot of time simply opening the filelogs for pointless files (the constant factors for opening even a tiny filelog is apparently pretty bad) So stop listing such files as described in the code. Impacted merge commits and their descendants get a different hash than they would have without this. This doesn't seem problematic, except for convert. The previous commit helped with that in the hg->hg case (but if you do svn->hg twice from scratch, hashes can still change). The rest of the description is numbers. I don't have much to report, because recreating the files list of existing repositories is not easy: - debugupgradeformat and bundle/unbundle don't recreate the list - export/import tends to choke quickly applying patches or on description that contain diffs, - merge commits from the convert extension don't have the right files list for reasons orthogonal to the current commit - replaying the merge with hg update/hg merge/hg revert --all/hg commit can end up failing in hg revert - I wasn't sure that using debugsetparents + debugrebuilddirstate would really build the right thing I measured commit time before and after this change, in a case with no files filtered out, several files filtered out (no difference) and 5k files filtered out (+1% time). Recreating the 100 more recent merges in a private repo, the concatenated uncompressed files lists goes from 1.12MB to 0.52MB. Excluding 3 merges that are not representative, then the size goes from 570k to 15k. I converted part of mozilla-central, and observed file list shrinking quite a bit too, starting at the very first merge, 733641d9feaf, going from 550 files to 10 files (although they have relatively few merges, so they probably wouldn't care). Differential Revision: https://phab.mercurial-scm.org/D6613

convert: add a config option to help doing identity hg->hg conversion I want to change the computation of the list of files modified by a commit. In principle, this would simply change a cache. But since this information is stored in commits rather than a cache, changing it means changing commit hashes (going forward). Some users rely on the convert extension from hg to hg not changing hashes when nothing changes (usually). Allow these users to preserve hashes despite changes to the changelog files computation by reusing these files lists when the manifest is unchanged (since these files list are derived from the manifest). Differential Revision: https://phab.mercurial-scm.org/D6643

tests: show the files fields of changelogs for many merges I don't think there's coverage for many of the subtle cases, and I found it hard to understand what the code is doing by reading it. The test takes 40s to run on a laptop, or 9s with --chg. I have yet to find a description of what the files field is supposed to be for merges. I thought it could be one of: 1. the files added/modified/removed relative to p1 (wouldn't seem useful, but `hg diff -c -r mergerev` has this behavior) 2. the files with filelog nodes not in either parent (i.e., what is needed to create a bundle out of a commit) 3. the files added/removed/modified files by merge itself [1] It's clearly not 1, because file contents merges are symmetric. It's clearly not 2 because removed files and exec bit changes are listed. It's also not 3 but I think it's intended to be 3 and the differences are bugs. Assuming 3, the test shows that, for merges, the list of files both overapproximates and underapproximates. All the cases involve file changes not in the filelog but in the manifest (existence of file at revision, exec bit and file vs symlink). I didn't look at all underapproximations, but they looked minor. The two overapproximations are problematic though because they both cause potentially long lists of files when merging cleanly. [1] even what it means for the merge commit itself to change a file is not completely trivial. A file in the merge being the same as in one of the parent is too lax as it would consider that merges change nothing when they revert all the changes done on one side. The criteria used in the test and in the next commit for "merge didn't touch a file" is: - the parents and the merge all have the same file - or, one parent didn't touch the file and the other parent contains the same file as the merge Differential Revision: https://phab.mercurial-scm.org/D6612

phabricator: handle local:commits time being string or int When setting local:commits arcanist has different behaviour depending on whether the repo is git or hg. With hg it sets the time as a number, since it calls PHP's strtotime on the value, but with git it sets it as a string. Normally this wouldn't be an issue since phabread wouldn't be interacting with Phabricator Revisions for git repos, but Mozilla has a secondary workflow for git users that uses the git-cinnabar tool to interact with their hg repos. When a git-cinnabar user uses the moz-phab tool to submit patches for mozilla-central it makes use of Mozilla's fork of arcanist, which works with their local git version of m-c, and thus sets the local:commit time as a string, and then translates the commit hashes. Currently when encountering such DREVS phabread dies with "TypeError: %d format: a number is required, not str". phabsend also used to set it as a string but wouldn't have encountered the issue with its own DREVs since it would read hg:meta first. Differential Revision: https://phab.mercurial-scm.org/D6650

phabricator: demonstrate broken phabread on string local:commit times Differential Revision: https://phab.mercurial-scm.org/D6649

unshelve: add interactive mode Until now, there is no way to `unshelve` selected changes only from the stored shelve as given in issue6162. This patch makes `unshelve` perform with certain changes only by adding an interactive mode. Differential Revision: https://phab.mercurial-scm.org/D6596

blackbox: disable extremely verbose logging (issue6110) This is maybe not the best way to go about fixing this, but anything is better than the status quo. Differential Revision: https://phab.mercurial-scm.org/D6611

continue: added support for unshelve This patch adds the support for `ushelve` in `hg continue` plan. `hgcontinueunshelve()` has been created for independent calls. In case an interrupted unshelve is resumed via hg continue the shelvedstate needs to be loaded seperately. This has been ensured by `_loadunshelvedstate()` `hgcontinueunshelve()` is then registered as `continuefunc` for state detection API. Results are shown as tests. Differential Revision: https://phab.mercurial-scm.org/D6652

continue: added support for rebase This adds support of rebase to hg continue plan. An independent continue logic for rebase is created under continuerebase() function. For this a seperate rebaseruntime object is created under the function to handle an interrupted rebasestate. Results of tests are shown. Differential Revision: https://phab.mercurial-scm.org/D6646

continue: added logic for hg continue This is part of GSoC19 project `Implement abort and continue commands`. This patch is part of the continue plan. This adds the basic logic for hg continue. This command aborts an multistep operation like graft, histedit, rebase, transplant and unshelve if they are in an unfinished state. The first part of the logic is determining the unfinished operation from the state detection API under statemod. This API is extended to support hg continue by adding a method to register the abort logic as a function (here continuefunc). Once the unfinished operation is determined the registered logic is used to resume the command in case it is interrupted. The benefit of this kind of framework is that any new extension developed can support hg continue by registering the command and logic under statedetection API. hg continue currently supports --dry-run/-n flag only. It is used to dry run hg abort Differential Revision: https://phab.mercurial-scm.org/D6645

rust-utils: remove buggy assertion While this assertion had good intentions, it broke existing behavior with a nasty panic. Differential Revision: https://phab.mercurial-scm.org/D6651

rust-utils: add docstrings and doctests for utils.rs Differential Revision: https://phab.mercurial-scm.org/D6635

rust: switch hg-core and hg-cpython to rust 2018 edition Many interesting changes have happened in Rust since the Oxidation Plan was introduced, like the 2018 edition and procedural macros: - Opting in to the 2018 edition is a clear benefit in terms of future proofing, new (nice to have) syntactical sugar notwithstanding. It also has a new non-lexical, non-AST based borrow checker that has fewer bugs(!) and allows us to write correct code that in some cases would have been rejected by the old one. - Procedural macros allow us to use the PyO3 crate which maintainers have expressed the clear goal of compiling on stable, which would help in code maintainability compared to rust-cpython. In this patch are the following changes: - Removing most `extern crate` uses - Updating `use` clauses (`crate` keyword, nested `use`) - Removing `mod.rs` in favor of an aptly named module file Like discussed in the mailing list ( https://www.mercurial-scm.org/pipermail/mercurial-devel/2019-July/132316.html ), until Rust integration in Mercurial is considered to be out of the experimental phase, the maximum version of Rust allowed is whatever the latest version Debian packages. Differential Revision: https://phab.mercurial-scm.org/D6597

rust-utils: use new find_dirs iterator In cad3dde7a573, the `find_dirs` util was introduced, but the second changeset that made use of it didn't apply. This change fixes the issue. Differential Revision: https://phab.mercurial-scm.org/D6639

inno: correct the path display in a literal block of the readme Otherwise, the path components allrantogether. Differential Revision: https://phab.mercurial-scm.org/D6648

copies: remove unnecessary override of p[12]copies() in workingctx The implementation is identical to the version inherited from basectx. Differential Revision: https://phab.mercurial-scm.org/D6647

tests: properly position conditional output on Windows in test-subrepo.t The test runner doesn't always guess the right location when optional output is missing. This goes with f6540aba8e3e. Differential Revision: https://phab.mercurial-scm.org/D6640

abort: removed labels argument from abortmerge() Labels are used to label the code that belongs to `working copy` and `merge rev` in case of a conflicted state. No such labelling is required while aborting merge as conflicted parts are reverted to normal. Differential Revision: https://phab.mercurial-scm.org/D6638

py3: source-transform only call-sites of iteritems(), not definitions branchmap.branchcache, among other classes, defines a iteritems(). That currently gets replaced by items() by the source transformer. That makes it harder for extensions to work with both py2 and py3, since they have to call either items() or iteritems() on branchcache. Let's not replace definitions of iteritems() (and itervalues()) and only replace the call-sites. We need to also add an items() alias to branchcache (etc) so our transformer call-sites will find it. Differential Revision: https://phab.mercurial-scm.org/D6641

py3: fix formatting of branchmap log messages with repo.filtername=None `"%s" % None` does not work on py3. I've extracted a little function for producing a formatted message given the filter name. Differential Revision: https://phab.mercurial-scm.org/D6644

automation: correct the path separator in LIBPATH on Windows I haven't tried building the x86 installer, but happened to notice this when working on the thg installer. Experimenting in PowerShell seems to show that LIBPATH was expanded at the end, but with ':' between, it effectively corrupted `${root}\WinSDK\Lib` and the first path in LIBPATH. Differential Revision: https://phab.mercurial-scm.org/D6642

abort: added support for merge This adds support of `hg merge --abort` to `hg abort` plan. This involves refactoring `hg.merge` into two different functions removing the abort logic of `merge` from `hg.merge` and then creating a seperate `hg.abortmerge` to handle the abort logic so that the abortion of merge can be called independently. `hg.abortmerge` is then registered as `abortfunc` for the state detection API so that `commands.abort` can use it to deal with an unfinished merge operation. Results are shown as tests. Differential Revision: https://phab.mercurial-scm.org/D6588

abort: added support for unshelve This patch adds the support for shelve in `hg abort` plan. For this the logic to load a `shelvedstate` and the error handling for it had been shifted to a seperate function `_loadunshelvedstate()`. This returns a tuple with `state` file and `opts.` `hgabortunshelve()` has been created for independent calls. In case abortion of `unshelve` is called via `hg abort` the `shelvedstate` needs to be loaded seperately. This has been ensured by `_loadunshelvedstate()` `hgabortunshelve()` is then registered as `abortfunc` for state detection API. Results are shown as tests. Differential Revision: https://phab.mercurial-scm.org/D6579

unshelve: changed Corruptedstate error msg from ui.warn to error.Abort This changes the message type of Corruptedstate error in case of `hg unshelve --abort` to error.Abort from warning message. This is done so as to avoid the return statement after the warning. Differential Revision: https://phab.mercurial-scm.org/D6636

mq: fix for merge detection methods Differential Revision: https://phab.mercurial-scm.org/D6548

py3: store _origdoc as str Since __doc__ is str, it seems natural that _origdoc also is. Differential Revision: https://phab.mercurial-scm.org/D6623

copies: follow copies across merge base without source file (issue6163) As in the previous patch, consider these two histories: @ 4 'rename x to y' | o 3 'add x again' | o 2 'remove x' | | o 1 'modify x' |/ o 0 'add x' @ 4 'rename x to y' | o 3 'add x again' | | o 2 'modify x' | | | o 1 'add x' |/ o 0 'base' We trace copies from the 'modify x' commit to commit 4 by going via the merge base (commit 0). When tracing file 'y' (_tracefile()) in the first case, we immediately find the rename from 'x'. We check to see if 'x' exists in the merge base, which it does, so we consider it a valid copy. In the second case, 'x' does not exist in the merge base, so it's not considered a valid copy. As a workaround, this patch makes it so we also attempt the check in mergecopies's base commit (commit 1 in the second case). That feels pretty ugly to me, but I don't have any better ideas. Note that we actually also check not only that the filename matches, but also that the file's nodeid matches. I don't know why we do that, but it was like that already before I rewrote mergecopies(). That means that the rebase will still fail in cases like this (again, it already failed before my rewrite): @ 4 'rename x to y' | o 3 'add x again with content X2' | o 2 'remove x' | | o 1 'modify x to content X2' |/ o 1 'modify x to content X1' | o 0 'add x with content X0' Differential Revision: https://phab.mercurial-scm.org/D6604

copies: filter invalid copies only at end of pathcopies() (issue6163) copies._filter() filters out copies whose source file does not exist in the start commit or whose target file does not exist in the end commit. We do that after chaining copies with dirstate copies or backward renames from another branch. We also do at the end of the changeset-centric copy tracing. The filtering means that we will remove copies to/from files that did not exist in some intermediate commit. That is inconsistent with what we do if a file has been deleted and then re-added (we allow updating across that). Copying the two first examples from issue6163: @ 4 'rename x to y' | o 3 'add x again' | o 2 'remove x' | | o 1 'modify x' |/ o 0 'add x' @ 4 'rename x to y' | o 3 'add x again' | | o 2 'modify x' | | | o 1 'add x' |/ o 0 'base' When doing `hg rebase -r 1 -d 4` in the first case, it succeeds, but `hg rebase -r 2 -d 4` in the second case does not. That's because we chain and filter via commit 0, which does not have file 'x' in the second case. IMO, that's clearly inconsistent. So this patch removes the filtering step so it only happens at the end. If a file was temporarily removed, whether via a merge base or not, it will now still be considered the same file. That fixes issue6163 for the changeset-centric case. Differential Revision: https://phab.mercurial-scm.org/D6603

copies: inline _chainandfilter() to prepare for next patch Differential Revision: https://phab.mercurial-scm.org/D6602

copies: remove most early returns from pathcopies() and _forwardcopies() I want to split up _chainandfilter() more so the call to _filter() consistently happens at the end of pathcopies(). This prepares for that change. Differential Revision: https://phab.mercurial-scm.org/D6601

copies: move short-circuiting of dirstate copies out of _forwardcopies() I'd like to move the filtering of copies we do after chaining to the end of all chaining (in a single place in pathcopies()). One problem that came up when trying that was that we allow things like `hg cp -f <file> <existing file>` so the user can later amend that in. Filtering at the end would mean that we remove those copies. That would break `hg st -C`. This patch therefore moves the short-circuiting of dirstate copies into pathcopies() so we can more easily handle the dirstate-only case differently. I initially thought this might change some behavior when the user does `hg status --rev 'wdir()' --rev .` during an uncommitted merge, since _backwardrenames() would reverse the copies in that case. However, I couldn't come up with a test case where it made a difference. Differential Revision: https://phab.mercurial-scm.org/D6600

tests: add more tests of copy tracing with removed and re-added files We had a test where the destination of a copy was removed and then added back. This patch adds similar cases where the break in history instead happens to the source file. There are three versions of this: 1. The break happens before the rename. 2. The break happens on a branch parallel to the rename (where copy tracing is done via the merge base) 3. The source is added on each side of the merge base. The break in history is thus in the form of a deletion when going backwards to the merge base and the re-add happens on the other branch. I've also added calls to `hg graft` in these cases to show the breakage in issue 6163. Another factor in these cases is matching nodeid (checked in copies._tracefile()). I've made two copies each of the cases to show the impact of that. One of these is the same as a test in test-rename-merge1.t, so I also deleted that test from there. Some of these tests currently fail, where "fail" is based on my current thinking of how things should work. I had initially thought that we should be more strict about not tracing copies across commits where the file did not exist, but issue 6163 made me reconsider. The only test case here that behaved differently in 4.9 is the exact case reported in issue 6163. Differential Revision: https://phab.mercurial-scm.org/D6599

tests: split out tests for unrelated copy source/target into separate file I've realized only recently how many cases there are where a file is treated differently if it's considered "related" to another file (not deleted and re-added). I'll add more tests for some of these cases soon. Differential Revision: https://phab.mercurial-scm.org/D6598

subrepos: make last line of prompts <40 english chars (issue6158) Differential Revision: https://phab.mercurial-scm.org/D6572

largefiles: make last line of prompts <40 english chars (issue6158) Differential Revision: https://phab.mercurial-scm.org/D6571

rust-dirstate: add helper to iterate ancestor paths This is modeled after std::path::Path::ancestors(). find_dirs(b"") yields b"" because Mercurial's util.finddirs() works in that way, and the test case for DirsMultiset expects such behavior.

tests: update test-commit-interactive.t for no-execbit platforms These changes correspond with f802a75da585. Differential Revision: https://phab.mercurial-scm.org/D6624

abort: added support for histedit This patch adds the support for `histedit` in `hg abort` plan. As seperate `hgaborthistedit()` function is created to handle independent calls for abortion of `histedit`. This function is then registered as `abortfunc` for state detection API. hg abort in case of `histedit` also supports ` history-editing-backup` config option. Results are shown as tests. Differential Revision: https://phab.mercurial-scm.org/D6582

abort: added support for rebase This adds support of `rebase` to `hg abort` plan. An independent abort logic for `rebase` is created under `abortrebase()` function. For this a seperate `rebaseruntime` object is created under the function to handle an unfinished `rebasestate` and abort that using abort logic under `_prepareabortorcontinue`. Results of tests are shown. Differential Revision: https://phab.mercurial-scm.org/D6568

abort: added support for graft This adds support of `graft` to `hg abort` plan. The patch creates a seperate function `cmdutil.hgabortgraft` so that abort logic for graft can be called independently. This logic is registered to the statedetection API as `abortfunc`. Results are shown as tests. Differential Revision: https://phab.mercurial-scm.org/D6567

abort: added logic for of hg abort This is part of `GSoC19` project `Implement abort and continue commands`. This patch is part of the `abort plan`. This adds the basic logic for `hg abort`. This command aborts an multistep operation like graft, histedit, rebase, merge and unshelve if they are in an unfinished state. The first part of the logic is determining the unfinished operation from the state detection API under `statemod`. This API is extended to support `hg abort` by adding a method to register the abort logic as a function (here `abortfunc`). Once the unfinished operation is determined the registered logic is used to abort the command. The benefit of this kind of framework is that any new extension developed can support `hg abort` by registering the command and logic under statedetection API. `hg abort` currently supports `--dry-run/-n` flag only. It is used to dry run `hg abort` Further patches sequentially add support for `graft`, `rebase`, `unshelve`, `histedit` and `merge`. Differential Revision: https://phab.mercurial-scm.org/D6566

merge with stable

merge: disallow merge abort in case of an unfinished operation (issue6160) This patch disallows `hg merge --abort` in case an operation of higher precedence i.e unshelve, rebase, histedit are in unfinished states. This is done so as to avoid partial abort of these operations in case merge abort is called at an interrupted step. The patch adds a `cmdutil.getunfinishedstate` function which checks for operations under progress and returns a `statecheck` object for it. Differential Revision: https://phab.mercurial-scm.org/D6607

relnotes: document new range-select mechanism in crecord Differential Revision: https://phab.mercurial-scm.org/D6622

statecheck: updated docstrings related to afterresolvedstates Differential Revision: https://phab.mercurial-scm.org/D6606

extdata: avoid crashing inside subprocess when we get a revset parse error Differential Revision: https://phab.mercurial-scm.org/D6616

extdata: demonstrate bad behavior when a subprocess emits garbage Differential Revision: https://phab.mercurial-scm.org/D6615

py3: don't run source transformer on hgext3rd (extensions) It's unclear why the source transformer runs on hgext3rd. It's been like that since it was introduced in 1c22400db72d (mercurial: implement a source transforming module loader on Python 3, 2016-07-04), and that commit didn't say anything about it (but it says that it doesn't have "support [...] for extensions"). I find that the current handling of hgext3rd just makes it harder to convert extensions to Python 3. It makes you convert a bunch of strings passed to getattr() and kwargs[] to r'' that could otherwise have been left alone. It's also really confusing that the source transformer runs when you import the extension as "extensions.foo=", but not as "extension.foo=/some/path". I suppose there is small number of (very simple) extensions that would have worked without this patch that would now be broken. It seems okay to me to break those. Differential Revision: https://phab.mercurial-scm.org/D6614

crecord: provide 'X' as a range-select mechanism Differential Revision: https://phab.mercurial-scm.org/D6621

crecord: make KEY_ENTER usable in tests (by not updating UI) Differential Revision: https://phab.mercurial-scm.org/D6620

crecord: fix if -> elif when handling key presses This shouldn't actually change any behavior, I only noticed it since I started using KEY_UP in tests, and it was complaining when it got down to the ^L handler that initscr hadn't been called yet. Differential Revision: https://phab.mercurial-scm.org/D6619

crecord: add "x" alias for space, remove test-only "TOGGLE" alias Differential Revision: https://phab.mercurial-scm.org/D6618

crecord: stop using test-only "X" as alternative for "c" Differential Revision: https://phab.mercurial-scm.org/D6617

graft: moved abortgraft and readgraft to cmdutil This patch moves `abortgraft` and `readgraft` to `cmdutil`. Various callers are updated accordingly. This is done because these serve as ulitlity functions for command `graft` and so that new functions regarding graft can be built from them. Differential Revision: https://phab.mercurial-scm.org/D6608

cleanup: use named constants for second arg to .seek() Differential Revision: https://phab.mercurial-scm.org/D6556

patch: use a short, fixed-size message for last line of prompt (issue6158) See issue6158 and the previous commit for examples of what might go wrong if we have some combinations of readline version and terminal and need to wrap the line. Briefly: readline may not display the beginning of the last line of the prompt, or it may print over it with the end of the prompt, making it difficult for users to know what's going on. Differential Revision: https://phab.mercurial-scm.org/D6563

filemerge: make last line of prompts <40 english chars (issue6158) I've chosen <40 as the target so that other languages that may have a 2x blowup in character count can still have a chance to fit into an 80 column screen. Previously, we would show a prompt like: ``` keep (l)ocal [dest], take (o)ther [source], or leave (u)nresolved for some/potentially/really/long/path? ``` On at least some systems, if readline was in use then the last line of the prompt would be wrapped strangely if it couldn't fit entirely on one line. This strange wrapping may be just a carriage return without a line feed, overwriting the beginning of the line; example (100 columns wide, 65 character filename, and yes there's 10 spaces on the end, I assume this is to handle the user inputting longest word we provide as an option, "unresolved"): ``` ng/dir/name/that/does/not/work/well/with/readline/file.txt? ave (u)nresolved for some/lon ``` In some cases it may partially wrap onto the next line, but still be missing earlier parts in the line, such as below (60 columns wide, 65 character filename): ``` rev], or leave (u)nresolved for some/long/dir/name/that/do s/not/work/well/with/readline/file.txt? ``` With this fix, this looks like this on a 60 column screen: ``` tool vim_with_markers (for pattern some/long/dir/name/that/d oes/not/work/well/with/readline/file.txt) can't handle binar y tool meld can't handle binary tool vim_with_markers can't handle binary tool internal:merge3 can't handle binary tool merge can't handle binary no tool found to merge some/long/dir/name/that/does/not/work /well/with/readline/file.txt file 'some/long/dir/name/that/does/not/work/well/with/readli ne/file.txt' needs to be resolved. You can keep (l)ocal [working copy], take (o)ther [merge rev ], or leave (u)nresolved. What do you want to do? ``` Differential Revision: https://phab.mercurial-scm.org/D6562

tweakdefaults: make hg resolve require --re-merge flag to re-merge Pulkit suggested it in https://phab.mercurial-scm.org/D4379, and a discussion with Octobus people reminded me that people still use the error-prone default behavior of `hg resolve`. Differential Revision: https://phab.mercurial-scm.org/D6610

unshelve: rename _dounshelve() to dounshelve() This is a follow-up patch to 3de4f17f4824. Differential Revision: https://phab.mercurial-scm.org/D6605

rust: remove Deref in favor of explicit methods Differential Revision: https://phab.mercurial-scm.org/D6593

rust: simplify overly complicated expression Differential Revision: https://phab.mercurial-scm.org/D6592

rust: run rfmt on all hg-core/hg-cpython code Differential Revision: https://phab.mercurial-scm.org/D6591

changelog: fix handling of empty copy entries in changeset Before this patch, when an empty value was found in the changeset, we would get a ValueError, which would result in None being returned for addedfiles/removedfiles and p1copies/p2copies. That made 278dcb24e535 (copies: write empty entries in changeset when also writing to filelog, 2019-04-23) ineffective at helping the read path not look for copies in the filelogs. Differential Revision: https://phab.mercurial-scm.org/D6595

relnotes: document the new --force-close-branch flag Differential Revision: https://phab.mercurial-scm.org/D6590

py3: hack around inconsistency of type of name passed to DNSQuestion I don't like this patch but this is the easiest way I could fix it. There are some callers which pass name which is bytes, some pass name which is str. I just encode() that if that's str. This does makes test-paths.t pass, but I am not confident whether the whole of zeroconf will work on py3 or not. Differential Revision: https://phab.mercurial-scm.org/D6511

py3: add r'' prefixes and do ('%d' % int) instead of str(int) This addresses more failures related to zeroconf on py3. Differential Revision: https://phab.mercurial-scm.org/D6510

zeroconf: port to Python 3 Since we're using the source transformer on Python 3, calls into Zeroconf and return values from it are generally bytes. But various socket functions require str on Python 3. This commit contains enough changes to coerce test-paths.t into passing on Python 3. I suspect there are still a handful of bugs on Python 3. But the tests do pass. Differential Revision: https://phab.mercurial-scm.org/D5805

copies: return only path from _tracefile() since that's all caller needs Differential Revision: https://phab.mercurial-scm.org/D6587

extensions: add shelve to _builtin This is a follow-up patch to 3de4f17f4824. This adds `shelve` to `extensions._builtin` so that the shelve extension is silently ignored. Differential Revision: https://phab.mercurial-scm.org/D6589

merge with stable

automv: access status fields by name, not index Differential Revision: https://phab.mercurial-scm.org/D6586

automv: use public API for getting copies Differential Revision: https://phab.mercurial-scm.org/D6585

commit: add --force-close-branch flag to close a non-head changeset While closing branch from a changeset which is not a branch head current implementation abort this action in every case but, there can be the situations where the changeset is not a local head but could be a remote head. This patch adds the functionality to bypass the "abort: can only close branch heads" by introducing --force-close-branch flag. Test case changes demonstrate the new functionality added. Differential Revision: https://phab.mercurial-scm.org/D6490

shelve: move shelve extension to core Until now, `shelve` was bootstrapped as an extension. This patch adds `shelve` on core. Differential Revision: https://phab.mercurial-scm.org/D6553

shelve: remove rebase.clearstatus() This is a follow-up patch to c829749e7639. After this, shelve will be no longer dependent on rebase. This removes rebase.clearstatus() from shelve. Differential Revision: https://phab.mercurial-scm.org/D6584

shelve: removed redundant merge detection method Differential Revision: https://phab.mercurial-scm.org/D6547

rust-dirstate: call new "dirs" rust implementation from Python This is a simple module attribute replacement, will take precedence over the Python and C implementations. Differential Revision: https://phab.mercurial-scm.org/D6395

rust-dirstate: add "dirs" rust-cpython binding There is an obvious performance and memory issue with those bindings on larger repos as it copies and allocates everything at once, round-trip. Like in the previous patch series, this is only temporary and will only get better once we don't have large data structures going to and from Python. Differential Revision: https://phab.mercurial-scm.org/D6394

rust-dirstate: add "dirs" Rust implementation Following the work done in d1786c1d34fa and working towards the goal of a complete Rust implementation of the dirstate, this rewrites the `dirs` class. There is already a C implementation, which relies heavily on CPython hacks and protocol violations for performance, so I don't expect this to perform as well for now, as this is very straight-forward code. The immediate benefits are new high-level documentation and some unit tests. Differential Revision: https://phab.mercurial-scm.org/D6393

relnotes: added description about statemod._statecheck Differential Revision: https://phab.mercurial-scm.org/D6557

statecheck: shifted defaults to addunfinished() This shifts the definitions and defaults of `_statecheck()` class to `addunfinished()` registration method. Differential Revision: https://phab.mercurial-scm.org/D6583

statecheck: added support for cmdutil.afterresolvedstates This removes `afterresolvedstates` from `cmdutil` and adds support for it in `_statecheck` class. A new flag `continueflag` is added to the class to check whether an operation supports `--continue` option or not. Tests remain unchanged. Differential Revision: https://phab.mercurial-scm.org/D6551

statecheck: added support for STATES This removes `STATES` from `state.py` and adds support to `statecheck` class to handle its features. `getrepostate()` function is modified accordingly. This adds a method 'cmdutil.addunfinished()' for appending to the unfinishedstate list so as to keep 'merge' and 'bisect' at the last. This also makes two separate message formats for `checkunfinished()` and `getrepostate()` as there were previously present. Results of test changed are shown. Differential Revision: https://phab.mercurial-scm.org/D6503

state: moved cmdutil.STATES and utilities to state.py This commit moves `cmdutil.STATES` and adjoining functions to `state.py`. The existing users are updated accordingly. Tests remain unchanged. Differential Revision: https://phab.mercurial-scm.org/D6502

state: created new class statecheck to handle unfinishedstates For the purpose of handling states for various multistep operations like `hg graft`, `hg histedit`, `hg bisect` et al a new class called statecheck is created .This will help in having a unified approach towards these commands and handle them with ease. The class takes in 4 basic arguments which include the name of the command, the name of the state file associated with it , clearable flag , allowcommit flag. This also also adds the support of`checkunfinished()` and `clearunfinished()` to the class. Tests remain unchanged. Differential Revision: https://phab.mercurial-scm.org/D6501

states: moved cmdutil.unfinishedstates to state.py This moves `cmdutil.unfinishedstates`, `checkunfinished()`,`clearunfinished()` to `state.py`. the already existing users of this module are updated accordingly. Test results remain unchanged. Differential Revision: https://phab.mercurial-scm.org/D6484

rebase: fix in-memory rebasing of copy of empty file Classic Python mistake of unintentionally treating None and empty string the same. Differential Revision: https://phab.mercurial-scm.org/D6570

tests: demonstrate broken in-memory rebase of copy to empty file Differential Revision: https://phab.mercurial-scm.org/D6569

zsh: enable completion support for chg as well When verifying this change, you may need to clear/rebuild the completion cache; I did this by deleting the ~/.zcompdump file and then starting a new shell. Differential Revision: https://phab.mercurial-scm.org/D6574

py3: make catapult usable from the test runner in py3 Differential Revision: https://phab.mercurial-scm.org/D6577

py3: use integer division for the value passed to xrange Differential Revision: https://phab.mercurial-scm.org/D6576

pycompat: make fewer assumptions about sys.executable There are many Python "bundlers" which create an archive to run a Python binary from, and they may not set sys.executable at all - handle that case properly, especially to run tests. Differential Revision: https://phab.mercurial-scm.org/D6575

update: fix spurious unclean status bug shown by previous commit The crux of the problem is: - the dirstate is corrupted (the sizes/dates are assigned to the wrong files) - because when worker.worker is used with a return value (batchget in merge.py here), the return value when worker.worker effectively parallelizes is permuted - this is because worker.worker's partition of input and combination of output values are not inverses of one another: it split [1,2,3,4,5,6] into [[1,3,5],[2,4,6]], but combines that into [1,3,5,2,4,6]. Given that worker.worker doesn't call its function argument on contiguous chunks on the input arguments, sticking with lists means we'd need to know the relation between the inputs of worker.worker function argument (for instance, requiring that every input element is mapped to exactly one output element). It seems better to instead switch return values to dicts, which can combined reliably with a straighforward restriction. Differential Revision: https://phab.mercurial-scm.org/D6581

tests: show bug in update introduced in 87a34c767384 As reported by Martin at https://phab.mercurial-scm.org/D6475. Differential Revision: https://phab.mercurial-scm.org/D6580

copies: document how 'copies' dict instances are reused We avoid copying these instances as much as we can, so it's not obvious what's safe to do with them. This patch tries to explain what is safe and what is not. Differential Revision: https://phab.mercurial-scm.org/D6578

copies: simplify merging of copy dicts on merge commits After we removed some filtering in 35d674a3d5db (copies: don't filter out copy targets created on other side of merge commit, 2019-04-18), we will always include all entries from "copies1", so we can simplify the code based on that. Differential Revision: https://phab.mercurial-scm.org/D6561

copies: remove a redundant matcher filtering in _changesetforwardcopies() We filter before pushing items on the queue, so we don't need to filter after popping. Differential Revision: https://phab.mercurial-scm.org/D6560

copies: delete obsolete comment in _changesetforwardcopies() IIRC, the comment applied to the filtering we did before 35d674a3d5db (copies: don't filter out copy targets created on other side of merge commit, 2019-04-18). Differential Revision: https://phab.mercurial-scm.org/D6559

merge with stable

copies: avoid reusing the same variable for two different copy dicts "childcopies" is initally the copies the current changeset to one of its children and then we reassign it with the copies from the start of the chain to the child. Let's use different names for these two things. Differential Revision: https://phab.mercurial-scm.org/D6564

drawdag: don't crash when writing copy info to changesets When writing copies to the changeset, localrepo.commitctx() will call ctx.p1copies() and ctx.p2copies(). These crashed on simplecommitctx because they ended up trying to access the manifest. drawdag doesn't support copies at all, so we can simply override the methods to return empty dicts. Differential Revision: https://phab.mercurial-scm.org/D6565

merge with stable

log: pass getcopies() function instead of getrenamed() to displayer (API) This reduces the duplication between the two displayer functions (and between them and scmutil.getcopiesfn()). It's still more code than two patches ago, but there's less duplication. Differential Revision: https://phab.mercurial-scm.org/D6546

copies: create helper for getting all copies for changeset There are a few places where we get all the copies for a changeset (at least the {file_copies} template and in two places in `hg log --copies` code). These places currently call scmutil.getrenamedfn() to get a caching "getrenamed" function. They all use it in a similar way. We will be able to reuse more code by having a function for getting all the copies for a changeset. This patch introduces such a function. It uses it in the {file_copies} template to show that it works. It relies on the existing scmutil.getrenamedfn() for caching in the filelog-centric case. Differential Revision: https://phab.mercurial-scm.org/D6545

logcmdutil: also check for copies in null revision and working copy It's safe (and fast) to look for copies in the null revision, and it's incorrect not to look for them in the working copy, so let's look in both places. Differential Revision: https://phab.mercurial-scm.org/D6544

tests: demonstrate missing copy information in working copy with graphlog Differential Revision: https://phab.mercurial-scm.org/D6543

remotefilelog: handle copies in changesets in getrenamedfn() override E.g. the {file_copies} template keyword didn't work with copies in changesets before this patch because remotefilelog overrides the getrenamedfn() and didn't handle the changeset-centric case. Differential Revision: https://phab.mercurial-scm.org/D6542

remotefilelog: check if RFL is enabled in getrenamedfn() override In 8a0e03f7baf4 (remotefilelog: move most setup from onetimesetup() to uisetup(), 2019-05-01), I said: All the wrappers moved in this patch check if remotefilelog is enabled before they change behavior, so it's safe to always wrap. That was clearly a lie, because getrenamedfn() didn't. That made e.g. `hg log -T {file_copies}` unbearably slow. This patch fixes that. Differential Revision: https://phab.mercurial-scm.org/D6541

relnotes: document template support for `hg root` Differential Revision: https://phab.mercurial-scm.org/D6540

remotefilelog: tell runbgcommand to not block on child process startup These two invocations will always find a binary because they're re-running hg. As a result, we can skip waiting for the subprocess to start running and save a little bit of wall-time. Differential Revision: https://phab.mercurial-scm.org/D6539

procutil: allow callers of runbgcommand to assume the process starts Experimentally starting the subprocess can take as much as 40ms, and for some of our use cases that's frivolous: we know the binary will start, and if it doesn't we'd only ever ignore it and continue anyway. This lets those use cases be faster. Differential Revision: https://phab.mercurial-scm.org/D6537

shallowrepo: remove backwards compat code that predates in-tree remotefilelog Differential Revision: https://phab.mercurial-scm.org/D6538

commit: make the error message more specific while aborting branch closing Differential Revision: https://phab.mercurial-scm.org/D6493

commit: add a check if it is trying to close an already closed branch head It would check if the revision we are going to close is already a closed branch head and print the error message accordingly. Differential Revision: https://phab.mercurial-scm.org/D6491

strip: move checksubstate() to mq (its only caller) Differential Revision: https://phab.mercurial-scm.org/D6536

strip: use bailifchanged() instead of reimplementing it This also means that we get the standard error messages (see changed test cases). Differential Revision: https://phab.mercurial-scm.org/D6535

strip: remove unused excsuffix argument from checklocalchanges() It was only used by mq, and mq now has its own copy of the function. Differential Revision: https://phab.mercurial-scm.org/D6534

mq: remove dependency on strip's checklocalchanges() Some of the functionality in strip.checklocalchanges() was only used by mq, so let's move it to mq so we can clean up strip. Differential Revision: https://phab.mercurial-scm.org/D6533

copies: avoid calling matcher if matcher.always() When storing copy information in the changesets (experimental.copies.read-from=changeset-only), this patch speeds up hg debugpathcopies FENNEC_58_0_2_BUILD1 FIREFOX_59_0b8_BUILD2 from 5.9s to 4.7s. At the start of this series (b162229e), that command took 18min. Differential Revision: https://phab.mercurial-scm.org/D6422

copies: avoid unnecessary copying of copy dict When storing copy information in the changesets, this patch speeds up hg debugpathcopies FENNEC_58_0_2_BUILD1 FIREFOX_59_0b8_BUILD2 from 11s to 5.9s. That command takes 6.2s when storing copy information in filelogs. Differential Revision: https://phab.mercurial-scm.org/D6421

copies: don't filter out copy targets created on other side of merge commit If file X is copied to Y on one side of merge and the other side creates Y (no copy), we would not mark that as copy. In the changeset-centric pathcopies() version, that was done by checking if the copy target existed on the other branch. Even though merge commits are pretty uncommon, it still turned out to be too expensive to load the manifest of the parents of merge commits. In a repo of mozilla-unified converted to storing copies in changesets, about 2m30s of `hg debugpathcopies FIREFOX_BETA_59_END FIREFOX_BETA_60_BASE` is spent on this check of merge commits. I tried to think of a way of storing more information in the changesets in order to cheaply detect these cases, but I couldn't think of a solution. So this patch simply removes those checks. For reference, these extra copies are reported from the aforementioned command after this patch: browser/base/content/sanitize.js -> browser/modules/Sanitizer.jsm testing/mozbase/mozprocess/tests/process_normal_finish_python.ini -> testing/mozbase/mozprocess/tests/process_normal_finish.ini testing/mozbase/mozprocess/tests/process_waittimeout_python.ini -> testing/mozbase/mozprocess/tests/process_waittimeout.ini testing/mozbase/mozprocess/tests/process_waittimeout_10s_python.ini -> testing/mozbase/mozprocess/tests/process_waittimeout_10s.ini Since these copies were created on one side of some merge, it still seems reasonable to include them, so I'm not even sure it's worse than filelog pathcopies(), just different. Differential Revision: https://phab.mercurial-scm.org/D6420

copies: do full filtering at end of _changesetforwardcopies() As mentioned earlier, pathcopies() is very slow when copies are stored in the changeset. Most of the cost comes from calling _chain() for every changeset, which is slow because it needs to read manifests. It needs to read manifests to be able to filter out copies that are were created in one commit and then deleted. (It also filters out copies that were created from a file that didn't exist in the starting revision, but that's a fixed revision across calls to _chain(), so it's much cheaper.) This patch changes from _chainandfilter() to just _chain() in the main loop in _changesetforwardcopies(). It instead removes copies that have subsequently been removed by using ctx.filesremoved(). We thus rely on that to be fast. It timed this command in mozilla-unified: hg debugpathcopies FIREFOX_59_0b3_BUILD2 FIREFOX_BETA_59_END It took 18s before and 1.1s after. It's still faster when copy information is stored in filelogs: 0.70s. It also still gets slow when there are merge commits involved, because we read manifests there too. We'll deal with that later. Differential Revision: https://phab.mercurial-scm.org/D6419

rust-filepatterns: add comment about Windows path handling As I replied to the Phabricator message, this is wrong. And I even suspect it wouldn't compile because of multiple type mismatches. I think, in Rust where type system is rock solid, we can live with UTF-8 strings except for the bottom storage layer and the top UI/command layer. We'll still have to get around undecodable characters not to be lost, but I think it's okay to drop such filenames from match result if they don't match in UTF-8 world, not in Latin-1 world.

rust-filepatterns: silence warning of non_upper_case_globals

rust: update Cargo.lock to include @generated comment cargo 1.34.0 of Debian sid inserts this comment, and I'm tired of reverting the change every time I do make local. https://github.com/rust-lang/cargo/commit/bd0e4a08471b8bc7957829b4fd294b8985d4fa2d

merge with stable

lfs: correct an error in the TODO file

cat: don't prefetch files unless the output requires it It's a waste to cache lfs blobs when cat'ing the raw data at best, but a hassle debugging when the blob is missing. I'm not sure if there are other commands that have '{data}' for output, and if there's a general way to prefetch on that keyword. It's interesting that the verbose output seems to leak into the JSON output, but that seems like an existing bug.

tracing: add support for emitting counters Differential Revision: https://phab.mercurial-scm.org/D6526

tracing: extract tracing-active logic to separate function I'm about to add support for counters, and want to avoid duplicating this logic. Differential Revision: https://phab.mercurial-scm.org/D6525

catapipe: add support for COUNTER events Differential Revision: https://phab.mercurial-scm.org/D6524

demandimport: add tracing coverage for Python 3 This makes things feel a little less mysterious when modules are being imported. Differential Revision: https://phab.mercurial-scm.org/D6523

export: don't prefetch *all* files in manifest `hg export` only shows changed files, not all files, but we still prefetched all files in cmdutil.export(). The same is true for the other commands calling cmdutil.exportfile(). That meant that `hg export` with remotefilelog (or lfs, I assume) could take much longer than expected because it would download all the files in the repo. Differential Revision: https://phab.mercurial-scm.org/D6532

remotefilelog: remove obsolete filtering of treemanifest directories I think this has been obsolete since 2cf18f46a1ce (narrow: only walk files within narrowspec also for committed revisions, 2018-09-28). Differential Revision: https://phab.mercurial-scm.org/D6531

py3: add test-dirstate-race2.t to list of passing tests This test was added new recently. The py3 buildbot found that it passes, so let's add it to the list of passing tests. Differential Revision: https://phab.mercurial-scm.org/D6530

strip: during merge allow strip only when -f is used This ensures to abort strip to `hg strip` when we have a merge in progress and allow it only when a `--force` flag is used. Differential Revision: https://phab.mercurial-scm.org/D6529

deltas: set estimated compression upper bound to "3x" instead of "10x" In pratice, we very rarely observer compression better than "3x" on manifest deltas. Having a more aggressive estimate significantly helps our pathological use case on a private repository. Here are a comparison of timings using different upper bound. Estimated compression | ø | ×10 | ×5 | ×3 | timing | 14.11 | 2.61 | 1.96 | 1.53 | We also tested the impact of this series on an array of public repositories. This shown no impact in either size nor timing. Full data set below for those interested. Size ---- Regarding size, not significant impact have been noticed on neither public nor private repositories. Here are the number we gathered on public repositories: zlib/upperbound | no | 10x | 5x | 3x mercurial | 5 875 730 | 5 875 730 | 5 875 730 | 5 875 730 pypy | 27 782 913 | 27 782 913 | 27 782 913 | 27 782 913 netbeans | 159 161 207 | 159 161 207 | 159 161 207 | 159 959 879 (+0.5%) mozilla-central | 323 841 642 | 323 841 642 | 323 841 642 | 319 867 519 (-2.5%) mozilla-try | 746 649 123 | 746 649 123 | 746 649 123 | 741 155 568 (-0.7%) private-repo | 1 485 287 294 | 1 485 287 294 | 1 485 287 294 | 1 409 248 382 (-5.1%) zstd/upperbound | no | 10x | 5x | 3x mercurial | 5 895 206 | 5 895 206 | 5 895 206 | 5 895 206 pypy | 28 689 230 | 28 689 230 | 28 689 230 | 28 689 230 netbeans | 157 636 387 | 157 636 387 | 157 636 387 | 159 692 678 (+1.3%) mozilla-central | 317 650 281 | 317 650 281 | 317 650 281 | 319 613 603 (+0.6%) mozilla-try | 737 555 275 | 737 555 275 | 737 555 275 | 738 079 473 (+0.1%) private-repo | 1 352 362 982 | 1 352 362 982 | 1 346 961 880 | 1 361 327 384 (+0.7%) Speed ------ Timing gathered using `hg perfrevlogwrite -m`. Value are in seconds. mercurial zlib | no | 10x | 5x | 3x | total | 65.551783 | 65.388887 | 65.260658 | 65.321199 | max | 0.034544 | 0.034571 | 0.034659 | 0.034521 | 99.99% | 0.034544 | 0.034571 | 0.034659 | 0.034521 | zstd | no | 10x | 5x | 3x | total | 49.118449 | 49.054062 | 48.753588 | 48.740230 | max | 0.009338 | 0.009239 | 0.009202 | 0.009178 | 99.99% | 0.007618 | 0.007639 | 0.007626 | 0.007621 | pypy zlib | no | 10x | 5x | 3x | total | 560.865984 | 558.983817 | 559.083815 | 559.349152 | max | 0.219614 | 0.215922 | 0.218112 | 0.218107 | 99.99% | 0.219614 | 0.215922 | 0.218112 | 0.218107 | zstd | no | 10x | 5x | 3x | total | 349.393280 | 347.395819 | 347.185407 | 345.643985 | max | 0.084143 | 0.083536 | 0.081834 | 0.082178 | 99.99% | 0.039445 | 0.039639 | 0.039612 | 0.039175 | netbeans zlib | no | 10x | 5x | 3x | total | 33103.327727 | 33314.932260 | 33211.745233 | 33345.891778 | max | 2.666852 | 2.672059 | 2.662453 | 2.662936 | 99.99% | 2.058772 | 2.070429 | 2.069569 | 2.064653 | zstd | no | 10x | 5x | 3x | total | 20112.102708 | 20095.879719 | 20083.390300 | 20123.221859 | max | 2.063482 | 2.062851 | 2.065229 | 2.060147 | 99.99% | 1.146647 | 1.143794 | 1.142933 | 1.146529 | mozilla zlib | no | 10x | 5x | 3x | total | 41374.102138 | 41418.816773 | 41381.956370 | 41334.280732 | max | 3.383474 | 3.387400 | 3.405711 | 3.387316 | 99.99% | 1.006755 | 1.005954 | 1.007700 | 1.007373 | zstd | no | 10x | 5x | 3x | total | 24689.691520 | 24643.939662 | 24664.630027 | 24664.512714 | max | 1.460822 | 1.449640 | 1.439747 | 1.465304 | 99.99% | 0.527111 | 0.527377 | 0.527807 | 0.527226 |

deltas: skip if projected compressed size is bigger than previous snapshot If we have a delta, we check constraints against a lower bound estimate of the resulting compressed delta. We then checks this projected size against the `size(snapshotⁿ) > size(snapshotⁿ⁺¹)` constraint. This allows to exclude potential base candidates before doing any expensive computation. This only apply to the intermediate-snapshot case since this constraint only apply to them. For some pathological cases of a private repository this step provide a further performance boost (timing from `hg perfrevlogwrite`): before: 3.010646 seconds after: 2.609307 seconds

deltas: skip if projected compressed size does not match text size constraint If we have a delta, we check constraints against a lower bound estimate of the resulting compressed delta. We then checks this projected size against the ½ⁿ size constraints. This allows to exclude potential base candidates before doing any expensive computation. This only apply to the intermediate-snapshot case since this constraint only apply to them. For some pathological cases of a private repository this step provide a further performance boost (timing from `hg perfrevlogwrite`): before: 3.145906 seconds after: 3.010646 seconds

deltas: accept and skip None return for delta info They are some extra computation that will shortcut the delta compression if the delta seems hopeless, returning None.

delta: move some delta chain related computation earlier in deltainfo They are some more optimization change that will make use of this in the function. So we retrieve the data earlier.

deltas: skip if projected delta size is bigger than previous snapshot Before computing any delta, we get a basic estimation of the delta size we can expect and the resulted compressed value. We then checks this projected size against the `size(snapshotⁿ) > size(snapshotⁿ⁺¹)` constraint. This allows to exclude potential base candidates before doing any expensive computation. This only apply to the intermediate-snapshot case since this constraint only apply to them. For some pathological cases of a private repository this step provide a significant performance boost (timing from `hg perfrevlogwrite`): before: 14.115908 seconds after: 3.145906 seconds

deltas: skip if projected delta size does not match text size constraint Before computing any delta, we get a basic estimation of the delta size we can expect and the resulted compressed value. We then checks this projected size against the ½ⁿ size constraints. This allows to exclude potential base candidates before doing any expensive computation. This only apply to the intermediate-snapshot case since this constraint only apply to them. In practice we only perform this new checks for the manifestlog. Manifest log combine two property: it is likely to have delta chain issue and its diffing/compression is fairly predictable. The initial author of this changeset is Valentin Gatien-Baron providing the initial idea and initial testing, Pierre-Yves David later consolidated the code in the right location and run more extensive testing.

revlog: add the option to track the expected compression upper bound There are various optimization we can do if we can estimate the size of delta before actually spending CPU compressing them. So we add a attributed dedicated to tracking that. We only use it on Manifest because (1) it structure is quite stable across all Mercurial repository so its compression ratio is fairly universal. This is the revlog with most extreme delta (cf the sparse-revlog optimization). This will be put to use in later changesets. Right now the compression upper bound is set to 10. This is a fairly conservative value (observed value is more around 3), but I prefer to be safe while introducing the optimization principles. We can tune the optimization threshold later.

perf: clarify some of the custom behavior of `perfrevlogwrite` This reduce the chance of developers being surprised by special cases.

perf: fix perfrevlogwrite --count documentation The help text was copy pasted from the previous option.

rust: switched to 'cargo rustc' in setup.py This is more flexible in the passing of additional flags, also what setuptools_rust does, giving less uncertainty about non-Linux platforms.

rust-cpython: fix build for MacOSX MacOSX needs special link flags. Quoting the README of rust-cpython: create a `.cargo/config` with the following content: ``` [target.x86_64-apple-darwin] rustflags = [ "-C", "link-arg=-undefined", "-C", "link-arg=dynamic_lookup", ] ``` This is tested with Python 2.7 (Anaconda install) and Python 3 (Homebrew install)

rust-cpython: management of shared libray suffix Before this changeset, the shared library objects suffixes were both (rustc output and Python input) hardcoded to '.so', which is wrong for Python3 and non Linux targets.

merge: fix race that could cause wrong size in dirstate The problem is that hg merge/update/etc work the following way: 1. figure out what files to update 2. apply the update to disk 3. apply the update to in-memory dirstate 4. write dirstate where step3 looks at the filesystem and assumes it sees the result of step2. If a file is changed between step2 and step3, step3 will record incorrect information in the dirstate. I avoid this by passing the size step3 needs directly from step2, for the common path (not implemented for change/delete conflicts for instance). I didn't fix the same race for the exec bit for now, because it's less likely to be problematic and I had trouble due to the fact that the dirstate stores the permissions differently from the manifest (st_mode vs '' 'l' 'x'), in combination with tests that pretend that symlinks are not supported. However, I moved the lstat from step3 to step2, which should tighten the race window markedly, both for the exec bit and for the mtime. Differential Revision: https://phab.mercurial-scm.org/D6475

worker: support parallelization of functions with return values Currently worker supports running functions that return a progress iterator. Generalize it to handle function that return a progress iterator then a return value. It's unused in this commit, but will be used in the next one. Differential Revision: https://phab.mercurial-scm.org/D6515

tests: show how the dirstate can end up containing wrong information which can result in bad status output. Concretely, this seems to be easily triggered by having a build system watching the filesystem for changes, and rebuilding files that are both tracked and generated while an update is happening. Differential Revision: https://phab.mercurial-scm.org/D6474

rust: new rust options in setup.py The --rust global option turns on usage (and by default compilation) of the rust-cpython based mercurial.rustext. Similarly to what's previously done for zstd, there is a --no-rust option for the build_ext subcommand in order not to build mercurial.rustext, allowing for an OS distribution to prebuild it. The HGWITHRUSTEXT environment variable is still honored, and has the same effect as before, but now it works mostly by making the --rust global option defaulting to True, with some special cases for the direct-ffi case (see more about that below) Coincidentally, the --rust flag can also be passed from the make commands, like actually all global options, in the PURE variable make local PURE=--rust This feels inappropriate, though, and we should follow up with a proper make variable for that case. Although the direct-ffi bindings aren't directly useful any more, we keep them at this stage because - they provide a short prototyping path for experiments in which a C extension module has to call into a Rust extension. The proper way of doing that would be to use capsules, and it's best to wait for our pull request onto rust-cpython for that: https://github.com/dgrunwald/rust-cpython/pull/169 - Build support for capsules defined in Rust will probably need to reuse some of what's currently in use for direct-ffi.

rust: using policy.importrust from Python callers This commit converts all current Python callers of mercurial.rustext to the new policy.importrust system. After this point, going through policy.importrust or policy.importmod (in some more distant future) is mandatory for callers of Rust code outside of Python tests. We felt it to be appropriate to keep Rust-specific tests run inconditionally if the Rust extensions are present.

rust: module policy with importrust We introduce two rust+c module policies and a new `policy.importrust()` that makes use of them. This simple approach provides runtime switching of implementations, which is crucial for the performance measurements such as those Octobus does with ASV. It can also be useful for bug analysis. It also has the advantage of making conditionals in Rust callers more uniform, in particular abstracting over specifics like `demandimport` At this point, the build stays unchanged, with the rust-cpython based `rustext` module being built if HGWITHRUSTEXT=cpython. More transparency for the callers, i.e., just using `policy.importmod` would be a much longer term and riskier effort for the following reasons: 1. It would require to define common module boundaries for the three or four cases (pure, c, rust+ext, cffi) and that is premature with the Rust extension currently under heavy development in areas that are outside the scope of the C extensions. 2. It would imply internal API changes that are not currently wished, as the case of ancestors demonstrates. 3. The lack of data or property-like attributes (tp_member and tp_getset) in current `rust-cpython` makes it impossible to achieve direct transparent replacement of pure Python classes by Rust extension code, meaning that the caller sometimes has to be able to make adjustments or provide additional wrapping.

help: add help entry for internals.mergestate This patch adds an entry for `internals.mergestate` as suggested by @marmoute. Most of the help text is taken from `merge.mergestate`. Differential Revision: https://phab.mercurial-scm.org/D6448 Differential Revision: https://phab.mercurial-scm.org/D6528

phabricator: use parents.set to always set dependencies Now that Mercurial's Phabricator instance has been updated to a version that supports the parents.set transaction on revision.edit we can use that to set dependency relationships in patch stacks instead of abusing the summary. This has the advantage that we can use it on every `phabsend` so commit reordering is picked up without spamming changes like abusing the summary would, and using parents.set will clear previous parents unlike parents.add. Differential Revision: https://phab.mercurial-scm.org/D6514

help: remove repeated word in 'hg help rebase' Specifically, the second 'with' in 'with which to merge with'. Differential Revision: https://phab.mercurial-scm.org/D6483

rebase: tweak description of inmemory working even w/ dirty working dir One of our users was confused because they read this, and then attempted to run `hg rebase` with a dirty working directory, and it still complained. The reason was that they were attempting to rebase the commit they currently had checked out, which (at least with evolve workflows enabled) involves updating the working directory to be based on the newly rebased commit. Differential Revision: https://phab.mercurial-scm.org/D6507

revlog: speed up isancestor Currently, it is implemented on top of commonancestorsheads. Implement it on top of reachableroots instead, as reachableroots could stop walking the graph much sooner than commonancestorsheads. Measuring repo.changelog.isancestorrev on two revisions in a private repository: before: ! wall 0.005175 comb 0.010000 user 0.010000 sys 0.000000 (best of 550) after : ! wall 0.000072 comb 0.000000 user 0.000000 sys 0.000000 (best of 36199) When hg does this kind of operations 1500 times in pull -> bookmarks.comparebookmarks -> bookmarks.validdest, that's 11s that drop from the --profile output. Differential Revision: https://phab.mercurial-scm.org/D6506

dagop: fix documentation of reachableroots The previous revset couldn't be correct as it is symmetric in <roots> and <heads>, but reachableroots has no such symmetry. It makes a difference with for instance reachableroots(2, 3) where 2 and 3 are both children of 1. Differential Revision: https://phab.mercurial-scm.org/D6505

phabricator: add --blocker argument to phabsend to specify blocking reviewers The way to signal to Conduit that a reviewer is considered blocking is just to wrap their PHID in "blocking()" when including it in the list of PHIDs passed to `reviewers.add`. arc doesn't have a --blocker, instead one is supposed to append a '!' to the end of reviewer names (I think reviewers are usually added in an editor rather than the command line, where '!'s can be more hazardous). moz-phab (Mozilla's arcanist wrapper) does have a --blocker argument, and being explicit like this is also more discoverable. Even `arc diff`'s help doesn't seem to mention the reviewer! syntax. Differential Revision: https://phab.mercurial-scm.org/D6512

phabricator: auto-sanitise API tokens and HTTP cookies from VCR recordings Currently when making VCR recordings one needs to manually sanitise sensitive credentials before committing and submitting them as part of tests. It is easy to imagine this being accidentally missed one time by a fallible human and said credentials being leaked. It is also possible that it wouldn't be noticed to alert the user to the leak since the recording files are so large and practically unreviewable. Thus do so automatically, so the only place that needs checking is in the test-phabricator.t file. Differential Revision: https://phab.mercurial-scm.org/D6513

py3: use .startswith() instead of bytes[0] Doing bytes[0] will return the ascii value of that position which breaks comparison with a bytechar. This makes test-absorb.t work again on py3. Differential Revision: https://phab.mercurial-scm.org/D6508

revset: fix merge() to fall back to changectx API if wdir specified I have a code which basically runs "0:wdir() & <user-revset>", and it crashed if merge() were passed in.

revset: use nullrev constant in merge()

mixedrepostorecache: fix a silly redundant updating of set Differential Revision: https://phab.mercurial-scm.org/D6470

rust-regex: fix shortcut for exact matches The current shortcut for rootglobs that can be simplified to exact matches does not work, it instead treats the pattern as a regex, which is not the same thing. This changes fixes the behavior and introduces a test for this behavior. Differential Revision: https://phab.mercurial-scm.org/D6489

rust-filepatterns: use bytes instead of String In my initial patch, I introduced an unnecessary hard constraint on UTF-8 filenames and patterns which I forgot to remove. Although the performance penalty for using String might be negligible, we don't want to break compatibility with non-UTF-8 encodings for no reason. Moreover, this change allows for a cleaner Rust core API. This patch introduces a new utils module that is used with this fix. Finally, PatternError was not put inside the Python module generated by Rust, which would have raised a NameError. Differential Revision: https://phab.mercurial-scm.org/D6485

doc: fix description of "predecessors" to match reality Differential Revision: https://phab.mercurial-scm.org/D6467

phabricator: make `hg debugcallconduit` work outside a hg repo I am trying to write some automations around phabricator and having debugcallconduit work outside a hg repo will be nice! Marking command as optionalrepo instead of norepo because we might to load repo/.hg/hgrc. Differential Revision: https://phab.mercurial-scm.org/D6499

phabricator: pass ui instead of repo to callconduit This will help us make `hg debugcallconduit` work outside a hg repo as next patch will mark that command as no repo. Differential Revision: https://phab.mercurial-scm.org/D6498

phabricator: pass ui into readurltoken instead of passing repo The goal of this series is to make `hg debugcallconduit` work outside of a hg repo. This patch, removes requirement of repo object from readurltoken as we only need ui there. It also updates the callers to pass in ui instead of repo. Differential Revision: https://phab.mercurial-scm.org/D6497

py3: add test-contrib-emacs.t to passing tests list I installed emacs on the server running buildbot and the test started passing on Python 3. Lets add it to the list of passing test. Differential Revision: https://phab.mercurial-scm.org/D6500

phabricator: add commenting to phabsend for new/updated Diffs Especially useful when sending updates to existing Revisions so one can specify the sort of changes e.g. "Address review comments" or "Rebase to tip" If the diff content hasn't changed then it only needs a metadata update which doesn't show in the Phabricator updates UI, so don't add a comment that will. Differential Revision: https://phab.mercurial-scm.org/D6496