automation: upgrade to latest packages in requirements.txt Let's stay modern. Differential Revision: https://phab.mercurial-scm.org/D6785

localrepo: push manifestlog and changelog construction code into store This feels substantially more appropriate, as the store is actually the layer with knowledge of how to handle this storage. I didn't move the caching decorators for now because that's going to require some more involved work, and this unblocks my current experimentation. Differential Revision: https://phab.mercurial-scm.org/D6732

notify: add option for deterministic message-id generation Copied from email by durin42: > Pierre-Yves asked offline why I asked a new option for this and why it > is not the default. Message-Id is supposed to be unique world-wide and > while it is desirable to have a deterministic mechanism for them for > creating follow-up emails, it needs organisational control for ensuring > the uniqueness. Differential Revision: https://phab.mercurial-scm.org/D6824

uncommit: add options to update to the current user or current date These are also from the evolve extension's version of uncommit. I tried adding validation that both forms of user or date can't be specified at the same time, but that fails because these show up in `opts` with a None value whether or not the option was given on the command line. Presumably that means the conditional in `resolvecommitoptions` could be simplified. But this is how both evolve and MQ handle it. Differential Revision: https://phab.mercurial-scm.org/D6828

uncommit: add support to modify the commit message and date Currently, the evolve extension's version of this command supports it. Differential Revision: https://phab.mercurial-scm.org/D6827

run-tests: add a dedicated 'isoptional' function This is clearer than repeated manual call to to 'endswith'. (This is a gratuitous cleanup that I made while investigating a bug).

run-tests: remove the artificial indentation

run-tests: extract a `process_out_line` from the main function The main function doing line comparison is quite complex. Slicing it in smaller piece should clarify it. To avoid a huge diff, the code is kept at the same indentation. We'll re-indent in the next changesets. (This is a gratuitous cleanup that I made while investigating a bug).

run-tests: extract a `process_cmd_line` from the main function The main function doing line comparison is quite complex. Slicing it in smaller piece should clarify it. (This is a gratuitous cleanup that I made while investigating a bug).

changegroup: move message about added changes to transaction summary Before that, applying multiple changegroups in the same transaction issued the message multiple time. This result in a confusing output: adding changesets adding manifests adding file changes added 32768 changesets with 60829 changes to 2668 files adding changesets adding manifests adding file changes added 8192 changesets with 16885 changes to 1553 files adding changesets adding manifests adding file changes added 1020 changesets with 1799 changes to 536 files adding changesets adding manifests ... Instead, we now only issue the message once at the end of the transaction, summing up all added changesets, changes and files. The line is identical, but happens sightly later in the output. There are other suboptimal behavior around issue multiple changegroup (eg: progress bar). We'll cover them later. This impact of lot of test as one would expect, but a two pass check show they are just the order change we expected. To deal with "under the hood" bundle application by internal code, we had to take a slightly hacky move. We could clean that up with a more official way to enter "under the hood" section, however I want to keep this series simple to get it landed. This kind of change have a very high bit rot rate since it impact a lot of test output.

sshserver: flush stream after command dispatch I am not sure why this is not working as expected, but without this client might not see some important output. Without this patch moving some output at transaction closing time makes it disapear for ssh client in various sitaution.

narrow: rely on setting `quiet` mode instead of `pushbuffer` The `quiet` approach is what `shelve` uses and give the same result. This will help us to add less code in future changesets.

transaction: issue "new obsmarkers" message at the end of the transaction Instead of making bundle2 code responsible for this, it seems better to have it handled and the transaction level. First, it means the message will be more consistently printed. Second it means we won't spam the message over and over if the data arrive in multiple piece. Third, we are planning to move other similar message at the same level (for the same reason) so having them all at the same location will help us to control the order they are displayed.

debugobsolete: also issue the "new obsmarkers" messsage We are going to improve the way this message is issued in the core codebase. This will make it appears for `hg debugobsolete` too. Since this seems like a good idea, we make the output change in a previous changesets to clarify the next changeset.

split: use literal syntax to build a set of one element

rust-cpython: leverage py_shared_iterator::from_inner() where appropriate

rust-cpython: remove Option<_> from interface of py_shared_iterator It's the implementation detail of the py_shared_iterator that the leaked reference is kept in Option<_> so that it can be dropped early.

rust-cpython: rename py_shared_iterator_impl to py_shared_iterator It's a public interface now.

rust-cpython: replace dyn Iterator<..> of mapping with concrete type See the previous commit for why. The docstring is moved accordingly.

rust-cpython: replace dyn Iterator<..> of sequence with concrete type We wouldn't care the cost of the dynamic dispatch, but I feel a concrete type helps understanding error messages.

rust-dirstate: provide CopyMapIter and StateMapIter types They will be used in the declaration of Python iterator types.

rust-dirstate: specify concrete return type of DirsMultiset::iter() This allows us to put a returned iterator in a struct. We could implement DirsMultisetIter(hash_map::Keys<..>) struct to hide the implementation detail, but I think type alias is good enough for us.

discovery: replace "heads" by "changesets" in a output note (BC) When using `hg push --rev X`, the subset considered by discovery is only `::X`. In addition, `X` can be any local revisions not just local heads. As a result the message "all local heads known locally" can be misleading. We replace it with the more accurate "all local changesets known remotely". The message appears when in verbose not, so this is stricly speaking a BC breakage. I am not sure this would be a real issue in practice...

py3: drop incorrect fsencode(findexe(...)) in procutil I recently added the bad call, thinking that findexe() was a standard library function returning a string result, but it's actually our own function returning bytes. Thanks to Yuya for noticing. Differential Revision: https://phab.mercurial-scm.org/D6826

flagprocessors: small code update to clarify parameters 'raw' is really a third mode, not a small variant. Differential Revision: https://phab.mercurial-scm.org/D6807

flagprocessors: deprecate _processflags People should use the specialized version instead now. Differential Revision: https://phab.mercurial-scm.org/D6806

simplestorerepo: stop using `_processflags` directly We now use the specialized versions. Differential Revision: https://phab.mercurial-scm.org/D6805

revlog: stop using `_processflags` directly We now use the specialized versions. Differential Revision: https://phab.mercurial-scm.org/D6804

flagprocessors: use _processflagsraw in easy cases When there are no ambiguity regarding the `raw` value, we can simply use the new method. Differential Revision: https://phab.mercurial-scm.org/D6803

flagprocessors: use _processflagsread in simple cases When there are no ambiguity regarding the `raw` value, we can simply use the new method. Differential Revision: https://phab.mercurial-scm.org/D6802

flagprocessors: use _processflagswrite for write operation There are no ambiguity for 'write' operation so it is simple to replace. Differential Revision: https://phab.mercurial-scm.org/D6801

flagprocessors: introduce specialized functions This make the call site clearer and the open the way to more diverse return types. For now, the same old code is still in use under the hood. Differential Revision: https://phab.mercurial-scm.org/D6800

flagutil: use it in simplestorerepo This remove the other code duplication of the `_processflags` code. To be honest, this code looks very dead since it is not run by either developer nor buildbot. However, we update the code to make the task of reviving this less daunting. Differential Revision: https://phab.mercurial-scm.org/D6799

flagutil: make the error class used by the mixin configurable One of the code duplication use a different error class. So let's make it possible to do so. Differential Revision: https://phab.mercurial-scm.org/D6798

flagutil: use the new mixin use in remotefilelog This remove one of the code duplication. Hooray \o/. Differential Revision: https://phab.mercurial-scm.org/D6797

flagutil: introduce a flagprocessorsmixin class To avoid code duplication, we will provide a simple "ready to use" mixin that carry the appropriate logic. First we use it in standard revlog, we'll remove code duplication in later changesets. Differential Revision: https://phab.mercurial-scm.org/D6796

check-code: allow command substitution with $(command) Both `command` and $(command) are specified by POSIX. The latter nests better. I don't see why we shouldn't allow both (or only the latter). Differential Revision: https://phab.mercurial-scm.org/D6789

run-tests: rename `lcmd` variable to `line_cmd` This is clearer and more in line with some other variable names. (This is a gratuitous cleanup that I made while investigating a bug).

run-tests: rename `lout` variable to `out_line` This is clearer and more in line with some other variable names. (This is a gratuitous cleanup that I made while investigating a bug).

run-tests: clarify "l" variable as "out_rawline" More explicit variable name never hurt. (This is a gratuitous cleanup that I made while investigating a bug).

run-tests: use symbolic constant instead of arbitrary number line matching (This is a gratuitous cleanup that I made while investigating a bug).

rustfilepatterns: shorter code for concatenating slices Differential Revision: https://phab.mercurial-scm.org/D6765

match: simplify the regexps created for glob patterns For legibility of the resulting regexes, although it may help with performance as well. Differential Revision: https://phab.mercurial-scm.org/D6764

rustfilepatterns: refactor the pattern of removing a prefix from a &[u8] Differential Revision: https://phab.mercurial-scm.org/D6766

tests: show the pattern generated for a relative glob Differential Revision: https://phab.mercurial-scm.org/D6763

copies: remove existing copy info from the changeset on amend (BC) When amending a changeset with copy information in the changeset and the new changeset doesn't have any copy information (or similar for "filesadded" and "filesremoved"), we shouldn't keep it. A drawback of this is that we now unconditionally remove these four entries from the extras, breaking any extensions that happened to write entries with the same names (which seems very unlikely). I think I'd heard that there was list of blacklisted keys that would be removed from the extras when a commit is rewritten, but I couldn't find that. It would make sense to add the keys mentioned above there instead of the custom filtering I've added in this patch. Differential Revision: https://phab.mercurial-scm.org/D6752

tests: show invalid copies when turning off copies-in-changeset If you turn on copies in changesets and write a commit with a copy, then turn it off and amend the commit while undoing the copy, the invalid copy information will remain. The read path doesn't crash in invalid copy data, but it seems better to not produce the invalid data. Differential Revision: https://phab.mercurial-scm.org/D6751

context: filter out invalid copies from workingctx.p[12]copies() workingctx normally gets its lists of modified, added, removed files etc. based on the dirstate status. Its constructor also accepts a "changes" argument to override the status from the dirstate. This is used for partial commits. If a "changed" argument was passed, we should clearly also filter out copies to those paths, which I had previously missed. This patch adds that filtering and fixes the bugs demonstrated in the previous patch. Differential Revision: https://phab.mercurial-scm.org/D6750

tests: demonstrate crash when committing subset of copies to changeset When writing copy metadata to the changeset and not committing all copies in the dirstate, we get a ProgrammingError. This commit adds two tests showing how to trigger this bug. Differential Revision: https://phab.mercurial-scm.org/D6749

bdiff-torture: fix pyflakes warning reporting undefined name 'inst' Looks like I got a latest version of pyflakes somehow or it's running on py3 and it spotted this. Differential Revision: https://phab.mercurial-scm.org/D6757

split: handle partial commit of renames when doing split or record (issue5723) When using split or record, using either interface (text or curses), selecting portions of the file to be committed/recorded did not work; the entire file was treated as having been selected. This was because the logic for handling partial application of the patches relies on knowing what files are "new with modifications" and it doesn't treat "rename destination" as "new". There was a complicating issue, however. We're relying on the patch header specifying the copy from/to information, which works as long as the 'copy from' file is there. In the case of renames, however, the 'rename from' file is *not* there, so we need to add it back. Differential Revision: https://phab.mercurial-scm.org/D6768

split: handle partial commit of copies when doing split or record When using split or record, using either interface (text or curses), selecting portions of the file to be committed/recorded did not work; the entire file was treated as having been selected. This appears to be because the logic for handling partial application of the patches relies on knowing what files are "new with modifications", and it doesn't treat "copy destination" as "new". Handling renames correctly is more difficult and will be done in a later patch. Differential Revision: https://phab.mercurial-scm.org/D6767

py3: use pycompat.sysargv[0] for instead of fsencode(sys.argv[0]) Yuya noted in a recent review that fsencode(sys.argv[0]) could be incorrect on Windows. Differential Revision: https://phab.mercurial-scm.org/D6782

httppeer: use context manager when reading temporary bundle to send Differential Revision: https://phab.mercurial-scm.org/D6784

httppeer: use context manager when writing temporary bundle to send Differential Revision: https://phab.mercurial-scm.org/D6783

rust-cpython: mark unsafe functions as such It wasn't trivial to fix leak_immutable() to be safe since we have to allow immutable operations (e.g. iter()) on the leaked reference. So let's mark it unsafe for now. Callers must take care of the returned object to guarantee the memory safety. I'll revisit this later. I think $leaked<T: 'static> could have a function that converts itself into $leaked<U: 'static> with a given FnOnce(&T) -> &U, where T is $inner_struct, and U is $iterator_type for example.

rust-cpython: pair leaked reference with its manager object Still leak_immutable() is unsafe since leak_handle must live longer than the leaked_ref.

rust-cpython: introduce restricted variant of RefCell This should catch invalid borrow_mut() calls. Still the ref-sharing interface is unsafe.

rust-cpython: fix unsafe inner(py).borrow_mut() calls Since self.inner is managed by PySharedState, it must not be borrowed mutably through the RefCell interface. Otherwise, the underlying object could be mutated while a reference is leaked to Python world.

revlog: deprecate the use of `revision(..., raw=True)` We have an official `rawdata` function now.

remotefilelog: reduce probability of race-condition in remotefilelog tests ca1014ad3de4 introduced a new parameter `ensurestart` to speed up remotefilelog background processes start. Unfortunately it seems to have increased the possibility of race-conditions in remotefilelog tests testing those background processes. With `ensurestart=False`, it seems that it's more probable to enter in a race condition with `debugwaitonprefetch` and `debugwaitonrepack` in remotefilelog background tests. Our CI seems to have a high probability of triggering this race condition so make it configurable to ensure tests are stable. Differential Revision: https://phab.mercurial-scm.org/D6772

rust: apply more formatting fixes My cargo fmt updated these lines and they look good.

rust-utils: add normalize_case util to mirror Python one While we still don't handle filenames properly cross-platform, this at least sticks closer to the Python behavior. Differential Revision: https://phab.mercurial-scm.org/D6756

rust: fix warnings about trait objects without dyn being deprecated Differential Revision: https://phab.mercurial-scm.org/D6770

py3: convert hg executable path to bytes in missing case in procutil We (Google) noticed this in our tests when we use chg and a hg wrapper script not called 'hg'. The executable then ended up being a native string, which then failed in chgserver when trying to convert the environment dict to a byte string. Differential Revision: https://phab.mercurial-scm.org/D6775

py3: make statprof's chrome output work With this patch, this command works: python3 hg --profile --config profiling.statformat=chrome st (and it works with s/python3/python2/ as well) Differential Revision: https://phab.mercurial-scm.org/D6777

py3: for statprof's Chrome output, write json to string, then encode to bytes `json.dump(obj, fp)` requires `fp.write()` to accept str output, and since the file pointer we have there only accepts bytes, we need to change to json.dumps() and then encode as utf-8. We have already done the same thing for the json (non-Chrome) format in 4b7eb862692e (py3: encode json output to bytes and use write(), 2018-10-12). Differential Revision: https://phab.mercurial-scm.org/D6781

statprof: use context manager for file when writing flame graph Differential Revision: https://phab.mercurial-scm.org/D6780

statprof: use context manager when reading source from file Differential Revision: https://phab.mercurial-scm.org/D6779

statprof: clarify by naming tuple members while enumerate()'ing Differential Revision: https://phab.mercurial-scm.org/D6778

upgrade: make sure we reclone all revlogs when updating to some format Adding or removing some requirement (eg: sparserevlog), requires to reclone revlog to use the new format. We cannot simply copy the original files. In this case, we issue a warning to proceed with clone every revlogs.

upgrade: add an argument to control changelog upgrade Same as for `--manifest` we can now select more selection.

upgrade: add an argument to control manifest upgrade The argument can be used to only "clone" manifest revlog or clone all of them but this one. The selection will make more sense once we have a `--changelog` flag in the next changesets.

unionrepo: drop the custom `rawdata` implementation We can rely on the main one now.

unionrepo: drop `baserevdiff` It has no caller anymore.

unionrepo: use normal inheritance scheme to call revdiff

unionrepo: fix `revdiff` implementation to use `rawdata` The parent code is using rawdata so we should use it here. Before this change, union repo was probably broken with some flag processors.

unionrepo: get rid of `baserevision` The method is not called anywhere anymore, so we can safely drop it. Some of the comment get moved to `baserevdiff` because we did not got rid of it (yet).

unionrepo: use a lower level overide in unionrepo too The unionrepo class also have a strange `baserevision` hack, let's try to get ride of it too.

bundlerepo: drop the custom `rawdata` implementation We can rely on the main one now.

bundlerepo: drop the `baserevision` hack It is not used anywhere anymore, so we can safely drop it.

bundlerepo: simplify code to take advantage of `_rawtext` In the revlog code, the code getting the raw text is now isolated. We take advantage of this to simplify the bundlerepo code.

merge with stable

rust: run cargo fmt

py3: use pycompat.maplist() in chgserver test-chg.t almost passes on py3 after this patch. Differential Revision: https://phab.mercurial-scm.org/D6771

run-tests: handle --local before --with-hg We no longer support them both together, so this is now safe to do. By checking --local first, we avoid error out about an invalid --with-hg script if --local was also given (we instead tell the user that the options are mutually exclusive). I also had to wrap the 'binpath' we pass to setattr() in _strpath() to keep `python3 run-tests.py -l` working. That change also made `python3 run-tests.py -l --chg` work, which was the reason for this series. Differential Revision: https://phab.mercurial-scm.org/D6760

run-tests: error out on `--local --with-[c]hg` I don't see much reason to allow these combinations. You could use --local and override only one of --with-hg or --with-chg, but I don't see much practical use for that. It would be easy to work around anyway by passing both --with-hg and --with-chg. By erroring out, it makes the code a bit easier to reason about to allow the next few patches. Differential Revision: https://phab.mercurial-scm.org/D6759

contrib: simplify the genosxversion.py command to find the hg libraries I forget what problem I ran into while trying to teach the makefile to use a non-system python. (It might have ben missing hg-evolve and/or keyring, but `check_output()` was raising an error.) This still isn't great because it will return non zero for something like the username not being set, even though we aren't asking for it. But I suppose it's still useful to simplify. Differential Revision: https://phab.mercurial-scm.org/D6753

interfaceutil: move to interfaces/ Now that we have a dedicated folder for interfaces, let's move interfaceutil there. Differential Revision: https://phab.mercurial-scm.org/D6742

interfaces: create a new folder for interfaces and move repository.py in it I was trying to understand current interfaces and write new ones and I realized we need to improve how current interfaces are organised. This creates a dedicated folder for defining interfaces and move `repository.py` which defines all the current interfaces inside it. Differential Revision: https://phab.mercurial-scm.org/D6741

narrow: fix typo "respositories" Differential Revision: https://phab.mercurial-scm.org/D6758

merge with stable

merge: hint about using `hg resolve` for resolving conflicts This was suggested by one of our users at Google. Makes sense to me. Differential Revision: https://phab.mercurial-scm.org/D6755

rust-dirstate: remove test case for DirsMultiset::new(Manifest, Some) It's no longer possible.

rust-dirstate: split DirsMultiset constructor per input type Since skip_state only applies to dirstate, it doesn't make sense to unify these constructors and dispatch by enum.

rust-dirstate: remove excessive clone() of parameter and return value I think pass-by-ref is preferred in general.

rust-dirstate: handle invalid length of p1/p2 parameters It uses match syntax since map_err() failed to deduce the argument type.

rust: simply use TryInto to convert slice to array Since our rust module depends on TryInto, there's no point to avoid using it. While rewriting copy_into_array(), I noticed CPython interface doesn't check the length of the p1/p2 values, which is marked as TODO.

rust-dirstate: use PARENT_SIZE constant where appropriate

rust-dirstate: rename NULL_REVISION to NULL_ID which isn't a revision number

rust-dirstate: remove repetition in array literal

rust-dirstate: remove too abstracted way of getting &[u8]

rust-dirstate: remove unneeded "ref" At 7cae6bc29ff9, .to_owned() was rewritten as .to_owned().to_vec(), which is no longer needed since the filename is a single reference.

rust-parsers: fix unboxing of PyInt on Python 3 Broken at 7cae6bc29ff9.

revlog: split `rawtext` retrieval out of _revisiondata This part is reasonably independent. Having it on its own clarify the code flow and will help code that inherit from revlog to overwrite specific area only.

revlog: avoid caching raw text too early in _revisiondata Without this change, we could cache the rawtext without considering for it validating the cache or not. If the exception raised by the invalid hash were to be caught and the same revision accessed again, the invalid rawtext would be returned.

revlog: add some documentation to `_revisiondata` code

revlog: move `nullid` early return sooner in `_revisiondata` Let us deal with the special case before we start dealing with more generic case.

revlog: stop calling `basetext` `rawtext` in _revisiondata If the cache entry is used as a base test for delta, it is not the rawtext we need. We update the variable name to clarify this.

revlog: assign rawtext earlier in `_revisiondata` Assigning the revision earlier make the code easier to read.

revlog: drop silly `raw` parameter to `rawdata` function This is a leftover from `revision` and does not make sense for `rawdata`

perf: don't depend on pycompat for older Mercurial versions We already define local _sysstr() and _xrange() for compatibility, so we should use those. Also create a similar local _bytestr() corresponding to pycompat.bytestr(). Differential Revision: https://phab.mercurial-scm.org/D6745

perf: don't try to call `util.queue` on Mercurial version before it existed Differential Revision: https://phab.mercurial-scm.org/D6744

perf: handle NameError for `pycompat.foo` when pycompat wasn't imported On old Mercurial versions, we won't have a pycompat variable defined, and then `pycompat.foo` will raise a NameError. Differential Revision: https://phab.mercurial-scm.org/D6743

rawdata: update callers in shallowbundle We update callers incrementally because this help bisecting failures. This was useful during development, so we expect it might be useful again in the future.

rawdata: update callers in storageutils We update callers incrementally because this help bisecting failures. This was useful during development, so we expect it might be useful again in the future.

rawdata: update callers in delta utils We update callers incrementally because this help bisecting failures. This was useful during development, so we expect it might be useful again in the future.

rawdata: update callers in repository We update callers incrementally because this help bisecting failures. This was useful during development, so we expect it might be useful again in the future.

rawdata: update callers in testing/storage.py We update callers incrementally because this help bisecting failures. This was useful during development, so we expect it might be useful again in the future.

rawdata: update callers in test-revlog-raw We update callers incrementally because this help bisecting failures. This was useful during development, so we expect it might be useful again in the future.

rawdata: update callers in lfs' tests We update callers incrementally because this help bisecting failures. This was useful during development, so we expect it might be useful again in the future.

rawdata: update callers in lfs' wrapper We update callers incrementally because this help bisecting failures. This was useful during development, so we expect it might be useful again in the future.

rawdata: update caller in wireprotov2server We update callers incrementally because this help bisecting failures. This was useful during development, so we expect it might be useful again in the future.

rawdata: update callers in debugcommands We update callers incrementally because this help bisecting failures. This was useful during development, so we expect it might be useful again in the future.

rawdata: update callers in sqlitestore We update callers incrementally because this help bisecting failures. This was useful during development, so we expect it might be useful again in the future.

rawdata: update caller in remotefilelog We update callers incrementally because this help bisecting failures. This was useful during development, so we expect it might be useful again in the future.

rawdata: update callers in bundlerepo We update callers incrementally because this help bisecting failures. This was useful during development, so we expect it might be useful again in the future.

rawdata: update callers in context We update callers incrementally because this help bisecting failures. This was useful during development, so we expect it might be useful again in the future.

rawdata: update caller in revlog We update callers incrementally because this help bisecting failures. This was useful during development, so we expect it might be useful again in the future.

setup: fix a sorting issue I noticed in package names Differential Revision: https://phab.mercurial-scm.org/D6733

py3: do not convert rust module/attribute names to bytes policy.import*() functions expect system strings.

transplant: unnest --stop case It should be aligned with --continue.

rust-discovery: use while loop instead of match + break This looks slightly nicer.

rust-discovery: remove useless extern crate

transplant: added support for --stop flag This adds fuctionality for `--stop` flag to `transplant`. A new method `stop` is added to `transplanter` class containing logic to abort transplant. Tests are updated as shown. Differential Revision: https://phab.mercurial-scm.org/D6695

unshelve: abort on using --keep and --interactive together I am working on making interactive mode support `--keep` flag. Until we support the usage of `--interactive` and `--keep` together, let us abort on it. Differential Revision: https://phab.mercurial-scm.org/D6699

config: add experimental argument to the config registrar Until now, there are almost 28 config items which are considered as `experimental` but, not present in the `experimental` section of the registrar. This patch adds an `experimental` argument to the config registrar to mark such config items. Differential Revision: https://phab.mercurial-scm.org/D6728 Differential Revision: https://phab.mercurial-scm.org/D6746

tests: split joint repo/changelog fake into one for each type I'm just not comfortable with fakes that get overloaded for multiple types: too often it gets out of hand and becomes difficult to trace. Differential Revision: https://phab.mercurial-scm.org/D6725

fix: pass line ranges as value instead of callback The callback no longer takes any arguments from the inner function, so we might as well call it sooner and pass the value instead. Note the value still needs to be recomputed every iteration to account for the previous iteration's changes to the file content. Differential Revision: https://phab.mercurial-scm.org/D6727

fix: correctly parse the :metadata subconfig It's being handled as a string instead of a bool, though the thruthiness of the string makes the feature still essentially work. Added a regression test. Differential Revision: https://phab.mercurial-scm.org/D6726

fix: allow tools to use :linerange, but also run if a file is unchanged The definition of "unchanged" here is subtle, because pure deletion diff hunks are ignored. That means this is different from using the --whole flag. This change allows you to configure, for example, a code formatter that: 1. Formats specific line ranges if specified via flags 2. Does not format the entire file when there are no line ranges provided 3. Performs some other kind of formatting regardless of provided line ranges This sounds a little far fetched, but it is meant to address a specific corner case encountered in Google's use of the fix extension. The default behavior is kept because it exists to prevent mistakes that could erase uncommitted changes. Differential Revision: https://phab.mercurial-scm.org/D6723

rust-dirstate: call rust dirstatemap from Python Since Rust-backed Python classes cannot be used as baseclasses (for rust-cpython anyway), we use composition rather than inheritance. This also allows us to keep the IO operations in the Python side, removing (for now) the need to rewrite VFS in Rust, which would be a heavy undertaking. Differential Revision: https://phab.mercurial-scm.org/D6634

rust-dirstate: rust-cpython bridge for dirstatemap This change also showcases the limitations of the `py_shared_ref!` macro. See the previous commit 'rust-dirstate: rust implementation of dirstatemap` for an explanation for the TODOs in the code. Differential Revision: https://phab.mercurial-scm.org/D6633

rust-dirstate: rust implementation of dirstatemap The `dirstatemap` is one of the last building blocks needed to get to a `dirstate.walk` Rust implementation. Disclaimer: This change is part of a big (10) series of patches, all of which started as one big changeset that took a long time to write. This `dirstatemap` implementation is a compromise in terms of complexity both for me and for the reviewers. I chose to submit this patch right now because while it is not perfect, it works and is simple enough (IMHO) to be reviewed. The Python implementation uses a lot of lazy propertycaches, breaks encapsulation and is used as an iterator in a lot of places, all of which dictated the somewhat unidiomatic patterns in this change. Like written in the comments, rewriting this struct to use the typestate pattern might be a good idea, but this is a good first step. Differential Revision: https://phab.mercurial-scm.org/D6632

rust-cpython: add macro for sharing references Following an experiment done by Georges Racinet, we now have a working way of sharing references between Python and Rust. This is needed in many points of the codebase, for example every time we need to expose an iterator to a Rust-backed Python class. In a few words, references are (unsafely) marked as `'static` and coupled with manual reference counting; we are doing manual borrow-checking. This changes introduces two declarative macro to help reduce boilerplate. While it is better than not using macros, they are not perfect. They need to: - Integrate with the garbage collector for container types (not needed as of yet), as stated in the docstring - Allow for leaking multiple attributes at the same time - Inject the `py_shared_state` data attribute in `py_class`-generated structs - Automatically namespace the functions and attributes they generate For at least the last two points, we will need to write a procedural macro instead of a declarative one. While this reference-sharing mechanism is being ironed out I thought it best not to implement it yet. Lastly, and implementation detail renders our Rust-backed Python iterators too strict to be proper drop-in replacements, as will be illustrated in a future patch: if the data structure referenced by a non-depleted iterator is mutated, an `AlreadyBorrowed` exception is raised, whereas Python would allow it, only to raise a `RuntimeError` if `next` is called on said iterator. This will have to be addressed at some point. Differential Revision: https://phab.mercurial-scm.org/D6631

rust-docstrings: add missing module docstrings Differential Revision: https://phab.mercurial-scm.org/D6630

rust-dirstate: improve API of `DirsMultiset` - Use opaque `Iterator` type instead of implementation-specific one from `HashMap` - Make `DirsMultiset` behave like a set both in Rust and from Python Differential Revision: https://phab.mercurial-scm.org/D6690

rust-dirstate: use EntryState enum instead of literals This improves code readability quite a bit, while also adding a layer of safety because we're checking the state byte against the enum. Differential Revision: https://phab.mercurial-scm.org/D6629

rust-parsers: switch to parse/pack_dirstate to mutate-on-loop Both `parse_dirstate` and `pack_dirstate` can operate directly on the data they're passed, which prevents the creation of intermediate data structures, simplifies the function signatures and reduces boilerplate. They are exposed directly to the Python for now, but a later patch will make use of them inside `hg-core`. Differential Revision: https://phab.mercurial-scm.org/D6628

rust-parsers: move parser bindings to their own file and Python module This tidies up the Rust side while simplifying the Python side. Differential Revision: https://phab.mercurial-scm.org/D6627

rust-dirstate: create dirstate submodule in hg-cpython This module will soon hold multiple files, this change is to make the review process easier. Differential Revision: https://phab.mercurial-scm.org/D6626

rust-discovery: using from Python code As previously done in other topics, the Rust version is used if it's been built. The version fully in Rust of the partialdiscovery class has the performance advantage over the Python version (actually using the Rust MissingAncestor) if the undecided set is big enough. Otherwise no sampling occurs, and the discovery is reasonably fast anyway. Note: it's hard to predict the size of the initial undecided set, it can depend on the kind of topological changes between the local and remote graphs. The point of the Rust version is to make the bad cases acceptable. More specifically, the performance advantages are: - faster sampling, especially takefullsample() - much faster addmissings() in almost all cases (see commit message in grandparent of the present changeset) - no conversion cost of the undecided set at the interface between Rust and Python == Measurements with big undecided sets For an extreme example, discovery between mozilla-try and mozilla-unified (over one million undecided revisions, same case as in dbd0fcca6dfc), we get roughly a x2.5/x3 better performance: Growing sample size (5% starting with 200): time goes down from 210 to 72 seconds. Constant sample size of 200: time down from 1853 to 659 seconds. With a sample size computed from number of roots and heads of the undecided set (`respectsize` is `False`), here are perfdiscovery results: Before ! wall 9.358729 comb 9.360000 user 9.310000 sys 0.050000 (median of 50) After ! wall 3.793819 comb 3.790000 user 3.750000 sys 0.040000 (median of 50) In that later case, the sample sizes are routinely in the hundreds of thousands of revisions. While still faster, the Rust iteration in addmissings has less of an advantage than with smaller sample sizes, but one sees addcommons becoming faster, probably a consequence of not having to copy big sets back and forth. This example is not a goal in itself, but it showcases several different areas in which the process can become slow, due to different factors, and how this full Rust version can help. == Measurements with small undecided sets In cases the undecided set is small enough than no sampling occurs, the Rust version has a disadvantage at init if `targetheads` is really big (some time is lost in the translation to Rust data structures), and that is compensated by the faster `addmissings()`. On a private repository with over one million commits, we still get a minor improvement, of 6.8%: Before ! wall 0.593585 comb 0.590000 user 0.550000 sys 0.040000 (median of 50) After ! wall 0.553035 comb 0.550000 user 0.520000 sys 0.030000 (median of 50) What's interesting in that case is the first addinfo() at 180ms for Rust and 233ms for Python+C, mostly due to add_missings and the children cache computation being done in less than 0.2ms on the Rust side vs over 40ms on the Python side. The worst case we have on hand is with mozilla-try, prepared with discovery-helper.sh for 10 heads and depth 10, time goes up 2.2% on the median. In this case `targetheads` is really huge with 165842 server heads. Before ! wall 0.823884 comb 0.810000 user 0.790000 sys 0.020000 (median of 50) After ! wall 0.842607 comb 0.840000 user 0.800000 sys 0.040000 (median of 50) If that would be considered a problem, more adjustments can be made, which are prematurate at this stage: cooking special variants of methods of the inner MissingAncestors object, retrieving local heads directly from Rust to avoid the cost of conversion. Effort would probably be better spent at this point improving the surroundings if needed. Here's another data point with a smaller repository, pypy, where performance is almost identical Before ! wall 0.015121 comb 0.030000 user 0.020000 sys 0.010000 (median of 186) After ! wall 0.015009 comb 0.010000 user 0.010000 sys 0.000000 (median of 184) Differential Revision: https://phab.mercurial-scm.org/D6430

rust-discovery: optimization of add commons/missings for empty arguments These two cases have to be catched early for different reasons. In the case of add_missing_revisions, we don't want to trigger the computation of the undecided set (and the children cache) too early: the later the better. In the case of add_common_revisions, the inner `MissingAncestors` object wouldn't know that all ancestors of its bases have already been removed from the undecided. In principle, that would in itself be a lead for further improvement: this remove_ancestors_from could be more incremental, but the current performance seems to be good enough. Differential Revision: https://phab.mercurial-scm.org/D6429

rust-discovery: using the children cache in add_missing The DAG range computation often needs to get back to very old revisions, and turns out to be disproportionately long, given that the end goal is to remove the descendents of the given missing revisons from the undecided set. The fast iteration capabilities available in the Rust case make it possible to avoid the DAG range entirely, at the cost of precomputing the children cache, and to simply iterate on children of the given missing revisions. This is a case where staying on the same side of the interface between the two languages has clear benefits. On discoveries with initial undecided sets small enough to bypass sampling entirely, the total cost of computing the children cache and the subsequent iteration becomes better than the Python + C counterpart, which relies on reachableroots2. For example, on a repo with more than one million revisions with an initial undecided set of 11 elements, we get these figures: Rust version with simple iteration addcommons: 57.287us first undecided computation: 184.278334ms first children cache computation: 131.056us addmissings iteration: 42.766us first addinfo total: 185.24 ms Python + C version first addcommons: 0.29 ms addcommons 0.21 ms first undecided computation 191.35 ms addmissings 45.75 ms first addinfo total: 237.77 ms On discoveries with large undecided sets, the initial price paid makes the first addinfo slower than the Python + C version, but that's more than compensated by the gain in sampling and subsequent iterations. Here's an extreme example with an undecided set of a million revisions: Rust version: first undecided computation: 293.842629ms first children cache computation: 407.911297ms addmissings iteration: 34.312869ms first addinfo total: 776.02 ms taking initial sample query 2: sampling time: 1318.38 ms query 2; still undecided: 1005013, sample size is: 200 addmissings: 143.062us Python + C version: first undecided computation 298.13 ms addmissings 80.13 ms first addinfo total: 399.62 ms taking initial sample query 2: sampling time: 3957.23 ms query 2; still undecided: 1005013, sample size is: 200 addmissings 52.88 ms Differential Revision: https://phab.mercurial-scm.org/D6428

discovery: new devel.discovery.randomize option By default, this is True, but setting it to False is a uniform way to kill all randomness in integration tests such as test-setdiscovery.t By "uniform" we mean that it can be passed to implementations in other languages, for which the monkey-patching of random.sample would be irrelevant. In the above mentioned test file, we use it right away, replacing the adhoc extension that had the same purpose, and to derandomize a case with many round-trips, that we'll need to behave uniformly in the Rust version. Differential Revision: https://phab.mercurial-scm.org/D6427

rust-discovery: optionally don't randomize at all, for tests As seen from Python, this is a new `randomize` kwarg in init of the discovery object. It replaces random picking by some arbitrary yet deterministic strategy. This is the same as what test-setdiscovery.t does, with the added benefit to be usable both in Python and Rust implementations. Differential Revision: https://phab.mercurial-scm.org/D6426

rust-discovery: exposing sampling to python Differential Revision: https://phab.mercurial-scm.org/D6425

rust-discovery: takefullsample() core implementation take_full_sample() browses the undecided set in both directions: from its roots as well as from its heads. Following what's done on the Python side, we alter update_sample() signature to take a closure returning an iterator: either ParentsIterator or an iterator over the children found in `children_cache`. These constructs should probably be split off in a separate module. This is a first concrete example where a more abstract graph notion (probably a trait) would be useful, as this is nothing but an operation on the reversed DAG. A similar motivation in the context of the discovery process would be to replace the call to dagops::range in `add_missing_revisions()` with a simple iteration over descendents, again an operation on the reversed graph. Differential Revision: https://phab.mercurial-scm.org/D6424

rust-discovery: core implementation for take_quick_sample() This makes in particular `rand` no longer a testing dependency. We keep a seedable random generator on the `PartialDiscovery` object itself, to avoid lengthy initialization. In take_quick_sample() itself, we had to avoid keeping the reference to `self.undecided` to cope with the mutable reference introduced by the the call to `limit_sample`, but it's still manageable without resorting to inner mutability. Sampling being prone to be improved in the mid-term future, testing is minimal, amounting to checking which code path got executed. Differential Revision: https://phab.mercurial-scm.org/D6423

rust-discovery: read the index from a repo passed at init This makes the API of the Rust PartialDiscovery object now the same (or rather a subset) of the Python object, hence easier to control through module policy down the road. Differential Revision: https://phab.mercurial-scm.org/D6517

rust-discovery: accept the new 'respectsize' init arg At this stage, we don't do anything about it: it will be meaningful in sampling methods that aren't implemented yet. Differential Revision: https://phab.mercurial-scm.org/D6516

merge with stable

unshelve: forget unknown files after a partial unshelve This is a follow-up patch to 6957f7b93e03. This allows hg to forget unknown files after a partial unshelve. Differential Revision: https://phab.mercurial-scm.org/D6724

flagutil: move addflagprocessor to the new module (API)

flagutil: move insertflagprocessor to the new module (API)

flagutil: move REVIDX_KNOWN_FLAGS source of truth in flagutil (API) Since REVIDX_KNOWN_FLAGS is "not really a constant" (extension can update it) and python integer,... it needs to be the responsability of a single module and always accessed through the module. We update all the user and move the source of truth in flagutil.

flagutil: move the `flagprocessors` mapping in the new module This module is meant to host most of the flag processing logic. We start with the mapping between flag and processors.

flagutil: create a `mercurial.revlogutils.flagutil` module The flagprocessings logic is duplicated in 2 extra places, and usually in a less robust flavor. This is a maintenance nightmare that I would like to see cleaned up. To do so I am creating a `flagutil` module to move flag processings related code and make it easily reusable by other code.

rawdata: register the method for `ifiledata` The interface have a `revision(..., raw=False)` method so it should get a `rawdata` one. I am not sure why nothing complained about the lack of it earlier.

rawdata: implement the method for `unionrepo` too This is required for all implementations.

rawdata: implement the method for `remotefilelog` too This is needed for all storage implementations.

rawdata: implement `rawdata` for `simplestore` too This is needed for all implementation.

rawdata: forward `rawdata` call on `manifestlog` This needs to be sent to the underlying `revlog` too.

rawdata: implement `rawdata` for `sqlitestore` too This is a different store, it needs it declared.

rawdata: add the method to bundlerevlog The bundlerepo logic has its own `revision` method on its own `revlog` object. We need to "implement" `rawdata` there too.

rawdata: forward the method call on `filelog` object We have a new method, we need to expose it.

rawdata: introduce a `rawdata` method on revlog This method aims at replacing `revision(..., raw=True)` call. The purpose of data returned without and without raw are different enough that having two different method would make sense. This split is motivated by other work aiming at storing data on the side of the main revision of a revlog. Having a cleaner API makes it simpler to add this work. The series following this first changesets is organised as follow: 1) add `rawdata` method everywhere it is useful 2) update all caller 3) implement all `rawdata` method without using `revision` 4) deprecate the `rawdata` parameter of `revision`

revlog: split a `_revisiondata` method to file `revision` job We are about to introduce more public method to access revision data (eg: `rawdata`). revset subclass tend to recursively call `revision` which will create all kind of issue with the coming series. To avoid them we introduce an explicit difference between the internal call and the public all. This will be useful for later work anyway (so the subclass issue is just moving it earlier in the series). I am not sure if the subclass are actually doing something sensible. However, I am certain I don't want to be rabbit holed into figuring it out right now.

continue: added support for transplant This creates a seperate function `continuetransplant()` containing logic for resuming transplant from interrupted state. `continuetransplant()` is then registered as `continuefunc` for state detection API. Results are shown in tests. Differential Revision: https://phab.mercurial-scm.org/D6689

merge with stable

branchmap: explicitly warm+write all subsets of the branchmap caches 'full' claims it will warm all of the caches that are known about, but this was not the case - it did not actually warm the branchmap caches for subsets that we haven't requested, or for subsets that are still considered "valid". By explicitly writing them to disk, we can force the subsets for ex: "served" to be written ("immutable" and "base"), making it cheaper to calculate "served" the next time it needs to be updated. Differential Revision: https://phab.mercurial-scm.org/D6710

changectx: extract explicit computechangesetfilesremoved method from context Right now, the logic around changeset centric removed files data are buried into the "changectx" code. We extract this code in a dedicated method (in the scmutil module) for clarity. This clarity will help to explicitly compute and caches these data in the future.

changectx: extract explicit computechangesetfilesadded method from context Right now, the logic around changeset centric added files data are buried into the "changectx" code. We extract this code in a dedicated method (in the scmutil module) for clarity. This clarity will help to explicitly compute and caches these data in the future.

copies: extract an explicit `computechangesetcopie` method from context Right now, the logic around changeset centric copies data are buried into the "changectx" code. We extract this code in a dedicated method (in the copies module) for clarity. This clarity will help to explicitly compute and caches these data in the future.

config: fix fm.data() handling of defaultvalue This is a follow-up patch to rHG51a2e3102db2. This moves `fm.data()` out of the if block in `commands.config()`. Differential Revision: https://phab.mercurial-scm.org/D6720

config: remove pycompat.bytestr() for defaultvalue This is a follow-up patch to 51a2e3102db2. This removes `pycompat.bytestr` to preserve `None` in `commands.config()`. Differential Revision: https://phab.mercurial-scm.org/D6712

unshelve: clear shelvedstate and _finishunshelve() on partial unshelve On a partial unshelve, `shelvedstate` was not cleared and `_finishunshelve()` was not called. Ideally, these should be called on this case. This patch makes `shelvedstate` to delete after a successful partial unshelve and calls `_finishunshelve()` in the same case. Differential Revision: https://phab.mercurial-scm.org/D6708

unshelve: delete shelvedstate after a successful unshelve --continue `unshelve --continue` was preventing the deletion of `shelvedstate` on a partial `unshelve`. Ideally, `shelvedstate` should be deleted after a successful `unshelve`. Now, the behavior of `unshelve --continue` will be as follows in interactive mode: 1] The user tried to `unshelve` changes interactively but, ran into conflicts. 2] They resolved the conflicts and triggered `unshelve --continue` but, unshelved changes partially. 3] Now, on trying to do `unshelve --continue` again will abort as the last `unshelve` was successful and we are deleting the `shelvedstate`. 4] If they want to unshelve the remaining shelved change, they need to trigger `unshelve` without `--continue`. Differential Revision: https://phab.mercurial-scm.org/D6694

unshelve: handle stripping changesets on interactive mode On interactive mode, changesets on `nodestoremove` should be stripped regardless of the shelve is partial or not. This patch modifies `unshelvecontinue()` to do that. Differential Revision: https://phab.mercurial-scm.org/D6686

byteify-strings: add --version argument This is indispensable for automated tools to detect changes in behavior.

byteify-strings: add space in special comments to silence flake8 error This is done soon enough that nobody has had the time to use this feature yet.

revset: drop argument when it's None getstack's definition is `getstack(repo, rev=None)`, so providing None explicitly is unnecessary. Moreover, when x is not None, it's definitely not a revision but a part of a parsed tree of revset arguments. Differential Revision: https://phab.mercurial-scm.org/D6707

stack: remove unnecessary reverse() predicate Stack already sorts revisions, so no need to do it twice. This change was a part of D2400, which didn't land for other reasons. See also D2399, where this change was suggested. Differential Revision: https://phab.mercurial-scm.org/D6706

automation: increase root volume size on Linux It is close to full in the AMI. I actually ran out of space running tests without increasing the volume size! Differential Revision: https://phab.mercurial-scm.org/D6716

automation: install Rust in Linux environment This will install Rust 1.31.1, 1.34.2, and whatever stable is at the time the install runs. We install 1.31.1 as our minimum supported Rust version (I think that's what we're currently targeting) and 1.34 because that's what Debian 10 is shipping. Differential Revision: https://phab.mercurial-scm.org/D6715

automation: update packages in requirements files We like keeping up to date. The content of the autogenerated files changed slightly because I used a newer version of `pip-compile` than what was used previously. Differential Revision: https://phab.mercurial-scm.org/D6714

automation: install latest Python versions This required bumping the pyenv commit so the new versions are available. Differential Revision: https://phab.mercurial-scm.org/D6713

upgrade: introduce the internal code for revlog cloning selection For now we still clone every single revlogs but all the selection mechanism is now in place in the lower layer. The next changesets will introduce the user interface part of the selection.

upgrade: introduce a _copyrevlog method This function copies a revlog from the old store to the new one, without re-adding all deltas manually. This will eventually save a lot of time when some revlog does not needs any conversions. Code actually using this will be introduced in later changesets.

upgrade: rename `_copyrevlogs` to `_clonerevlogs` The underlying revlog method is named `clone`, keeping the naming consistent seems clearer. This is motivated to clarify the difference with an (upcoming) function that simply copy revlog files as is.

upgrade: walk the source store file only once I don't expect this to have a significant performance impact, but it seems simpler and saner to do the operation only once and to keep the result around.

upgrade: always use full text if "full-add" mode is enable We should not be using a delta since the goal is to perform a full addition from scratch in all cases. Without this patch, `hg debugupgraderepo --optimize re-delta-fulladd --run` can crash.

byteify-strings: fix misalignment with multi-line parenthesis This improves the current fix to also take into account cases where the last line ended on the opening `(`, `[` or `{` and adds a regression test.

byteify-strings: add test for byteify-strings.py This tests the basic features expected from this script, some cases may not be covered yet. A future patch will demonstrate an issue with multi-line `(`, `[` and `{` alignment and propose a fix.

merge with stable

byteify-strings: add cli argument to handle `attr*()` when they are methods Certain code bases have useful utils that wrap the builtin functions, and are called like `util.setattr`.

byteify-strings: simplify default value for `--treat-as-kwargs`

byteify-strings: add --treat-as-kwargs argument to handle kwargs-like objects This argument will help extensions move to Python 3 as keyword arguments should not be byte-prefixed. Most of the time, code bases will call this object `kwargs`, but other conventions exist like `opts`, so it should make sense to allow for custom names. This is a best effort solution that does minimal static checking; cases like `options = [o for o in ('a', 'b', 'c') if kwargs.get(o)]` and other just as complicated will not be detected.

byteify-strings: add helpers to check for item access or method call These helpers will be used in a future patch, split for ease of review.

byteify-strings: add support for ignore comments Our simple token analysis is sometimes not clever enough, we need to be able to turn off our script for parts of the code. This change introduces three special comments: - `#no-py3-transform` to tell `byteify-strings` ignore the next line - `#py3-transform: off` to ignore everything until the end of the file - `#py3-transform: on` to stop ignoring The last two can be particularly useful within Python 2/3 compatibility files.

byteify-strings: handle triple quoted strings if they are not docstrings As with anything in this script, this is a best effort approach. Most of the time, when a triple quoted string is assigned to something, it's not a docstring.

byteify-strings: handle multi-line strings in _ensuresysstr The current implementation did not handle calls like `repo.ui.log("first line" "other line")` correctly.

fix: run fixer tools in the repo root as cwd so they can use the working copy This lets fixer tools do things like find configuration files, with the caveat that they'll only see the version of that file in the working copy, regardless of what revisions are being fixed. Differential Revision: https://phab.mercurial-scm.org/D6440

config: add defaultvalue template keyword This patch tries to fix one of the issues mentioned in issue6014. This adds a new `defaultvalue` template keyword to be used with `hg showconfig` to get the default value of the config item. Differential Revision: https://phab.mercurial-scm.org/D6704

merge with stable

module-policy: update rust extension import to use the new module policy Differential Revision: https://phab.mercurial-scm.org/D6677

transaction: leave unfinished without crashing when not properly released I think the transaction.__del__ is there just as a last resort in case we (or an extension) forgot to release the transaction. When that happens, the repo can (or will on Python 3?) get deleted before the transaction. This leads to a crash in test-devel-warnings.t on Python 3 because we tried to access repo.dirstate, where repo was retried from a weak reference. There's not much we can do here, but let's at least avoid the crash. The user will have run `hg recover` afterwards regardless. Differential Revision: https://phab.mercurial-scm.org/D6664

unshelve: add abort on using continue and interactive together `unshelve --continue --interactive` will not work as expected by the user as the mode of in-progress unshelve is preserved and cannot be overwritten. This patch makes `unshelve` to throw an error on using both `--continue` and `--interactive` together with `unshelve`. Differential Revision: https://phab.mercurial-scm.org/D6703

py3: add one more test to list of passing tests Found by buildbot. Differential Revision: https://phab.mercurial-scm.org/D6702

tests: sort imports in test-bookmarks-corner-case.t test-check-module-imports.t breaks on py3 without this change. Differential Revision: https://phab.mercurial-scm.org/D6701

fix: add some new test cases These cover a couple of behaviors we were testing at Google that weren't covered here before. Differential Revision: https://phab.mercurial-scm.org/D6698

unshelve: store information about interactive mode in shelvedstate This is a follow-up patch to 5162753c4c14. This makes `unshelve` stores the information about interactive mode on conflicts. Differential Revision: https://phab.mercurial-scm.org/D6679

unshelve: create a matcher only if required on creating unshelve ctx Differential Revision: https://phab.mercurial-scm.org/D6687

unshelve: changes how date is set on interactive mode On an interactive unshelve, the remaining changes are shelved again for later. This patch modifies the date of remaining shelved change to the time of interactive shelve. Differential Revision: https://phab.mercurial-scm.org/D6685

unshelve: unify logic around creating an unshelve changeset This is a follow-up patch to 5162753c4c14 on addressing reviews on the commit. This unifies the logic around creating an unshelve changeset. Differential Revision: https://phab.mercurial-scm.org/D6683

fix: ignore fixer tool configurations that are missing patterns This is to prevent a crash under the same circumstances. This is also to avoid data loss due to accidental application of a fixer tool to all files, if the matching logic somehow changed to that effect. Affecting all files until otherwise configured would be dangerous, and not very useful. We shouldn't abort because there may be other fixers, and it may still be useful to run them without having to adjust configuration. A user might not feel confident in changing configs, for example. Differential Revision: https://phab.mercurial-scm.org/D6693

fix: add a test case around the effect of cwd on pattern matching This was not covered by previous tests. It is related to a regression encountered at Google due to misconfiguration of [fix]. Differential Revision: https://phab.mercurial-scm.org/D6692

fix: remove support for :fileset sub-config in favor of :pattern Differential Revision: https://phab.mercurial-scm.org/D6691

fsmonitor: add support for extra `hg debuginstall` data This might make some things easier to debug, and for default bug report templates it'll help collect more data from users all at once. I don't actually need fsmonitor in our bug reports (we don't use it), but this demonstrates the utility of the preceding patches without having to add new things to core. Differential Revision: https://phab.mercurial-scm.org/D6682

debugcommands: add support for extensions adding their own debug info We've had a couple of cases where it'd be handy at Google to add data to `hg debuginstall`'s output. We've kludged around that at various times, but it seems reasonable to let extensions add their own data here so extension maintainers can get useful extra data. Differential Revision: https://phab.mercurial-scm.org/D6681

fsmonitor: refactor watchmanclient.client to accept ui and repo path This will make my next patch simpler. Differential Revision: https://phab.mercurial-scm.org/D6680

Added signature for changeset 181e52f2b62f

Added tag 5.1.2 for changeset 181e52f2b62f

merge: back out changeset a4ca0610c754 (parents order when grafting a merge) Turns out it's not enough to just swap parents, because when we do, there are unexpected bad side effects, such as a tracked file becoming untracked. These side effects need more code to be handled properly, but it's not written yet. Let's back this feature out from stable for now and some day implement it on default instead.

merge: respect parents order when using `graft` on a merge, this time for real See a4ca0610c754. potherp1 is a boolean variable that means "pother is ctx.p1", and parents is naturally [ctx.p1, ctx.p2]. pctx is always removed from parents, so if pctx is parents[0], then we end up using parents[1] as pother. To be true to its name, potherp1 should then be True only when pctx is at parents[1].

phabricator: don't abort if property writing fails during amending Currently if one of the writediffproperty calls fails due to network issues during the amending of commit messages to include the Diff. Rev. line, the transaction is aborted and rolled back. This means that the associations between the commits and the DREVs are lost for any already amended commits because the removal of the local tags isn't covered by the rollback. Differential Revision: https://phab.mercurial-scm.org/D6835

merge: respect parents order when using `graft` on a merge The previous code did not record the index of the replaced parent. It was always using the "graft" destination as `p1`. This could switch parents order in some situation (eg: some of the evolve evolving merge case). Recording and using the information fixes the issue in evolve. We are not aware of core commands calling graft in that fashion, so we could not build a simple test case for it using core commands.

tests: register test-merge-combination.t as small but slow run-tests.py use file size as an heuristic for test run time. The new `test-merge-combination.t` is a small file that do a lot of processing. As a result it tend to be scheduled really late but delay the full test run by a lot. On an example test run, the one-before-last test completed 279s after the start of the run, while `test-merge-combination.t` finished 355s after it. A 76s delay. This delay can be avoided since `test-merge-combination.t` only got started 175s after the start of the run.

test: allow different result for zstd compression (issue6188) test-repo-compengines fails on big-endian due to different file size, but the repo doesn't seem broken, so allow both sizes. Differential Revision: https://phab.mercurial-scm.org/D6787

Added signature for changeset a4e32fd539ab

Added tag 5.1.1 for changeset a4e32fd539ab

python-zstandard: apply big-endian fix (issue6188) This is a port of commit d4baf1f95b811f40773f5df0d8780fb2111ba6f5 from the upstream project to fix python-zstandard on 64-bit big-endian.

exchange: abort on pushing bookmarks pointing to secret changesets (issue6159) Until now, if there is a bookmark points to a changeset which is in secret phase, hg will push the bookmark, but not the changeset referenced by that bookmark. This leaves the server bookmarks in a bad state, because that bookmark now points to a revision that does not exist on the server. This patch makes hg to abort on such cases. Differential Revision: https://phab.mercurial-scm.org/D6731

tests: add test to demonstrate issue6159 Differential Revision: https://phab.mercurial-scm.org/D6740

packaging: add Bullseye, remove Jessie Jessie is oldoldstable now, and Bullseye is going to be the next stable (now testing). We're continuing to support the current oldstable, stable and testing releases. Differential Revision: https://phab.mercurial-scm.org/D6762

packaging: add Cosmic and Disco, remove Trusty and Artful - Trusty was publicly supported until 2019-04-30 - Artful was publicly supported until 2018-07-19 - Cosmic was publicly supported until 2019-07-18 - Disco will be publicly supported until 2020-01 Cosmic is officially out-of-date, but since it still may be in use, and because we didn't add it when it first came out, I think it would be nice to support it until the next time somebody decides to update this list of Ubuntu releases. Differential Revision: https://phab.mercurial-scm.org/D6761

makefile: run Rust tests if cargo is installed While no particular minimum toolchain version is targeted as of yet, this serves as a first step to make more people/machines run the Rust tests.

tests: use `tr -d` and not `tr --delete` as the latter is absent on BSD tr(1) Differential Revision: https://phab.mercurial-scm.org/D6729

fncache: make debugrebuildfncache not fail on broken fncache The code reading the fncache changed in 5.0, to complain if the file is not \n terminated. This makes apparent the fact that the fncache gets corrupted. Make it possible to recover, instead of having `hg debugrebuildfncache` failing by saying `(run hg debugrebuildfncache)`. The corruption itself is most likely due to hg not using fsync in general, and so various bad things can happen. Here, the reported problems happened when running out of disk space. So I suspect that because the fncache is much bigger than the average commit/pull, when running out of disk space, the bulk of the pull may succeed, but the new fncache may get half-written and still renamed into place. Differential Revision: https://phab.mercurial-scm.org/D6722

fncache: show that debugrebuildfncache is partly broken Differential Revision: https://phab.mercurial-scm.org/D6721

test: further fixes to matching for run-tests.py bug The fix in bac24a8a095a did not fix the full issue, because the extra number also eat some of the separator space. Since we are already matching arbitrary number of space, we easily fix the matching.

demandimport: explicitly declare `_session` at the module level The `_session` module level variable is set within a function using the `global` keyword. This confuses my `test-check-pyflakes.t`. Explicitly declaring the variable at the top level solves the issue (and seems absolutely reasonable).

tests: give more room for slowness in test-run-tests.t The test expected any run-test.py run to end in less than 10 seconds. On slower loaded CI machine, this gets slower than that. We give a bit more room to the regexp.

relnotes: copy "next" to "5.1" and clear "next" To avoid merge conflicts, we want to avoid modifying the file on multiple branches in parallel. This patch is therefore meant to be applied to the stable branch and then quickly be merged to default (at least before edits are made to relnotes/next there). Another option would have been to copy the file on the stable branch and to clear it on the default branch. However, that still results in conflicts if the copy is edited on the stable branch (Mercurial would try to apply the changes from the default branch to it). We could also delete the file in one commit and recreate it in another commit. However, Mercurial is quite inconsistent in what it considers a break in history (see test-copies-unrelated.t), so I'd like to avoid that. Differential Revision: https://phab.mercurial-scm.org/D6705

automation: push changes affecting .hgtags When I went to build the 5.1 tag using the in-repo automation, the automatic version calculation failed to deduce the clean 5.1 version string because we had only pushed the changeset corresponding to the 5.1 tag and not the changeset containing the 5.1 tag. So from the perspective of the remote repo, the 5.1 tag didn't exist yet and automatic version deduction failed. This commit changes the `hg push` to also push all changesets affecting the .hgtags file, ensuring the remote has up-to-date tags information. I tested this by creating a local draft changeset with a dummy tag value on a different DAG head and instructed the automation to build a revision that didn't have this change to .hgtags. The tag was successfully pushed and the built package had a version number incorporating that tag. Sending this to stable so the 5.1.1 automation hopefully "just works."

bookmarks: actual fix for race condition deleting bookmark This is a simple but efficient fix to prevent the issue tested in `test-bookmarks-corner-case.t`. It might be worth pursuing a more generic approach where filecache learn to depend on each other, but that would not be suitable for stable. The issue is complicated enough that I documented the race and its current solution as inline comment. See this comment for details on the fix.

strip: access bookmark before getting a reference to changelog Bookmark access might invalidate the current changelog (to make sure both are in a reasonable synchronisation state). So we should grab the reference to changelog after we access bookmark. Otherwise we risk using a dead object for the whole strip process. (note: this dead object business probably requires a new layers of checking)

test: use a more verbose output in the test While debugging some test failure, I released the test never checks if the relevant changesets were preserved. So I am updating the test from `hg parents` usage to `hg log -G` with a special template. This increase the area covered by the test and clarify the test failures.

Added signature for changeset e91930d712e8

Added tag 5.1 for changeset e91930d712e8

automation: execute powershell when connecting For some reason, the ability to execute PS scripts appears to come online after the ability to execute regular command scripts. This is creating race conditions when connecting to instances resulting in our wait_for_winrm() returning before PS is available leading to an exception being thrown in other code. Let's change the client connection code to execute a minimal PS script so we can try to trap the exception in wait_for_winrm().

automation: allow exit code of 1 for `hg push` `hg push` exits 1 for no-ops. No-op pushes should be fine in the context of automation.

curses: do not setlocale() at import time (issue5261) setlocale() can break date formatting/parsing functions because they are locale dependent. We should avoid doing setlocale() as possible. This patch moves setlocale() just before curses.wrapper(), which function is documented to "initialize curses." I don't know the details about the curses initialization, but I *think* this would work as well. Maybe we can extract a curses setup function later. https://www.mercurial-scm.org/pipermail/mercurial-devel/2019-February/128788.html

contrib: install Python 3.8b2 instead of 3.8a2 Let's install the most recent Python 3.8 distribution. Differential Revision: https://phab.mercurial-scm.org/D6674

automation: make Windows base image name configurable Since automation broke in the middle of the 5.0 release cycle, there's a good chance it will break again in the future. While a robust solution might be to search for all available images and choose the newest one, it does seem useful to be able to explicitly choose the name of the image to find and use so users can opt in to using a different image. This commit implements that functionality. Differential Revision: https://phab.mercurial-scm.org/D6673

automation: extract strings to constants Differential Revision: https://phab.mercurial-scm.org/D6672

automation: use newer Windows base image It looks like the old base image disappeared. Let's use a newer image that exists today. Differential Revision: https://phab.mercurial-scm.org/D6671

copies: fix crash on in changeset-centric tracing from commit to itself When we trace copies from a changeset to itself, the "work" queue ends up empty and we hit the "assert False" after it. It was only the last of the three added tests that failed before this patch. That is because the other two cases have fast paths, so _committedforwardcopies() is never reached. Differential Revision: https://phab.mercurial-scm.org/D6675

unshelve: add help text on --interactive in verbose mode This is a follow-up patch to rHG9eace8d6d537. This modifies the help text of unshelve in verbose mode to mention the details about `--interactive` flag. Differential Revision: https://phab.mercurial-scm.org/D6676

amend: stop committing unrequested file reverts (issue6157) Differential Revision: https://phab.mercurial-scm.org/D6667

amend: add a test for a simplified version of issue6157 Differential Revision: https://phab.mercurial-scm.org/D6666

py: error out if a "skip" character was given with non-dict to util.dirs() util.dirs() keeps track of the directories in its input collection. If a "skip" character is given to it, it will assume the input is a dirstate map and it will skip entries that are in the given "skip" state. I think this is used only for skipping removed entries ("r") in the dirtate. The C implementation of util.dirs() errors out if it was given a skip character and a non-dict was passed. The pure implementation simply ignored the request skip state. Let's make it easier to discover bugs here by erroring out in the pure implementation too. Let's also switch to checking for the dict-ness, to make the C implementation (since that's clearly been sufficient for many years). This last change makes test-issue660.t pass on py3 in pure mode, since the old check was for existence of iteritems(), which doesn't exist on py3. Differential Revision: https://phab.mercurial-scm.org/D6669

py3: fix incorrect fix of test-setdiscovery.t in eb27d9eee2cc Both places should have been changed from 185 to 187. Differential Revision: https://phab.mercurial-scm.org/D6668

Added signature for changeset e386b5f4f836

Added tag 5.1rc0 for changeset e386b5f4f836

merge default into stable for 5.1 release

rust-filepatterns: unescape comment character property There were multiple issues in the original implementation: a. the local variable "line" dropped soon after replace_slice() applied b. replace_slice() was noop since br"\#".len() != b"#" This patch uses bytes::Regex::replace_all() since it seems the simplest way to replace bytes of arbitrary length, and I don't think we have to avoid using Regexp here.

rust-filepatterns: use literal b'#' instead of cast

rust-filepatterns: fix type of warnings tuple to (bytes, bytes) Otherwise warn() in match.py would fail if the warning contains non-ASCII character. We might want to add a thin ByteString wrapper around Vec<u8> to implement ToPyObject<ObjectType = PyBytes>, but I'm not sure.

hgignore: add escape syntax test for glob patterns The last example, [\#], is what the rust implementation fails to parse. The other escapes can be removed by regexp engine or _globre().

hgignore: add a few more weird patterns to test case

hgignore: update \-escape test to reflect actual behavior "\\<char>" is not an escape character but "\\" + <char>.

py3: add a b'' prefix in tests/test-convert-identity.t Differential Revision: https://phab.mercurial-scm.org/D6662

lookup: don't use "00changelog.i@None" when lookup of prefix fails We were shadowing the "node" variable, so we always passed None to the LookupError instead of the node we meant to pass. (This showed up in py3 tests since py3 doesn't like to format None using "%s".) Differential Revision: https://phab.mercurial-scm.org/D6661

py3: fix test-setdiscovery.t on Python 3 by conditionalizing two lines I'm not clear why this behaves very slightly differently on Python 3, but I'm also not concerned about it. Differential Revision: https://phab.mercurial-scm.org/D6658

commands: removed part of description from abort and continue The description for registration of new `continuefunc` or `abortfunc` is removed as it is not required from user perspective. Differential Revision: https://phab.mercurial-scm.org/D6660

tests: glob over some timing numbers in test-shelve.t The Windows bot is slow enough that it was 2s in the first hunk. Differential Revision: https://phab.mercurial-scm.org/D6663

py3: another passing test Differential Revision: https://phab.mercurial-scm.org/D6656

cleanup: remove redundant import For some reason the import checker only caught this on py3. Differential Revision: https://phab.mercurial-scm.org/D6657

shelve: modify help text on --interactive We now have `unshelve --interactive` after rHG5162753c4c14. So, the help text on `shelve --interactive` suggesting that it only works for `shelve` can be removed. Differential Revision: https://phab.mercurial-scm.org/D6654

unshelve: mark unshelve interactive as experimental This is a follow-up patch to rHG5162753c4c14. We have the logic for interactive unshelve under `_rebaserestorecommit()`. So, we might get conflicts even if there are conflicting changes other than selected changes by the user. We should mark unshelve `--interactive` as `EXPERIMENTAL` until we solve this issue. Differential Revision: https://phab.mercurial-scm.org/D6653

commit: improve the files field of changelog for merges Currently, the files list of merge commits repeats all the deletions (either actual deletions, or files that got renamed) that happened between base and p2 of the merge. If p2 is the main branch, the list can easily be much bigger than the change being merged. This results in various problems worth improving: - changelog is bigger than necessary - `hg log directory` lists many unrelated merge commits, and `hg log -v -r commit` frequently fills multiple screens worth of files - it possibly slows down adjustlinkrev, by forcing it to read more manifests, and that function can certainly be a bottleneck - the server side of pulls can waste a lot of time simply opening the filelogs for pointless files (the constant factors for opening even a tiny filelog is apparently pretty bad) So stop listing such files as described in the code. Impacted merge commits and their descendants get a different hash than they would have without this. This doesn't seem problematic, except for convert. The previous commit helped with that in the hg->hg case (but if you do svn->hg twice from scratch, hashes can still change). The rest of the description is numbers. I don't have much to report, because recreating the files list of existing repositories is not easy: - debugupgradeformat and bundle/unbundle don't recreate the list - export/import tends to choke quickly applying patches or on description that contain diffs, - merge commits from the convert extension don't have the right files list for reasons orthogonal to the current commit - replaying the merge with hg update/hg merge/hg revert --all/hg commit can end up failing in hg revert - I wasn't sure that using debugsetparents + debugrebuilddirstate would really build the right thing I measured commit time before and after this change, in a case with no files filtered out, several files filtered out (no difference) and 5k files filtered out (+1% time). Recreating the 100 more recent merges in a private repo, the concatenated uncompressed files lists goes from 1.12MB to 0.52MB. Excluding 3 merges that are not representative, then the size goes from 570k to 15k. I converted part of mozilla-central, and observed file list shrinking quite a bit too, starting at the very first merge, 733641d9feaf, going from 550 files to 10 files (although they have relatively few merges, so they probably wouldn't care). Differential Revision: https://phab.mercurial-scm.org/D6613

convert: add a config option to help doing identity hg->hg conversion I want to change the computation of the list of files modified by a commit. In principle, this would simply change a cache. But since this information is stored in commits rather than a cache, changing it means changing commit hashes (going forward). Some users rely on the convert extension from hg to hg not changing hashes when nothing changes (usually). Allow these users to preserve hashes despite changes to the changelog files computation by reusing these files lists when the manifest is unchanged (since these files list are derived from the manifest). Differential Revision: https://phab.mercurial-scm.org/D6643

tests: show the files fields of changelogs for many merges I don't think there's coverage for many of the subtle cases, and I found it hard to understand what the code is doing by reading it. The test takes 40s to run on a laptop, or 9s with --chg. I have yet to find a description of what the files field is supposed to be for merges. I thought it could be one of: 1. the files added/modified/removed relative to p1 (wouldn't seem useful, but `hg diff -c -r mergerev` has this behavior) 2. the files with filelog nodes not in either parent (i.e., what is needed to create a bundle out of a commit) 3. the files added/removed/modified files by merge itself [1] It's clearly not 1, because file contents merges are symmetric. It's clearly not 2 because removed files and exec bit changes are listed. It's also not 3 but I think it's intended to be 3 and the differences are bugs. Assuming 3, the test shows that, for merges, the list of files both overapproximates and underapproximates. All the cases involve file changes not in the filelog but in the manifest (existence of file at revision, exec bit and file vs symlink). I didn't look at all underapproximations, but they looked minor. The two overapproximations are problematic though because they both cause potentially long lists of files when merging cleanly. [1] even what it means for the merge commit itself to change a file is not completely trivial. A file in the merge being the same as in one of the parent is too lax as it would consider that merges change nothing when they revert all the changes done on one side. The criteria used in the test and in the next commit for "merge didn't touch a file" is: - the parents and the merge all have the same file - or, one parent didn't touch the file and the other parent contains the same file as the merge Differential Revision: https://phab.mercurial-scm.org/D6612

phabricator: handle local:commits time being string or int When setting local:commits arcanist has different behaviour depending on whether the repo is git or hg. With hg it sets the time as a number, since it calls PHP's strtotime on the value, but with git it sets it as a string. Normally this wouldn't be an issue since phabread wouldn't be interacting with Phabricator Revisions for git repos, but Mozilla has a secondary workflow for git users that uses the git-cinnabar tool to interact with their hg repos. When a git-cinnabar user uses the moz-phab tool to submit patches for mozilla-central it makes use of Mozilla's fork of arcanist, which works with their local git version of m-c, and thus sets the local:commit time as a string, and then translates the commit hashes. Currently when encountering such DREVS phabread dies with "TypeError: %d format: a number is required, not str". phabsend also used to set it as a string but wouldn't have encountered the issue with its own DREVs since it would read hg:meta first. Differential Revision: https://phab.mercurial-scm.org/D6650

phabricator: demonstrate broken phabread on string local:commit times Differential Revision: https://phab.mercurial-scm.org/D6649

unshelve: add interactive mode Until now, there is no way to `unshelve` selected changes only from the stored shelve as given in issue6162. This patch makes `unshelve` perform with certain changes only by adding an interactive mode. Differential Revision: https://phab.mercurial-scm.org/D6596

blackbox: disable extremely verbose logging (issue6110) This is maybe not the best way to go about fixing this, but anything is better than the status quo. Differential Revision: https://phab.mercurial-scm.org/D6611

continue: added support for unshelve This patch adds the support for `ushelve` in `hg continue` plan. `hgcontinueunshelve()` has been created for independent calls. In case an interrupted unshelve is resumed via hg continue the shelvedstate needs to be loaded seperately. This has been ensured by `_loadunshelvedstate()` `hgcontinueunshelve()` is then registered as `continuefunc` for state detection API. Results are shown as tests. Differential Revision: https://phab.mercurial-scm.org/D6652

continue: added support for rebase This adds support of rebase to hg continue plan. An independent continue logic for rebase is created under continuerebase() function. For this a seperate rebaseruntime object is created under the function to handle an interrupted rebasestate. Results of tests are shown. Differential Revision: https://phab.mercurial-scm.org/D6646

continue: added logic for hg continue This is part of GSoC19 project `Implement abort and continue commands`. This patch is part of the continue plan. This adds the basic logic for hg continue. This command aborts an multistep operation like graft, histedit, rebase, transplant and unshelve if they are in an unfinished state. The first part of the logic is determining the unfinished operation from the state detection API under statemod. This API is extended to support hg continue by adding a method to register the abort logic as a function (here continuefunc). Once the unfinished operation is determined the registered logic is used to resume the command in case it is interrupted. The benefit of this kind of framework is that any new extension developed can support hg continue by registering the command and logic under statedetection API. hg continue currently supports --dry-run/-n flag only. It is used to dry run hg abort Differential Revision: https://phab.mercurial-scm.org/D6645

rust-utils: remove buggy assertion While this assertion had good intentions, it broke existing behavior with a nasty panic. Differential Revision: https://phab.mercurial-scm.org/D6651

rust-utils: add docstrings and doctests for utils.rs Differential Revision: https://phab.mercurial-scm.org/D6635

rust: switch hg-core and hg-cpython to rust 2018 edition Many interesting changes have happened in Rust since the Oxidation Plan was introduced, like the 2018 edition and procedural macros: - Opting in to the 2018 edition is a clear benefit in terms of future proofing, new (nice to have) syntactical sugar notwithstanding. It also has a new non-lexical, non-AST based borrow checker that has fewer bugs(!) and allows us to write correct code that in some cases would have been rejected by the old one. - Procedural macros allow us to use the PyO3 crate which maintainers have expressed the clear goal of compiling on stable, which would help in code maintainability compared to rust-cpython. In this patch are the following changes: - Removing most `extern crate` uses - Updating `use` clauses (`crate` keyword, nested `use`) - Removing `mod.rs` in favor of an aptly named module file Like discussed in the mailing list ( https://www.mercurial-scm.org/pipermail/mercurial-devel/2019-July/132316.html ), until Rust integration in Mercurial is considered to be out of the experimental phase, the maximum version of Rust allowed is whatever the latest version Debian packages. Differential Revision: https://phab.mercurial-scm.org/D6597

rust-utils: use new find_dirs iterator In cad3dde7a573, the `find_dirs` util was introduced, but the second changeset that made use of it didn't apply. This change fixes the issue. Differential Revision: https://phab.mercurial-scm.org/D6639

inno: correct the path display in a literal block of the readme Otherwise, the path components allrantogether. Differential Revision: https://phab.mercurial-scm.org/D6648

copies: remove unnecessary override of p[12]copies() in workingctx The implementation is identical to the version inherited from basectx. Differential Revision: https://phab.mercurial-scm.org/D6647

tests: properly position conditional output on Windows in test-subrepo.t The test runner doesn't always guess the right location when optional output is missing. This goes with f6540aba8e3e. Differential Revision: https://phab.mercurial-scm.org/D6640

abort: removed labels argument from abortmerge() Labels are used to label the code that belongs to `working copy` and `merge rev` in case of a conflicted state. No such labelling is required while aborting merge as conflicted parts are reverted to normal. Differential Revision: https://phab.mercurial-scm.org/D6638

py3: source-transform only call-sites of iteritems(), not definitions branchmap.branchcache, among other classes, defines a iteritems(). That currently gets replaced by items() by the source transformer. That makes it harder for extensions to work with both py2 and py3, since they have to call either items() or iteritems() on branchcache. Let's not replace definitions of iteritems() (and itervalues()) and only replace the call-sites. We need to also add an items() alias to branchcache (etc) so our transformer call-sites will find it. Differential Revision: https://phab.mercurial-scm.org/D6641

py3: fix formatting of branchmap log messages with repo.filtername=None `"%s" % None` does not work on py3. I've extracted a little function for producing a formatted message given the filter name. Differential Revision: https://phab.mercurial-scm.org/D6644

automation: correct the path separator in LIBPATH on Windows I haven't tried building the x86 installer, but happened to notice this when working on the thg installer. Experimenting in PowerShell seems to show that LIBPATH was expanded at the end, but with ':' between, it effectively corrupted `${root}\WinSDK\Lib` and the first path in LIBPATH. Differential Revision: https://phab.mercurial-scm.org/D6642

abort: added support for merge This adds support of `hg merge --abort` to `hg abort` plan. This involves refactoring `hg.merge` into two different functions removing the abort logic of `merge` from `hg.merge` and then creating a seperate `hg.abortmerge` to handle the abort logic so that the abortion of merge can be called independently. `hg.abortmerge` is then registered as `abortfunc` for the state detection API so that `commands.abort` can use it to deal with an unfinished merge operation. Results are shown as tests. Differential Revision: https://phab.mercurial-scm.org/D6588

abort: added support for unshelve This patch adds the support for shelve in `hg abort` plan. For this the logic to load a `shelvedstate` and the error handling for it had been shifted to a seperate function `_loadunshelvedstate()`. This returns a tuple with `state` file and `opts.` `hgabortunshelve()` has been created for independent calls. In case abortion of `unshelve` is called via `hg abort` the `shelvedstate` needs to be loaded seperately. This has been ensured by `_loadunshelvedstate()` `hgabortunshelve()` is then registered as `abortfunc` for state detection API. Results are shown as tests. Differential Revision: https://phab.mercurial-scm.org/D6579

unshelve: changed Corruptedstate error msg from ui.warn to error.Abort This changes the message type of Corruptedstate error in case of `hg unshelve --abort` to error.Abort from warning message. This is done so as to avoid the return statement after the warning. Differential Revision: https://phab.mercurial-scm.org/D6636

mq: fix for merge detection methods Differential Revision: https://phab.mercurial-scm.org/D6548

py3: store _origdoc as str Since __doc__ is str, it seems natural that _origdoc also is. Differential Revision: https://phab.mercurial-scm.org/D6623

copies: follow copies across merge base without source file (issue6163) As in the previous patch, consider these two histories: @ 4 'rename x to y' | o 3 'add x again' | o 2 'remove x' | | o 1 'modify x' |/ o 0 'add x' @ 4 'rename x to y' | o 3 'add x again' | | o 2 'modify x' | | | o 1 'add x' |/ o 0 'base' We trace copies from the 'modify x' commit to commit 4 by going via the merge base (commit 0). When tracing file 'y' (_tracefile()) in the first case, we immediately find the rename from 'x'. We check to see if 'x' exists in the merge base, which it does, so we consider it a valid copy. In the second case, 'x' does not exist in the merge base, so it's not considered a valid copy. As a workaround, this patch makes it so we also attempt the check in mergecopies's base commit (commit 1 in the second case). That feels pretty ugly to me, but I don't have any better ideas. Note that we actually also check not only that the filename matches, but also that the file's nodeid matches. I don't know why we do that, but it was like that already before I rewrote mergecopies(). That means that the rebase will still fail in cases like this (again, it already failed before my rewrite): @ 4 'rename x to y' | o 3 'add x again with content X2' | o 2 'remove x' | | o 1 'modify x to content X2' |/ o 1 'modify x to content X1' | o 0 'add x with content X0' Differential Revision: https://phab.mercurial-scm.org/D6604

copies: filter invalid copies only at end of pathcopies() (issue6163) copies._filter() filters out copies whose source file does not exist in the start commit or whose target file does not exist in the end commit. We do that after chaining copies with dirstate copies or backward renames from another branch. We also do at the end of the changeset-centric copy tracing. The filtering means that we will remove copies to/from files that did not exist in some intermediate commit. That is inconsistent with what we do if a file has been deleted and then re-added (we allow updating across that). Copying the two first examples from issue6163: @ 4 'rename x to y' | o 3 'add x again' | o 2 'remove x' | | o 1 'modify x' |/ o 0 'add x' @ 4 'rename x to y' | o 3 'add x again' | | o 2 'modify x' | | | o 1 'add x' |/ o 0 'base' When doing `hg rebase -r 1 -d 4` in the first case, it succeeds, but `hg rebase -r 2 -d 4` in the second case does not. That's because we chain and filter via commit 0, which does not have file 'x' in the second case. IMO, that's clearly inconsistent. So this patch removes the filtering step so it only happens at the end. If a file was temporarily removed, whether via a merge base or not, it will now still be considered the same file. That fixes issue6163 for the changeset-centric case. Differential Revision: https://phab.mercurial-scm.org/D6603

copies: inline _chainandfilter() to prepare for next patch Differential Revision: https://phab.mercurial-scm.org/D6602

copies: remove most early returns from pathcopies() and _forwardcopies() I want to split up _chainandfilter() more so the call to _filter() consistently happens at the end of pathcopies(). This prepares for that change. Differential Revision: https://phab.mercurial-scm.org/D6601

copies: move short-circuiting of dirstate copies out of _forwardcopies() I'd like to move the filtering of copies we do after chaining to the end of all chaining (in a single place in pathcopies()). One problem that came up when trying that was that we allow things like `hg cp -f <file> <existing file>` so the user can later amend that in. Filtering at the end would mean that we remove those copies. That would break `hg st -C`. This patch therefore moves the short-circuiting of dirstate copies into pathcopies() so we can more easily handle the dirstate-only case differently. I initially thought this might change some behavior when the user does `hg status --rev 'wdir()' --rev .` during an uncommitted merge, since _backwardrenames() would reverse the copies in that case. However, I couldn't come up with a test case where it made a difference. Differential Revision: https://phab.mercurial-scm.org/D6600

tests: add more tests of copy tracing with removed and re-added files We had a test where the destination of a copy was removed and then added back. This patch adds similar cases where the break in history instead happens to the source file. There are three versions of this: 1. The break happens before the rename. 2. The break happens on a branch parallel to the rename (where copy tracing is done via the merge base) 3. The source is added on each side of the merge base. The break in history is thus in the form of a deletion when going backwards to the merge base and the re-add happens on the other branch. I've also added calls to `hg graft` in these cases to show the breakage in issue 6163. Another factor in these cases is matching nodeid (checked in copies._tracefile()). I've made two copies each of the cases to show the impact of that. One of these is the same as a test in test-rename-merge1.t, so I also deleted that test from there. Some of these tests currently fail, where "fail" is based on my current thinking of how things should work. I had initially thought that we should be more strict about not tracing copies across commits where the file did not exist, but issue 6163 made me reconsider. The only test case here that behaved differently in 4.9 is the exact case reported in issue 6163. Differential Revision: https://phab.mercurial-scm.org/D6599

tests: split out tests for unrelated copy source/target into separate file I've realized only recently how many cases there are where a file is treated differently if it's considered "related" to another file (not deleted and re-added). I'll add more tests for some of these cases soon. Differential Revision: https://phab.mercurial-scm.org/D6598

subrepos: make last line of prompts <40 english chars (issue6158) Differential Revision: https://phab.mercurial-scm.org/D6572

largefiles: make last line of prompts <40 english chars (issue6158) Differential Revision: https://phab.mercurial-scm.org/D6571

rust-dirstate: add helper to iterate ancestor paths This is modeled after std::path::Path::ancestors(). find_dirs(b"") yields b"" because Mercurial's util.finddirs() works in that way, and the test case for DirsMultiset expects such behavior.

tests: update test-commit-interactive.t for no-execbit platforms These changes correspond with f802a75da585. Differential Revision: https://phab.mercurial-scm.org/D6624

abort: added support for histedit This patch adds the support for `histedit` in `hg abort` plan. As seperate `hgaborthistedit()` function is created to handle independent calls for abortion of `histedit`. This function is then registered as `abortfunc` for state detection API. hg abort in case of `histedit` also supports ` history-editing-backup` config option. Results are shown as tests. Differential Revision: https://phab.mercurial-scm.org/D6582

abort: added support for rebase This adds support of `rebase` to `hg abort` plan. An independent abort logic for `rebase` is created under `abortrebase()` function. For this a seperate `rebaseruntime` object is created under the function to handle an unfinished `rebasestate` and abort that using abort logic under `_prepareabortorcontinue`. Results of tests are shown. Differential Revision: https://phab.mercurial-scm.org/D6568

abort: added support for graft This adds support of `graft` to `hg abort` plan. The patch creates a seperate function `cmdutil.hgabortgraft` so that abort logic for graft can be called independently. This logic is registered to the statedetection API as `abortfunc`. Results are shown as tests. Differential Revision: https://phab.mercurial-scm.org/D6567

abort: added logic for of hg abort This is part of `GSoC19` project `Implement abort and continue commands`. This patch is part of the `abort plan`. This adds the basic logic for `hg abort`. This command aborts an multistep operation like graft, histedit, rebase, merge and unshelve if they are in an unfinished state. The first part of the logic is determining the unfinished operation from the state detection API under `statemod`. This API is extended to support `hg abort` by adding a method to register the abort logic as a function (here `abortfunc`). Once the unfinished operation is determined the registered logic is used to abort the command. The benefit of this kind of framework is that any new extension developed can support `hg abort` by registering the command and logic under statedetection API. `hg abort` currently supports `--dry-run/-n` flag only. It is used to dry run `hg abort` Further patches sequentially add support for `graft`, `rebase`, `unshelve`, `histedit` and `merge`. Differential Revision: https://phab.mercurial-scm.org/D6566

merge with stable

merge: disallow merge abort in case of an unfinished operation (issue6160) This patch disallows `hg merge --abort` in case an operation of higher precedence i.e unshelve, rebase, histedit are in unfinished states. This is done so as to avoid partial abort of these operations in case merge abort is called at an interrupted step. The patch adds a `cmdutil.getunfinishedstate` function which checks for operations under progress and returns a `statecheck` object for it. Differential Revision: https://phab.mercurial-scm.org/D6607

relnotes: document new range-select mechanism in crecord Differential Revision: https://phab.mercurial-scm.org/D6622

statecheck: updated docstrings related to afterresolvedstates Differential Revision: https://phab.mercurial-scm.org/D6606

extdata: avoid crashing inside subprocess when we get a revset parse error Differential Revision: https://phab.mercurial-scm.org/D6616

extdata: demonstrate bad behavior when a subprocess emits garbage Differential Revision: https://phab.mercurial-scm.org/D6615

py3: don't run source transformer on hgext3rd (extensions) It's unclear why the source transformer runs on hgext3rd. It's been like that since it was introduced in 1c22400db72d (mercurial: implement a source transforming module loader on Python 3, 2016-07-04), and that commit didn't say anything about it (but it says that it doesn't have "support [...] for extensions"). I find that the current handling of hgext3rd just makes it harder to convert extensions to Python 3. It makes you convert a bunch of strings passed to getattr() and kwargs[] to r'' that could otherwise have been left alone. It's also really confusing that the source transformer runs when you import the extension as "extensions.foo=", but not as "extension.foo=/some/path". I suppose there is small number of (very simple) extensions that would have worked without this patch that would now be broken. It seems okay to me to break those. Differential Revision: https://phab.mercurial-scm.org/D6614

crecord: provide 'X' as a range-select mechanism Differential Revision: https://phab.mercurial-scm.org/D6621

crecord: make KEY_ENTER usable in tests (by not updating UI) Differential Revision: https://phab.mercurial-scm.org/D6620

crecord: fix if -> elif when handling key presses This shouldn't actually change any behavior, I only noticed it since I started using KEY_UP in tests, and it was complaining when it got down to the ^L handler that initscr hadn't been called yet. Differential Revision: https://phab.mercurial-scm.org/D6619

crecord: add "x" alias for space, remove test-only "TOGGLE" alias Differential Revision: https://phab.mercurial-scm.org/D6618

crecord: stop using test-only "X" as alternative for "c" Differential Revision: https://phab.mercurial-scm.org/D6617

graft: moved abortgraft and readgraft to cmdutil This patch moves `abortgraft` and `readgraft` to `cmdutil`. Various callers are updated accordingly. This is done because these serve as ulitlity functions for command `graft` and so that new functions regarding graft can be built from them. Differential Revision: https://phab.mercurial-scm.org/D6608

cleanup: use named constants for second arg to .seek() Differential Revision: https://phab.mercurial-scm.org/D6556

patch: use a short, fixed-size message for last line of prompt (issue6158) See issue6158 and the previous commit for examples of what might go wrong if we have some combinations of readline version and terminal and need to wrap the line. Briefly: readline may not display the beginning of the last line of the prompt, or it may print over it with the end of the prompt, making it difficult for users to know what's going on. Differential Revision: https://phab.mercurial-scm.org/D6563

filemerge: make last line of prompts <40 english chars (issue6158) I've chosen <40 as the target so that other languages that may have a 2x blowup in character count can still have a chance to fit into an 80 column screen. Previously, we would show a prompt like: ``` keep (l)ocal [dest], take (o)ther [source], or leave (u)nresolved for some/potentially/really/long/path? ``` On at least some systems, if readline was in use then the last line of the prompt would be wrapped strangely if it couldn't fit entirely on one line. This strange wrapping may be just a carriage return without a line feed, overwriting the beginning of the line; example (100 columns wide, 65 character filename, and yes there's 10 spaces on the end, I assume this is to handle the user inputting longest word we provide as an option, "unresolved"): ``` ng/dir/name/that/does/not/work/well/with/readline/file.txt? ave (u)nresolved for some/lon ``` In some cases it may partially wrap onto the next line, but still be missing earlier parts in the line, such as below (60 columns wide, 65 character filename): ``` rev], or leave (u)nresolved for some/long/dir/name/that/do s/not/work/well/with/readline/file.txt? ``` With this fix, this looks like this on a 60 column screen: ``` tool vim_with_markers (for pattern some/long/dir/name/that/d oes/not/work/well/with/readline/file.txt) can't handle binar y tool meld can't handle binary tool vim_with_markers can't handle binary tool internal:merge3 can't handle binary tool merge can't handle binary no tool found to merge some/long/dir/name/that/does/not/work /well/with/readline/file.txt file 'some/long/dir/name/that/does/not/work/well/with/readli ne/file.txt' needs to be resolved. You can keep (l)ocal [working copy], take (o)ther [merge rev ], or leave (u)nresolved. What do you want to do? ``` Differential Revision: https://phab.mercurial-scm.org/D6562

tweakdefaults: make hg resolve require --re-merge flag to re-merge Pulkit suggested it in https://phab.mercurial-scm.org/D4379, and a discussion with Octobus people reminded me that people still use the error-prone default behavior of `hg resolve`. Differential Revision: https://phab.mercurial-scm.org/D6610

unshelve: rename _dounshelve() to dounshelve() This is a follow-up patch to 3de4f17f4824. Differential Revision: https://phab.mercurial-scm.org/D6605

rust: remove Deref in favor of explicit methods Differential Revision: https://phab.mercurial-scm.org/D6593

rust: simplify overly complicated expression Differential Revision: https://phab.mercurial-scm.org/D6592

rust: run rfmt on all hg-core/hg-cpython code Differential Revision: https://phab.mercurial-scm.org/D6591

changelog: fix handling of empty copy entries in changeset Before this patch, when an empty value was found in the changeset, we would get a ValueError, which would result in None being returned for addedfiles/removedfiles and p1copies/p2copies. That made 278dcb24e535 (copies: write empty entries in changeset when also writing to filelog, 2019-04-23) ineffective at helping the read path not look for copies in the filelogs. Differential Revision: https://phab.mercurial-scm.org/D6595

relnotes: document the new --force-close-branch flag Differential Revision: https://phab.mercurial-scm.org/D6590

py3: hack around inconsistency of type of name passed to DNSQuestion I don't like this patch but this is the easiest way I could fix it. There are some callers which pass name which is bytes, some pass name which is str. I just encode() that if that's str. This does makes test-paths.t pass, but I am not confident whether the whole of zeroconf will work on py3 or not. Differential Revision: https://phab.mercurial-scm.org/D6511

py3: add r'' prefixes and do ('%d' % int) instead of str(int) This addresses more failures related to zeroconf on py3. Differential Revision: https://phab.mercurial-scm.org/D6510

zeroconf: port to Python 3 Since we're using the source transformer on Python 3, calls into Zeroconf and return values from it are generally bytes. But various socket functions require str on Python 3. This commit contains enough changes to coerce test-paths.t into passing on Python 3. I suspect there are still a handful of bugs on Python 3. But the tests do pass. Differential Revision: https://phab.mercurial-scm.org/D5805

copies: return only path from _tracefile() since that's all caller needs Differential Revision: https://phab.mercurial-scm.org/D6587

extensions: add shelve to _builtin This is a follow-up patch to 3de4f17f4824. This adds `shelve` to `extensions._builtin` so that the shelve extension is silently ignored. Differential Revision: https://phab.mercurial-scm.org/D6589

merge with stable

automv: access status fields by name, not index Differential Revision: https://phab.mercurial-scm.org/D6586

automv: use public API for getting copies Differential Revision: https://phab.mercurial-scm.org/D6585

commit: add --force-close-branch flag to close a non-head changeset While closing branch from a changeset which is not a branch head current implementation abort this action in every case but, there can be the situations where the changeset is not a local head but could be a remote head. This patch adds the functionality to bypass the "abort: can only close branch heads" by introducing --force-close-branch flag. Test case changes demonstrate the new functionality added. Differential Revision: https://phab.mercurial-scm.org/D6490

shelve: move shelve extension to core Until now, `shelve` was bootstrapped as an extension. This patch adds `shelve` on core. Differential Revision: https://phab.mercurial-scm.org/D6553

shelve: remove rebase.clearstatus() This is a follow-up patch to c829749e7639. After this, shelve will be no longer dependent on rebase. This removes rebase.clearstatus() from shelve. Differential Revision: https://phab.mercurial-scm.org/D6584

shelve: removed redundant merge detection method Differential Revision: https://phab.mercurial-scm.org/D6547

rust-dirstate: call new "dirs" rust implementation from Python This is a simple module attribute replacement, will take precedence over the Python and C implementations. Differential Revision: https://phab.mercurial-scm.org/D6395

rust-dirstate: add "dirs" rust-cpython binding There is an obvious performance and memory issue with those bindings on larger repos as it copies and allocates everything at once, round-trip. Like in the previous patch series, this is only temporary and will only get better once we don't have large data structures going to and from Python. Differential Revision: https://phab.mercurial-scm.org/D6394

rust-dirstate: add "dirs" Rust implementation Following the work done in d1786c1d34fa and working towards the goal of a complete Rust implementation of the dirstate, this rewrites the `dirs` class. There is already a C implementation, which relies heavily on CPython hacks and protocol violations for performance, so I don't expect this to perform as well for now, as this is very straight-forward code. The immediate benefits are new high-level documentation and some unit tests. Differential Revision: https://phab.mercurial-scm.org/D6393

relnotes: added description about statemod._statecheck Differential Revision: https://phab.mercurial-scm.org/D6557

statecheck: shifted defaults to addunfinished() This shifts the definitions and defaults of `_statecheck()` class to `addunfinished()` registration method. Differential Revision: https://phab.mercurial-scm.org/D6583

statecheck: added support for cmdutil.afterresolvedstates This removes `afterresolvedstates` from `cmdutil` and adds support for it in `_statecheck` class. A new flag `continueflag` is added to the class to check whether an operation supports `--continue` option or not. Tests remain unchanged. Differential Revision: https://phab.mercurial-scm.org/D6551

statecheck: added support for STATES This removes `STATES` from `state.py` and adds support to `statecheck` class to handle its features. `getrepostate()` function is modified accordingly. This adds a method 'cmdutil.addunfinished()' for appending to the unfinishedstate list so as to keep 'merge' and 'bisect' at the last. This also makes two separate message formats for `checkunfinished()` and `getrepostate()` as there were previously present. Results of test changed are shown. Differential Revision: https://phab.mercurial-scm.org/D6503

state: moved cmdutil.STATES and utilities to state.py This commit moves `cmdutil.STATES` and adjoining functions to `state.py`. The existing users are updated accordingly. Tests remain unchanged. Differential Revision: https://phab.mercurial-scm.org/D6502

state: created new class statecheck to handle unfinishedstates For the purpose of handling states for various multistep operations like `hg graft`, `hg histedit`, `hg bisect` et al a new class called statecheck is created .This will help in having a unified approach towards these commands and handle them with ease. The class takes in 4 basic arguments which include the name of the command, the name of the state file associated with it , clearable flag , allowcommit flag. This also also adds the support of`checkunfinished()` and `clearunfinished()` to the class. Tests remain unchanged. Differential Revision: https://phab.mercurial-scm.org/D6501

states: moved cmdutil.unfinishedstates to state.py This moves `cmdutil.unfinishedstates`, `checkunfinished()`,`clearunfinished()` to `state.py`. the already existing users of this module are updated accordingly. Test results remain unchanged. Differential Revision: https://phab.mercurial-scm.org/D6484

rebase: fix in-memory rebasing of copy of empty file Classic Python mistake of unintentionally treating None and empty string the same. Differential Revision: https://phab.mercurial-scm.org/D6570

tests: demonstrate broken in-memory rebase of copy to empty file Differential Revision: https://phab.mercurial-scm.org/D6569

zsh: enable completion support for chg as well When verifying this change, you may need to clear/rebuild the completion cache; I did this by deleting the ~/.zcompdump file and then starting a new shell. Differential Revision: https://phab.mercurial-scm.org/D6574

py3: make catapult usable from the test runner in py3 Differential Revision: https://phab.mercurial-scm.org/D6577

py3: use integer division for the value passed to xrange Differential Revision: https://phab.mercurial-scm.org/D6576

pycompat: make fewer assumptions about sys.executable There are many Python "bundlers" which create an archive to run a Python binary from, and they may not set sys.executable at all - handle that case properly, especially to run tests. Differential Revision: https://phab.mercurial-scm.org/D6575

update: fix spurious unclean status bug shown by previous commit The crux of the problem is: - the dirstate is corrupted (the sizes/dates are assigned to the wrong files) - because when worker.worker is used with a return value (batchget in merge.py here), the return value when worker.worker effectively parallelizes is permuted - this is because worker.worker's partition of input and combination of output values are not inverses of one another: it split [1,2,3,4,5,6] into [[1,3,5],[2,4,6]], but combines that into [1,3,5,2,4,6]. Given that worker.worker doesn't call its function argument on contiguous chunks on the input arguments, sticking with lists means we'd need to know the relation between the inputs of worker.worker function argument (for instance, requiring that every input element is mapped to exactly one output element). It seems better to instead switch return values to dicts, which can combined reliably with a straighforward restriction. Differential Revision: https://phab.mercurial-scm.org/D6581

tests: show bug in update introduced in 87a34c767384 As reported by Martin at https://phab.mercurial-scm.org/D6475. Differential Revision: https://phab.mercurial-scm.org/D6580

copies: document how 'copies' dict instances are reused We avoid copying these instances as much as we can, so it's not obvious what's safe to do with them. This patch tries to explain what is safe and what is not. Differential Revision: https://phab.mercurial-scm.org/D6578

copies: simplify merging of copy dicts on merge commits After we removed some filtering in 35d674a3d5db (copies: don't filter out copy targets created on other side of merge commit, 2019-04-18), we will always include all entries from "copies1", so we can simplify the code based on that. Differential Revision: https://phab.mercurial-scm.org/D6561

copies: remove a redundant matcher filtering in _changesetforwardcopies() We filter before pushing items on the queue, so we don't need to filter after popping. Differential Revision: https://phab.mercurial-scm.org/D6560

copies: delete obsolete comment in _changesetforwardcopies() IIRC, the comment applied to the filtering we did before 35d674a3d5db (copies: don't filter out copy targets created on other side of merge commit, 2019-04-18). Differential Revision: https://phab.mercurial-scm.org/D6559

merge with stable

copies: avoid reusing the same variable for two different copy dicts "childcopies" is initally the copies the current changeset to one of its children and then we reassign it with the copies from the start of the chain to the child. Let's use different names for these two things. Differential Revision: https://phab.mercurial-scm.org/D6564

drawdag: don't crash when writing copy info to changesets When writing copies to the changeset, localrepo.commitctx() will call ctx.p1copies() and ctx.p2copies(). These crashed on simplecommitctx because they ended up trying to access the manifest. drawdag doesn't support copies at all, so we can simply override the methods to return empty dicts. Differential Revision: https://phab.mercurial-scm.org/D6565

merge with stable

log: pass getcopies() function instead of getrenamed() to displayer (API) This reduces the duplication between the two displayer functions (and between them and scmutil.getcopiesfn()). It's still more code than two patches ago, but there's less duplication. Differential Revision: https://phab.mercurial-scm.org/D6546

copies: create helper for getting all copies for changeset There are a few places where we get all the copies for a changeset (at least the {file_copies} template and in two places in `hg log --copies` code). These places currently call scmutil.getrenamedfn() to get a caching "getrenamed" function. They all use it in a similar way. We will be able to reuse more code by having a function for getting all the copies for a changeset. This patch introduces such a function. It uses it in the {file_copies} template to show that it works. It relies on the existing scmutil.getrenamedfn() for caching in the filelog-centric case. Differential Revision: https://phab.mercurial-scm.org/D6545

logcmdutil: also check for copies in null revision and working copy It's safe (and fast) to look for copies in the null revision, and it's incorrect not to look for them in the working copy, so let's look in both places. Differential Revision: https://phab.mercurial-scm.org/D6544

tests: demonstrate missing copy information in working copy with graphlog Differential Revision: https://phab.mercurial-scm.org/D6543

remotefilelog: handle copies in changesets in getrenamedfn() override E.g. the {file_copies} template keyword didn't work with copies in changesets before this patch because remotefilelog overrides the getrenamedfn() and didn't handle the changeset-centric case. Differential Revision: https://phab.mercurial-scm.org/D6542

remotefilelog: check if RFL is enabled in getrenamedfn() override In 8a0e03f7baf4 (remotefilelog: move most setup from onetimesetup() to uisetup(), 2019-05-01), I said: All the wrappers moved in this patch check if remotefilelog is enabled before they change behavior, so it's safe to always wrap. That was clearly a lie, because getrenamedfn() didn't. That made e.g. `hg log -T {file_copies}` unbearably slow. This patch fixes that. Differential Revision: https://phab.mercurial-scm.org/D6541

relnotes: document template support for `hg root` Differential Revision: https://phab.mercurial-scm.org/D6540

remotefilelog: tell runbgcommand to not block on child process startup These two invocations will always find a binary because they're re-running hg. As a result, we can skip waiting for the subprocess to start running and save a little bit of wall-time. Differential Revision: https://phab.mercurial-scm.org/D6539

procutil: allow callers of runbgcommand to assume the process starts Experimentally starting the subprocess can take as much as 40ms, and for some of our use cases that's frivolous: we know the binary will start, and if it doesn't we'd only ever ignore it and continue anyway. This lets those use cases be faster. Differential Revision: https://phab.mercurial-scm.org/D6537

shallowrepo: remove backwards compat code that predates in-tree remotefilelog Differential Revision: https://phab.mercurial-scm.org/D6538

commit: make the error message more specific while aborting branch closing Differential Revision: https://phab.mercurial-scm.org/D6493

commit: add a check if it is trying to close an already closed branch head It would check if the revision we are going to close is already a closed branch head and print the error message accordingly. Differential Revision: https://phab.mercurial-scm.org/D6491

strip: move checksubstate() to mq (its only caller) Differential Revision: https://phab.mercurial-scm.org/D6536

strip: use bailifchanged() instead of reimplementing it This also means that we get the standard error messages (see changed test cases). Differential Revision: https://phab.mercurial-scm.org/D6535

strip: remove unused excsuffix argument from checklocalchanges() It was only used by mq, and mq now has its own copy of the function. Differential Revision: https://phab.mercurial-scm.org/D6534

mq: remove dependency on strip's checklocalchanges() Some of the functionality in strip.checklocalchanges() was only used by mq, so let's move it to mq so we can clean up strip. Differential Revision: https://phab.mercurial-scm.org/D6533

copies: avoid calling matcher if matcher.always() When storing copy information in the changesets (experimental.copies.read-from=changeset-only), this patch speeds up hg debugpathcopies FENNEC_58_0_2_BUILD1 FIREFOX_59_0b8_BUILD2 from 5.9s to 4.7s. At the start of this series (b162229e), that command took 18min. Differential Revision: https://phab.mercurial-scm.org/D6422

copies: avoid unnecessary copying of copy dict When storing copy information in the changesets, this patch speeds up hg debugpathcopies FENNEC_58_0_2_BUILD1 FIREFOX_59_0b8_BUILD2 from 11s to 5.9s. That command takes 6.2s when storing copy information in filelogs. Differential Revision: https://phab.mercurial-scm.org/D6421

copies: don't filter out copy targets created on other side of merge commit If file X is copied to Y on one side of merge and the other side creates Y (no copy), we would not mark that as copy. In the changeset-centric pathcopies() version, that was done by checking if the copy target existed on the other branch. Even though merge commits are pretty uncommon, it still turned out to be too expensive to load the manifest of the parents of merge commits. In a repo of mozilla-unified converted to storing copies in changesets, about 2m30s of `hg debugpathcopies FIREFOX_BETA_59_END FIREFOX_BETA_60_BASE` is spent on this check of merge commits. I tried to think of a way of storing more information in the changesets in order to cheaply detect these cases, but I couldn't think of a solution. So this patch simply removes those checks. For reference, these extra copies are reported from the aforementioned command after this patch: browser/base/content/sanitize.js -> browser/modules/Sanitizer.jsm testing/mozbase/mozprocess/tests/process_normal_finish_python.ini -> testing/mozbase/mozprocess/tests/process_normal_finish.ini testing/mozbase/mozprocess/tests/process_waittimeout_python.ini -> testing/mozbase/mozprocess/tests/process_waittimeout.ini testing/mozbase/mozprocess/tests/process_waittimeout_10s_python.ini -> testing/mozbase/mozprocess/tests/process_waittimeout_10s.ini Since these copies were created on one side of some merge, it still seems reasonable to include them, so I'm not even sure it's worse than filelog pathcopies(), just different. Differential Revision: https://phab.mercurial-scm.org/D6420

copies: do full filtering at end of _changesetforwardcopies() As mentioned earlier, pathcopies() is very slow when copies are stored in the changeset. Most of the cost comes from calling _chain() for every changeset, which is slow because it needs to read manifests. It needs to read manifests to be able to filter out copies that are were created in one commit and then deleted. (It also filters out copies that were created from a file that didn't exist in the starting revision, but that's a fixed revision across calls to _chain(), so it's much cheaper.) This patch changes from _chainandfilter() to just _chain() in the main loop in _changesetforwardcopies(). It instead removes copies that have subsequently been removed by using ctx.filesremoved(). We thus rely on that to be fast. It timed this command in mozilla-unified: hg debugpathcopies FIREFOX_59_0b3_BUILD2 FIREFOX_BETA_59_END It took 18s before and 1.1s after. It's still faster when copy information is stored in filelogs: 0.70s. It also still gets slow when there are merge commits involved, because we read manifests there too. We'll deal with that later. Differential Revision: https://phab.mercurial-scm.org/D6419

rust-filepatterns: add comment about Windows path handling As I replied to the Phabricator message, this is wrong. And I even suspect it wouldn't compile because of multiple type mismatches. I think, in Rust where type system is rock solid, we can live with UTF-8 strings except for the bottom storage layer and the top UI/command layer. We'll still have to get around undecodable characters not to be lost, but I think it's okay to drop such filenames from match result if they don't match in UTF-8 world, not in Latin-1 world.

rust-filepatterns: silence warning of non_upper_case_globals

rust: update Cargo.lock to include @generated comment cargo 1.34.0 of Debian sid inserts this comment, and I'm tired of reverting the change every time I do make local. https://github.com/rust-lang/cargo/commit/bd0e4a08471b8bc7957829b4fd294b8985d4fa2d

merge with stable

lfs: correct an error in the TODO file

cat: don't prefetch files unless the output requires it It's a waste to cache lfs blobs when cat'ing the raw data at best, but a hassle debugging when the blob is missing. I'm not sure if there are other commands that have '{data}' for output, and if there's a general way to prefetch on that keyword. It's interesting that the verbose output seems to leak into the JSON output, but that seems like an existing bug.

tracing: add support for emitting counters Differential Revision: https://phab.mercurial-scm.org/D6526

tracing: extract tracing-active logic to separate function I'm about to add support for counters, and want to avoid duplicating this logic. Differential Revision: https://phab.mercurial-scm.org/D6525

catapipe: add support for COUNTER events Differential Revision: https://phab.mercurial-scm.org/D6524

demandimport: add tracing coverage for Python 3 This makes things feel a little less mysterious when modules are being imported. Differential Revision: https://phab.mercurial-scm.org/D6523

export: don't prefetch *all* files in manifest `hg export` only shows changed files, not all files, but we still prefetched all files in cmdutil.export(). The same is true for the other commands calling cmdutil.exportfile(). That meant that `hg export` with remotefilelog (or lfs, I assume) could take much longer than expected because it would download all the files in the repo. Differential Revision: https://phab.mercurial-scm.org/D6532

remotefilelog: remove obsolete filtering of treemanifest directories I think this has been obsolete since 2cf18f46a1ce (narrow: only walk files within narrowspec also for committed revisions, 2018-09-28). Differential Revision: https://phab.mercurial-scm.org/D6531

py3: add test-dirstate-race2.t to list of passing tests This test was added new recently. The py3 buildbot found that it passes, so let's add it to the list of passing tests. Differential Revision: https://phab.mercurial-scm.org/D6530

strip: during merge allow strip only when -f is used This ensures to abort strip to `hg strip` when we have a merge in progress and allow it only when a `--force` flag is used. Differential Revision: https://phab.mercurial-scm.org/D6529

deltas: set estimated compression upper bound to "3x" instead of "10x" In pratice, we very rarely observer compression better than "3x" on manifest deltas. Having a more aggressive estimate significantly helps our pathological use case on a private repository. Here are a comparison of timings using different upper bound. Estimated compression | ø | ×10 | ×5 | ×3 | timing | 14.11 | 2.61 | 1.96 | 1.53 | We also tested the impact of this series on an array of public repositories. This shown no impact in either size nor timing. Full data set below for those interested. Size ---- Regarding size, not significant impact have been noticed on neither public nor private repositories. Here are the number we gathered on public repositories: zlib/upperbound | no | 10x | 5x | 3x mercurial | 5 875 730 | 5 875 730 | 5 875 730 | 5 875 730 pypy | 27 782 913 | 27 782 913 | 27 782 913 | 27 782 913 netbeans | 159 161 207 | 159 161 207 | 159 161 207 | 159 959 879 (+0.5%) mozilla-central | 323 841 642 | 323 841 642 | 323 841 642 | 319 867 519 (-2.5%) mozilla-try | 746 649 123 | 746 649 123 | 746 649 123 | 741 155 568 (-0.7%) private-repo | 1 485 287 294 | 1 485 287 294 | 1 485 287 294 | 1 409 248 382 (-5.1%) zstd/upperbound | no | 10x | 5x | 3x mercurial | 5 895 206 | 5 895 206 | 5 895 206 | 5 895 206 pypy | 28 689 230 | 28 689 230 | 28 689 230 | 28 689 230 netbeans | 157 636 387 | 157 636 387 | 157 636 387 | 159 692 678 (+1.3%) mozilla-central | 317 650 281 | 317 650 281 | 317 650 281 | 319 613 603 (+0.6%) mozilla-try | 737 555 275 | 737 555 275 | 737 555 275 | 738 079 473 (+0.1%) private-repo | 1 352 362 982 | 1 352 362 982 | 1 346 961 880 | 1 361 327 384 (+0.7%) Speed ------ Timing gathered using `hg perfrevlogwrite -m`. Value are in seconds. mercurial zlib | no | 10x | 5x | 3x | total | 65.551783 | 65.388887 | 65.260658 | 65.321199 | max | 0.034544 | 0.034571 | 0.034659 | 0.034521 | 99.99% | 0.034544 | 0.034571 | 0.034659 | 0.034521 | zstd | no | 10x | 5x | 3x | total | 49.118449 | 49.054062 | 48.753588 | 48.740230 | max | 0.009338 | 0.009239 | 0.009202 | 0.009178 | 99.99% | 0.007618 | 0.007639 | 0.007626 | 0.007621 | pypy zlib | no | 10x | 5x | 3x | total | 560.865984 | 558.983817 | 559.083815 | 559.349152 | max | 0.219614 | 0.215922 | 0.218112 | 0.218107 | 99.99% | 0.219614 | 0.215922 | 0.218112 | 0.218107 | zstd | no | 10x | 5x | 3x | total | 349.393280 | 347.395819 | 347.185407 | 345.643985 | max | 0.084143 | 0.083536 | 0.081834 | 0.082178 | 99.99% | 0.039445 | 0.039639 | 0.039612 | 0.039175 | netbeans zlib | no | 10x | 5x | 3x | total | 33103.327727 | 33314.932260 | 33211.745233 | 33345.891778 | max | 2.666852 | 2.672059 | 2.662453 | 2.662936 | 99.99% | 2.058772 | 2.070429 | 2.069569 | 2.064653 | zstd | no | 10x | 5x | 3x | total | 20112.102708 | 20095.879719 | 20083.390300 | 20123.221859 | max | 2.063482 | 2.062851 | 2.065229 | 2.060147 | 99.99% | 1.146647 | 1.143794 | 1.142933 | 1.146529 | mozilla zlib | no | 10x | 5x | 3x | total | 41374.102138 | 41418.816773 | 41381.956370 | 41334.280732 | max | 3.383474 | 3.387400 | 3.405711 | 3.387316 | 99.99% | 1.006755 | 1.005954 | 1.007700 | 1.007373 | zstd | no | 10x | 5x | 3x | total | 24689.691520 | 24643.939662 | 24664.630027 | 24664.512714 | max | 1.460822 | 1.449640 | 1.439747 | 1.465304 | 99.99% | 0.527111 | 0.527377 | 0.527807 | 0.527226 |

deltas: skip if projected compressed size is bigger than previous snapshot If we have a delta, we check constraints against a lower bound estimate of the resulting compressed delta. We then checks this projected size against the `size(snapshotⁿ) > size(snapshotⁿ⁺¹)` constraint. This allows to exclude potential base candidates before doing any expensive computation. This only apply to the intermediate-snapshot case since this constraint only apply to them. For some pathological cases of a private repository this step provide a further performance boost (timing from `hg perfrevlogwrite`): before: 3.010646 seconds after: 2.609307 seconds

deltas: skip if projected compressed size does not match text size constraint If we have a delta, we check constraints against a lower bound estimate of the resulting compressed delta. We then checks this projected size against the ½ⁿ size constraints. This allows to exclude potential base candidates before doing any expensive computation. This only apply to the intermediate-snapshot case since this constraint only apply to them. For some pathological cases of a private repository this step provide a further performance boost (timing from `hg perfrevlogwrite`): before: 3.145906 seconds after: 3.010646 seconds

deltas: accept and skip None return for delta info They are some extra computation that will shortcut the delta compression if the delta seems hopeless, returning None.

delta: move some delta chain related computation earlier in deltainfo They are some more optimization change that will make use of this in the function. So we retrieve the data earlier.

deltas: skip if projected delta size is bigger than previous snapshot Before computing any delta, we get a basic estimation of the delta size we can expect and the resulted compressed value. We then checks this projected size against the `size(snapshotⁿ) > size(snapshotⁿ⁺¹)` constraint. This allows to exclude potential base candidates before doing any expensive computation. This only apply to the intermediate-snapshot case since this constraint only apply to them. For some pathological cases of a private repository this step provide a significant performance boost (timing from `hg perfrevlogwrite`): before: 14.115908 seconds after: 3.145906 seconds

deltas: skip if projected delta size does not match text size constraint Before computing any delta, we get a basic estimation of the delta size we can expect and the resulted compressed value. We then checks this projected size against the ½ⁿ size constraints. This allows to exclude potential base candidates before doing any expensive computation. This only apply to the intermediate-snapshot case since this constraint only apply to them. In practice we only perform this new checks for the manifestlog. Manifest log combine two property: it is likely to have delta chain issue and its diffing/compression is fairly predictable. The initial author of this changeset is Valentin Gatien-Baron providing the initial idea and initial testing, Pierre-Yves David later consolidated the code in the right location and run more extensive testing.

revlog: add the option to track the expected compression upper bound There are various optimization we can do if we can estimate the size of delta before actually spending CPU compressing them. So we add a attributed dedicated to tracking that. We only use it on Manifest because (1) it structure is quite stable across all Mercurial repository so its compression ratio is fairly universal. This is the revlog with most extreme delta (cf the sparse-revlog optimization). This will be put to use in later changesets. Right now the compression upper bound is set to 10. This is a fairly conservative value (observed value is more around 3), but I prefer to be safe while introducing the optimization principles. We can tune the optimization threshold later.

perf: clarify some of the custom behavior of `perfrevlogwrite` This reduce the chance of developers being surprised by special cases.

perf: fix perfrevlogwrite --count documentation The help text was copy pasted from the previous option.

rust: switched to 'cargo rustc' in setup.py This is more flexible in the passing of additional flags, also what setuptools_rust does, giving less uncertainty about non-Linux platforms.

rust-cpython: fix build for MacOSX MacOSX needs special link flags. Quoting the README of rust-cpython: create a `.cargo/config` with the following content: ``` [target.x86_64-apple-darwin] rustflags = [ "-C", "link-arg=-undefined", "-C", "link-arg=dynamic_lookup", ] ``` This is tested with Python 2.7 (Anaconda install) and Python 3 (Homebrew install)

rust-cpython: management of shared libray suffix Before this changeset, the shared library objects suffixes were both (rustc output and Python input) hardcoded to '.so', which is wrong for Python3 and non Linux targets.

merge: fix race that could cause wrong size in dirstate The problem is that hg merge/update/etc work the following way: 1. figure out what files to update 2. apply the update to disk 3. apply the update to in-memory dirstate 4. write dirstate where step3 looks at the filesystem and assumes it sees the result of step2. If a file is changed between step2 and step3, step3 will record incorrect information in the dirstate. I avoid this by passing the size step3 needs directly from step2, for the common path (not implemented for change/delete conflicts for instance). I didn't fix the same race for the exec bit for now, because it's less likely to be problematic and I had trouble due to the fact that the dirstate stores the permissions differently from the manifest (st_mode vs '' 'l' 'x'), in combination with tests that pretend that symlinks are not supported. However, I moved the lstat from step3 to step2, which should tighten the race window markedly, both for the exec bit and for the mtime. Differential Revision: https://phab.mercurial-scm.org/D6475

worker: support parallelization of functions with return values Currently worker supports running functions that return a progress iterator. Generalize it to handle function that return a progress iterator then a return value. It's unused in this commit, but will be used in the next one. Differential Revision: https://phab.mercurial-scm.org/D6515

tests: show how the dirstate can end up containing wrong information which can result in bad status output. Concretely, this seems to be easily triggered by having a build system watching the filesystem for changes, and rebuilding files that are both tracked and generated while an update is happening. Differential Revision: https://phab.mercurial-scm.org/D6474

rust: new rust options in setup.py The --rust global option turns on usage (and by default compilation) of the rust-cpython based mercurial.rustext. Similarly to what's previously done for zstd, there is a --no-rust option for the build_ext subcommand in order not to build mercurial.rustext, allowing for an OS distribution to prebuild it. The HGWITHRUSTEXT environment variable is still honored, and has the same effect as before, but now it works mostly by making the --rust global option defaulting to True, with some special cases for the direct-ffi case (see more about that below) Coincidentally, the --rust flag can also be passed from the make commands, like actually all global options, in the PURE variable make local PURE=--rust This feels inappropriate, though, and we should follow up with a proper make variable for that case. Although the direct-ffi bindings aren't directly useful any more, we keep them at this stage because - they provide a short prototyping path for experiments in which a C extension module has to call into a Rust extension. The proper way of doing that would be to use capsules, and it's best to wait for our pull request onto rust-cpython for that: https://github.com/dgrunwald/rust-cpython/pull/169 - Build support for capsules defined in Rust will probably need to reuse some of what's currently in use for direct-ffi.

rust: using policy.importrust from Python callers This commit converts all current Python callers of mercurial.rustext to the new policy.importrust system. After this point, going through policy.importrust or policy.importmod (in some more distant future) is mandatory for callers of Rust code outside of Python tests. We felt it to be appropriate to keep Rust-specific tests run inconditionally if the Rust extensions are present.

rust: module policy with importrust We introduce two rust+c module policies and a new `policy.importrust()` that makes use of them. This simple approach provides runtime switching of implementations, which is crucial for the performance measurements such as those Octobus does with ASV. It can also be useful for bug analysis. It also has the advantage of making conditionals in Rust callers more uniform, in particular abstracting over specifics like `demandimport` At this point, the build stays unchanged, with the rust-cpython based `rustext` module being built if HGWITHRUSTEXT=cpython. More transparency for the callers, i.e., just using `policy.importmod` would be a much longer term and riskier effort for the following reasons: 1. It would require to define common module boundaries for the three or four cases (pure, c, rust+ext, cffi) and that is premature with the Rust extension currently under heavy development in areas that are outside the scope of the C extensions. 2. It would imply internal API changes that are not currently wished, as the case of ancestors demonstrates. 3. The lack of data or property-like attributes (tp_member and tp_getset) in current `rust-cpython` makes it impossible to achieve direct transparent replacement of pure Python classes by Rust extension code, meaning that the caller sometimes has to be able to make adjustments or provide additional wrapping.

help: add help entry for internals.mergestate This patch adds an entry for `internals.mergestate` as suggested by @marmoute. Most of the help text is taken from `merge.mergestate`. Differential Revision: https://phab.mercurial-scm.org/D6448 Differential Revision: https://phab.mercurial-scm.org/D6528

phabricator: use parents.set to always set dependencies Now that Mercurial's Phabricator instance has been updated to a version that supports the parents.set transaction on revision.edit we can use that to set dependency relationships in patch stacks instead of abusing the summary. This has the advantage that we can use it on every `phabsend` so commit reordering is picked up without spamming changes like abusing the summary would, and using parents.set will clear previous parents unlike parents.add. Differential Revision: https://phab.mercurial-scm.org/D6514

help: remove repeated word in 'hg help rebase' Specifically, the second 'with' in 'with which to merge with'. Differential Revision: https://phab.mercurial-scm.org/D6483

rebase: tweak description of inmemory working even w/ dirty working dir One of our users was confused because they read this, and then attempted to run `hg rebase` with a dirty working directory, and it still complained. The reason was that they were attempting to rebase the commit they currently had checked out, which (at least with evolve workflows enabled) involves updating the working directory to be based on the newly rebased commit. Differential Revision: https://phab.mercurial-scm.org/D6507

revlog: speed up isancestor Currently, it is implemented on top of commonancestorsheads. Implement it on top of reachableroots instead, as reachableroots could stop walking the graph much sooner than commonancestorsheads. Measuring repo.changelog.isancestorrev on two revisions in a private repository: before: ! wall 0.005175 comb 0.010000 user 0.010000 sys 0.000000 (best of 550) after : ! wall 0.000072 comb 0.000000 user 0.000000 sys 0.000000 (best of 36199) When hg does this kind of operations 1500 times in pull -> bookmarks.comparebookmarks -> bookmarks.validdest, that's 11s that drop from the --profile output. Differential Revision: https://phab.mercurial-scm.org/D6506

dagop: fix documentation of reachableroots The previous revset couldn't be correct as it is symmetric in <roots> and <heads>, but reachableroots has no such symmetry. It makes a difference with for instance reachableroots(2, 3) where 2 and 3 are both children of 1. Differential Revision: https://phab.mercurial-scm.org/D6505

phabricator: add --blocker argument to phabsend to specify blocking reviewers The way to signal to Conduit that a reviewer is considered blocking is just to wrap their PHID in "blocking()" when including it in the list of PHIDs passed to `reviewers.add`. arc doesn't have a --blocker, instead one is supposed to append a '!' to the end of reviewer names (I think reviewers are usually added in an editor rather than the command line, where '!'s can be more hazardous). moz-phab (Mozilla's arcanist wrapper) does have a --blocker argument, and being explicit like this is also more discoverable. Even `arc diff`'s help doesn't seem to mention the reviewer! syntax. Differential Revision: https://phab.mercurial-scm.org/D6512

phabricator: auto-sanitise API tokens and HTTP cookies from VCR recordings Currently when making VCR recordings one needs to manually sanitise sensitive credentials before committing and submitting them as part of tests. It is easy to imagine this being accidentally missed one time by a fallible human and said credentials being leaked. It is also possible that it wouldn't be noticed to alert the user to the leak since the recording files are so large and practically unreviewable. Thus do so automatically, so the only place that needs checking is in the test-phabricator.t file. Differential Revision: https://phab.mercurial-scm.org/D6513

py3: use .startswith() instead of bytes[0] Doing bytes[0] will return the ascii value of that position which breaks comparison with a bytechar. This makes test-absorb.t work again on py3. Differential Revision: https://phab.mercurial-scm.org/D6508

revset: fix merge() to fall back to changectx API if wdir specified I have a code which basically runs "0:wdir() & <user-revset>", and it crashed if merge() were passed in.

revset: use nullrev constant in merge()

mixedrepostorecache: fix a silly redundant updating of set Differential Revision: https://phab.mercurial-scm.org/D6470

rust-regex: fix shortcut for exact matches The current shortcut for rootglobs that can be simplified to exact matches does not work, it instead treats the pattern as a regex, which is not the same thing. This changes fixes the behavior and introduces a test for this behavior. Differential Revision: https://phab.mercurial-scm.org/D6489

rust-filepatterns: use bytes instead of String In my initial patch, I introduced an unnecessary hard constraint on UTF-8 filenames and patterns which I forgot to remove. Although the performance penalty for using String might be negligible, we don't want to break compatibility with non-UTF-8 encodings for no reason. Moreover, this change allows for a cleaner Rust core API. This patch introduces a new utils module that is used with this fix. Finally, PatternError was not put inside the Python module generated by Rust, which would have raised a NameError. Differential Revision: https://phab.mercurial-scm.org/D6485

doc: fix description of "predecessors" to match reality Differential Revision: https://phab.mercurial-scm.org/D6467

phabricator: make `hg debugcallconduit` work outside a hg repo I am trying to write some automations around phabricator and having debugcallconduit work outside a hg repo will be nice! Marking command as optionalrepo instead of norepo because we might to load repo/.hg/hgrc. Differential Revision: https://phab.mercurial-scm.org/D6499

phabricator: pass ui instead of repo to callconduit This will help us make `hg debugcallconduit` work outside a hg repo as next patch will mark that command as no repo. Differential Revision: https://phab.mercurial-scm.org/D6498

phabricator: pass ui into readurltoken instead of passing repo The goal of this series is to make `hg debugcallconduit` work outside of a hg repo. This patch, removes requirement of repo object from readurltoken as we only need ui there. It also updates the callers to pass in ui instead of repo. Differential Revision: https://phab.mercurial-scm.org/D6497

py3: add test-contrib-emacs.t to passing tests list I installed emacs on the server running buildbot and the test started passing on Python 3. Lets add it to the list of passing test. Differential Revision: https://phab.mercurial-scm.org/D6500

phabricator: add commenting to phabsend for new/updated Diffs Especially useful when sending updates to existing Revisions so one can specify the sort of changes e.g. "Address review comments" or "Rebase to tip" If the diff content hasn't changed then it only needs a metadata update which doesn't show in the Phabricator updates UI, so don't add a comment that will. Differential Revision: https://phab.mercurial-scm.org/D6496

py3: fix test-bookmarks-corner-case.t For some reasons, the output of print was not going through. Replaced that ui.status(). Differential Revision: https://phab.mercurial-scm.org/D6481

py3: fix test-fix-metadata.t # skip-blame as just b'' prefixes Differential Revision: https://phab.mercurial-scm.org/D6480

py3: add b'' prefix at one place in run-tests.py #skip-blame because just b'' prefix Differential Revision: https://phab.mercurial-scm.org/D6482

copies: separate added/removed files by newline instead of null This makes it more consistent with how we encode copies (newline-separated lists of null-separated pairs). This perhaps makes {extras} a little less readable (?) despite avoiding the escaping. I don't know how I feel about this patch. I'm okay with it being queued or dropped. Differential Revision: https://phab.mercurial-scm.org/D6486

copies: also encode p[12]copies destination as index into "files" list This is mostly for consistency with the filesaddes/filesremoved fields, but it should also save a bit of space. Differential Revision: https://phab.mercurial-scm.org/D6431

discovery: be more conservative when adjusting the sample size Since 5b34972a0094, the discovery will increase the sample size when it detect a "complex" undecided set. However this detection focussed on the number of roots only, this could regress discovery performance when the undecided set has many roots that eventually get merged into a few heads. To prevent such misbehavior, we adjust the logic to take in account both heads and roots. The sample size will be increased only if both are especially large. Performance testing on the same case as 5b34972a0094, does not show a significant difference.

rust-dirstate: create dirstate submodule This change is here to facilitate a future patch that is written in a different file. I expect this module to grow a few different files. Differential Revision: https://phab.mercurial-scm.org/D6389

profiling: show actual time spent in hotpath display To get, for instance: ... \ 6.6% 4.08s lock.py: __exit__ line 1566: ... | 6.5% 4.01s exchange.py: close line 1191: ... | 6.5% 4.01s transaction.py: _active line 1443: ... | 6.5% 4.01s transaction.py: close line 47: ... | 6.2% 3.84s scmutil.py: wrapped line 529: ... | 6.2% 3.81s localrepo.py: wrapper line 2114: ... | 6.2% 3.81s localrepo.py: updatecaches line 177: ... ... instead of: ... \ 6.6% lock.py: __exit__ line 1566: ... | 6.5% exchange.py: close line 1191: ... | 6.5% transaction.py: _active line 1443: ... | 6.5% transaction.py: close line 47: ... | 6.2% scmutil.py: wrapped line 529: ... | 6.2% localrepo.py: wrapper line 2114: ... | 6.2% localrepo.py: updatecaches line 177: ... ... I find that if it's not displayed, I frequently end up estimating the numbers by hand. Differential Revision: https://phab.mercurial-scm.org/D6477

merge with stable

merge with stable

root: add template variables pointing to repository directories These paths are useful for GUI applications to detect changes. A GUI process typically monitors .hg and .hg/store directories so that it will be notified on lock/wlock deletion. Alternatively, maybe we can add debugpaths command if we don't want to extend the root command. I'm not sure which will be nicer.

root: add support for -Tformatter option It's useless right now, but it should just work and I want to add a few more fields.

narrow: pass the bundle to bundle2.widen_bundle() instead of generating there This will make the code in narrowwirepeer.py more better for further refactoring. Differential Revision: https://phab.mercurial-scm.org/D6438

narrow: refactor code around widening complicated by previous patch Previous patch while adding support for using narrow_widen wireproto command, complicated the code a bit. This patch refactors that. Differential Revision: https://phab.mercurial-scm.org/D6437

narrow: use narrow_widen wireproto command to widen in case of ellipses Few releases ago, we introduce narrow_widen wireproto command to be used to widen narrow repositories. Before this patch, that was used in non-ellipses cases only. In ellipses cases, we still do exchange.pull() which can pull more data than required. After this patch, the client will first check whether server supports doing ellipses widening using wireproto command or not by checking server's wireproto capability. If the server is upto date and support latest ellipses capability, we call the wireproto command. Otherwise we fallback to exchange.pull() like before. The compat code make sure that things works even if one of the client or server is old. The initial version of this patch does not had this compat code. It's added to help Google release things smoothly internally. I plan to drop the compat code before the upcoming major release. Due to change to wireproto command, the code looks a bit dirty, next patches will clean that up. Differential Revision: https://phab.mercurial-scm.org/D6436

merge: correct argument name in docstring Differential Revision: https://phab.mercurial-scm.org/D6476

narrowspec: replace one recursion-avoidance hack with another When updating the working copy narrowspec, we call context.walk() in order to find which files to update the working copy with. context.walk() calls repo.narrowmatch(). In order to avoid infinite recursion in this case, we have a hack that assigns the new values for repo.narrowpats and repo._narrowmatch. However, doing that of course breaks future invalidation of those properties (they're @storecache'd). Let's instead avoid the infinite recursion by setting a flag on the repo instance when we're updating the working copy. Differential Revision: https://phab.mercurial-scm.org/D6468

merge: simplify initialization of "pas" Differential Revision: https://phab.mercurial-scm.org/D6472

merge: reorder some initialization to make more sense This puts the closely related definitions of "pl", "p1", "p2", "pas" close together, and moves the definition of "overwrite" away and closer to where it's first used. Differential Revision: https://phab.mercurial-scm.org/D6471

rust-dirstate: architecture independence fix Apparently, c_char is u8 on ppc64le and i8 on amd64 Differential Revision: https://phab.mercurial-scm.org/D6473

context: get filesadded() and filesremoved() from changeset if configured This adds the read side for getting the sets of added and removed files from the changeset extras. I timed this command on the hg repo: hg log -T '{rev}\n {files}\n %:{file_mods}\n +{file_adds}\n -{file_dels}\n' It took 1m21s before and 6.4s after. I also used that command to check that the result didn't change compared to calculating the values from the manifests on the fly (it didn't change). In the mozilla-unified repo, the same command run on FIREFOX_BETA_58_END::FIREFOX_BETA_59_END went from 29s to 0.67s. Differential Revision: https://phab.mercurial-scm.org/D6417

changelog: optionally store added and removed files in changeset extras As mentioned in an earlier patch, copies._chain() is used a lot in the changeset-centric version of pathcopies(). It is expensive because it needs to look at the manifest in order to filter out copies whose target file has since been removed. I want to store the sets of added and removed files in the changeset in order to speed that up. This patch does the writing part of that. It could easily be a separate config, but it's currently tied to experimental.copies.write-to since that's the only real use case (it will also make the {file_*} template keywords faster, but I doubt that anyone cares enough about those to write extra metadata for them). The new information is stored in the changeset extras. Since they're always subsets of the changeset's "files" list, they're stored as indexes into that list. I've stored the indexes as stringified ints separated by NUL bytes. The size of 00changelog.d for the hg repo increased in size by 0.28% percent (compared to the size with only copy information in the changesets, which in turn is 0.17% larger than without copy information). We could store only the delta between the indexes and we could store them in binary, but the chosen format is more readable. We could also have implemented this as a cache outside the changelog. One advantage of doing it that way is that we would get the speedups from the {file_*} template keywords also on old repos. Another advantage is that it we can rewrite the cache if we find a bug in how we calculate the set of files. A disadvantage is that it would be more complex. Another is that it would surely use more space. We already write the copy information to the changeset extras, so it seems like a small step to also write these file sets. Differential Revision: https://phab.mercurial-scm.org/D6416

templatekw: make {file_*} compare to both merge parents (issue4292) This redefines the {file_adds}, {file_dels}, {file_mods} template keywords by getting the lists from the recently introduced context methods instead of getting them from status compared to p1. As mentioned before, these are better defined on merge commits. The total number of files from the three lists now always add up to the number of files in {files}. I timed this command: hg log -r 4.0::5.0 -T '{rev}\n {file_mods}\n {file_adds}\n {file_dels}\n' It went from 7.6s to 5.6s with this patch. So it's actually faster than before. Note that the "files:" field in the bazaar test log output was using "{file_mods}" (not "{files}" as one might think based on the label). Differential Revision: https://phab.mercurial-scm.org/D6369

narrowspec: use vfs.tryread() instead of reimplementing Note that parseconfig() works well with empty strings. Differential Revision: https://phab.mercurial-scm.org/D6465

help: remove a superfluous "the" in revlogs text Differential Revision: https://phab.mercurial-scm.org/D6466

setdiscovery: make progress on most connected groups each roundtrip Consider history like this: o | o | | | o | | | o |/ o | o | | | o | | | o |/ o | o | | | o | | | o |/ o ~ Assume the left mainline is available in the remote repo and the other commits are only in the local repo. Also imagine that instead of 3 local branches with 3 commits on each, there are 1000 branches (the number of commits on each doesn't matter much here). In such a scenario, the current setdiscovery code will pick a sample size of 200 among these branches and ask the remote which of them it has. However, the discovery for each such branch is completely independent of the discovery for the others -- knowing whether the remote has a commit in one branch doesn't give us any information about the other branches. The discovery will therefore take at least 5 roundtrips (maybe more depending on which commit in each linear chain was sampled). Since the discovery for each branch is independent, there is no reason to let one branch wait for another, so this patch makes it so we sample at least as many commits as there are branches. It may still happen (it's very likely, even) that we get multiple samples from one branch and none from another, but that will even out over a few rounds and I think this is still a big improvement. Because of http header size limits, we still use the old behavior unless experimental.httppostargs=true. I've timed this by running `hg debugdiscovery mozilla-unified --debug` in the mozilla-try repo. Both repos were local. Before this patch, last part of the output was: 2249 total queries in 5276.4859s elapsed time: 5276.652634 seconds heads summary: total common heads: 13 also local heads: 4 also remote heads: 8 both: 4 local heads: 28317 common: 4 missing: 28313 remote heads: 12 common: 8 unknown: 4 local changesets: 2014901 common: 530373 missing: 1484528 common heads: 1dad417c28ad 4a108e94d3e2 4d7ef530fffb 5350524bb654 777e60ca8853 7d97fafba271 9cd2ab4d0029 a55ce37217da d38398e5144e dcc6d7a0dc00 e09297892ada e24ec6070d7b fd559328eaf3 After this patch, the output was (including all the samples, since there were so few now): taking initial sample query 2; still undecided: 1599476, sample size is: 108195 sampling from both directions query 3; still undecided: 810922, sample size is: 194158 sampling from both directions query 4; still undecided: 325882, sample size is: 137302 sampling from both directions query 5; still undecided: 111459, sample size is: 74586 sampling from both directions query 6; still undecided: 26805, sample size is: 23960 sampling from both directions query 7; still undecided: 2549, sample size is: 2528 sampling from both directions query 8; still undecided: 21, sample size is: 21 8 total queries in 24.5064s elapsed time: 24.670051 seconds heads summary: total common heads: 13 also local heads: 4 also remote heads: 8 both: 4 local heads: 28317 common: 4 missing: 28313 remote heads: 12 common: 8 unknown: 4 local changesets: 2014901 common: 530373 missing: 1484528 common heads: 1dad417c28ad 4a108e94d3e2 4d7ef530fffb 5350524bb654 777e60ca8853 7d97fafba271 9cd2ab4d0029 a55ce37217da d38398e5144e dcc6d7a0dc00 e09297892ada e24ec6070d7b fd559328eaf3 Differential Revision: https://phab.mercurial-scm.org/D2647

help: clarify overlap of revlog header and first revlog entry Differential Revision: https://phab.mercurial-scm.org/D6449

py3: fix test-convert-svn-sink.t In cases where the root commit is empty commit, None will be returned as parents. This was implemented by 2c13e91ede6e. This breaks test on py3 because `b'%s' % None` does not work. It does not matter whether we return `None` or `'None'` as we skipped converting to svn step by doing an early return. So let's return `'None'`. I tried to patch all the users to convert `None` to `'None'`, but there were more users than I expected. I hit 3 of them and decided to fix it this way around. Differential Revision: https://phab.mercurial-scm.org/D6458

commit: respect --no-edit in combination with --amend Differential Revision: https://phab.mercurial-scm.org/D6464

commit: add test showing that commit --amend --no-edit still shows editor Differential Revision: https://phab.mercurial-scm.org/D6463

githelp: translate git stash show and clear actions and --patch flag Differential Revision: https://phab.mercurial-scm.org/D6461

githelp: add --dry-run for mv Differential Revision: https://phab.mercurial-scm.org/D6460

githelp: translate --directory of git apply to --prefix According to the help pages, these flags do the same. Differential Revision: https://phab.mercurial-scm.org/D6459

help: include subtopic in error message if passed Differential Revision: https://phab.mercurial-scm.org/D6442

help: check if a subtopic exists and raise an error if it doesn't (issue6145) Differential Revision: https://phab.mercurial-scm.org/D6441

perf: fix some missing b prefixes # skip-blame just b prefixes Differential Revision: https://phab.mercurial-scm.org/D6457

testparseutil: fix doctest to use str instead of bytes Differential Revision: https://phab.mercurial-scm.org/D6456

testparseutil: stop extracting using std* streams as bytes on py3 This is no longer required due to other cleanups in our linting tools. Differential Revision: https://phab.mercurial-scm.org/D6455

tests: sort some imports that were previously missed I'm a little unclear why the import checker didn't catch this before, but when I fixed it to work in Python 3 this failure started showing up. Sigh. Differential Revision: https://phab.mercurial-scm.org/D6454

contrib: fix import-checker to operate on str instead of bytes I believe this is fallout from other Python 3 cleanups, and our code linting tools are now leaning towards operating on str and not bytes. I don't feel strongly, so I've just restored this tool to working on Python 3. Differential Revision: https://phab.mercurial-scm.org/D6453

verify: use self._err not self.err, it changed in 7eaf4b1ac2a3 Differential Revision: https://phab.mercurial-scm.org/D6451

tests: make run-tests exit non-zero if there are "errors" Previously, if there was an error such as a broken .t file that caused run-tests.py to encounter an exception during parsing, the test would be considered in an "errored" state, which is separate from "failed". The check for whether to exit non-zero or not was based entirely on whether there were any tests in a "failed" state, so if there was only an error, run-tests would exit with 0. Our test infrastructure would then consider the test as passing, causing us to have some tests with false negatives that have gone undetected for a few weeks now. Differential Revision: https://phab.mercurial-scm.org/D6452

perf: add a `perfhelper-mergecopies` command This command gather data that are useful to pick argument for `perfmergecopies`.

perf: add a new `perfmergecopies` command This command benchmark calls to `mercurial.copies.mergecopies`

perf: factor selection of revisions involved in the merge out We will introduce more performance command around merge. As a first step we factor out pieces of `perfmergecalculate` that can be reused.

perf: allow to specify the base of the merge in perfmergecalculate We can now test the rebase case.

perf: add a --from flag to perfmergecalculate Before this change, `perfmergecalculate` was always benchmarking the merge of the working copy with another revision. We can now benchmark the `mergecalculate` call for any arbitrary pair of revision.

merge with stable

py3: fix test-narrow* which started failing because of recent changes #skip-blame because just r'' prefix Differential Revision: https://phab.mercurial-scm.org/D6447

tests: add test for {file_mods}, {file_adds}, {file_dels} on merge commit Differential Revision: https://phab.mercurial-scm.org/D6368

context: add ctx.files{modified,added,removed}() methods Changeset-centric copy tracing is currently very slow because it often reads manifests. One place it needs the manifest is in _chain(), where it removes a copy X->Y if Y has subsequently gotten removed. I want to speed that up by keeping track directly in the changeset of which files are removed in the changeset. These methods will be similar to ctx.p[12]copies() in that way: they will either read from the changeset or calculate the information from the manifests otherwise. Note that these are different from ctx.{modified,added,removed}() on merge commits. Those functions always compare to p1, but the new ones compare to both parents. filesadded() means "file does not exist in either parent but exists now", filesremoved() means "file existed in either parent but does not exist now", and filesmodified() means "file existed in either parent and still exists". The set of files in ctx.files() is the union of the files from the three new functions (and the three new ones are all disjoint sets). Also note that uncommitted merges are weird as usual. The invariant mentioned above still holds, but the functions compare to p1 (and are thus identical to the existing methods). Differential Revision: https://phab.mercurial-scm.org/D6367

copies: split up _chain() in naive chaining and filtering steps The function now has two clearly defined steps. The first step is the actual chaining. This step is very cheap. The second step is filtering out invalid copies. This step is expensive. For changeset-centric copy tracing, I want to do the filtering step only at the end. This patch prepares for that. Differential Revision: https://phab.mercurial-scm.org/D6418

relnotes: document changed behavior of ui.origbackuppath pointing to file Differential Revision: https://phab.mercurial-scm.org/D6446

templatekw: move showfileadds() close to showfile{mods,dels}() Differential Revision: https://phab.mercurial-scm.org/D6370

py3: use range() instead of xrange() The latter does not exist on Python 3. This makes test-contrib-perf.t pass on Python 3 again. Differential Revision: https://phab.mercurial-scm.org/D6443

narrow: move heads close to common as they are closely related Differential Revision: https://phab.mercurial-scm.org/D6445

narrow: pass binary nodeids to generateellipsesbundle2() We generally work with binary nodeids and it's should be expected that new function gets the nodeids in binary form already. Differential Revision: https://phab.mercurial-scm.org/D6444

match: stabilize _rootsdirsandparents doctest Changeset c4b8f8637d7a tried to stabilize some matcher test by using a set. This did not work because the set order is not stable. To fix it, we post process the result to display a sorted version of the set.

narrow: factor out logic to build ellipses related b2parts in separate fn This will help us switch more cleanly to using wireprotocol commands instead of using exchange.pull() which exchanges more things then required. Differential Revision: https://phab.mercurial-scm.org/D6435

narrow: remove unrequired compat code for old versions of hg As the comment says, that if is only required for servers having hg version 3.1 and 3.2. Any client connecting having hg 3.1 or 3.2 locally and trying to use narrow should already be broken taking in account the changes which have been done since narrow moved to core. Differential Revision: https://phab.mercurial-scm.org/D6434

perf: make sure to explicitly disable any profiler after the first iteration The current code work, because of some edge behavior of the `profile` class. We make it explicit that the profiler is not in effect more than once.

test: add missing 'cd ..' to test case Differential Revision: https://phab.mercurial-scm.org/D6439

match: remove an obsolete comment about util.finddirs() Obsolete since 8e55c0c642c (util: make util.dirs() and util.finddirs() include root directory (API), 2017-05-16). Differential Revision: https://phab.mercurial-scm.org/D6433

match: de-flake test-doctest.py by not depending on util.dirs() order util.dirs() yields directories in arbitrary order, which has made test-doctest.py flaky. I think they have been flaky since d8e55c0c642c (util: make util.dirs() and util.finddirs() include root directory (API), 2017-05-16). Before that commit, I think util.dirs() would return at most one entry, so there was only one iteration order. This patch fixes the problem by making _rootsdirsandparents() return a set (whose __str__() is defined to be in sorted order, I believe). The only caller wanted a set anyway. Differential Revision: https://phab.mercurial-scm.org/D6432

perf: add an option to profile the benchmark section Running a perf command with --profile gather data for the whole command execution, including setup and cleanup. This can significantly alter the data. To work around this we introduce a new option, it trigger the profiling of only one iteration of the benchmarked section.

perf: add a `pre-run` option sometimes, the initial run is necessary to warm some cache that are not relevant for the current measurement. We add a new `perf.pre-run` option to specify a number of run of the benchmark logic that will happens before measurement are taken.

narrow: consider empty commits to be "inside the narrow spec" for templates It doesn't seem useful to exclude them, or harmful to include them. Users writing log templates using outsidenarrow as a predicate might consider it unexpected if their locally created empty drafts are treated as if they contained something outside the clone. Differential Revision: https://phab.mercurial-scm.org/D6414

rust-python3: useless python2 specific import This python27_sys import prevents building with python3, it had been previously removed in a5fa9140ce4c, but that has been since pruned Differential Revision: https://phab.mercurial-scm.org/D6415

rust-python3: compatibility fix for incoming PyLong On Python3, PyInt is PyLong and it doesn't have the `value()` method. Re upcasting to PythonObj as done here works, but we might prefer taking a PythonObj from the onset (would require more testing) Differential Revision: https://phab.mercurial-scm.org/D6397

py3: add one new passing test found by buildbot Differential Revision: https://phab.mercurial-scm.org/D6412

discovery: slowly increase sampling size Some pathological discovery runs can requires many roundtrip. When this happens things can get very slow. To make the algorithm more resilience again such pathological case. We slowly increase the sample size with each roundtrip (+5%). This will have a negligible impact on "normal" discovery with few roundtrips, but a large positive impact of case with many roundtrips. Asking more question per roundtrip helps to reduce the undecided set faster. Instead of reducing the undecided set a linear speed (in the worst case), we reduce it as a guaranteed (small) exponential rate. The data below show this slow ramp up in sample size: round trip | 1 | 5 | 10 | 20 | 50 | 100 | 130 | sample size | 200 | 254 | 321 | 517 | 2 199 | 25 123 | 108 549 | covered nodes | 200 | 1 357 | 2 821 | 7 031 | 42 658 | 524 530 | 2 276 755 | To be a bit more concrete, lets take a very pathological case as an example. We are doing discovery from a copy of Mozilla-try to a more recent version of mozilla-unified. Mozilla-unified heads are unknown to the mozilla-try repo and there are over 1 million "missing" changesets. (the discovery is "local" to avoid network interference) Without this change, the discovery: - last 1858 seconds (31 minutes), - does 1700 round trip, - asking about 340 000 nodes. With this change, the discovery: - last 218 seconds (3 minutes, 38 seconds a -88% improvement), - does 94 round trip (-94%), - asking about 344 211 nodes (+1%). Of course, this is an extreme case (and 3 minutes is still slow). However this give a good example of how this sample size increase act as a safety net catching any bad situations. We could image a steeper increase than 5%. For example 10% would give the following number: round trip | 1 | 5 | 10 | 20 | 50 | 75 | 100 | sample size | 200 | 321 | 514 | 1 326 | 23 060 | 249 812 | 2 706 594 | covered nodes | 200 | 1 541 | 3 690 | 12 671 | 251 871 | 2 746 254 | 29 770 966 | In parallel, it is useful to understand these pathological cases and improve them. However the current change provides a general purpose safety net to smooth the impact of pathological cases. To avoid issue with older http server, the increase in sample size only occurs if the protocol has not limit on command argument size.

tests: make the grep pattern in remotefilelog-gcrepack portable (issue6122) test-remotefilelog-gcrepack was using "\" to escape "|" in the grep pattern. The most of implementations ignore "\" when it is followed by "|", so the regex works. However, OpenBSD doesn't ignore "\" and considers "|" part of the text instead of create two branches. Neither of both behaviors violate POSIX. This change removes the unnecessary escape character and changes grep to egrep, so the extended regular expression works on every unix. This is part of the bug 6122. Tested on OpenBSD, GNU, FreeBSD, NetBSD, Solaris 11 and BusyBox. Credits to Todd C. Miller, Paul de Weerd and Ingo Schwarze for helping me with it.

help: document new "bookmarksinstore" requirement in internals.requirements Differential Revision: https://phab.mercurial-scm.org/D6413

absorb: fix interactive mode I didn't know existed While investigating a bug in `hg absorb -e` I unintentionally discovered `hg absorb --interactive` and its brokenness. This adds a test and restores the functionality. Note that this interface is still marked experimental, so we can change this to be more sophisticated in the future. Differential Revision: https://phab.mercurial-scm.org/D6411

tests: work around libressl being different about error strings (issue6122) As far as I can tell, this is the right behavior. Thanks to Alex Gaynor for checking what the string means by looking at libressl sources for me. Differential Revision: https://phab.mercurial-scm.org/D6410

merge with stable

templatekw: change default value of 'requires' to () Since we dropped support for the old-style template keywords, we no longer have to distinguish None (old-style) and an empty requirement (new-style).

commit: move sorting of added and removed files list to lower level localrepo.commitctx() has lists of all changed files, as well as lists of added and removed files. The list of all files is unsorted and changelog.add() will sort it. Let's also sort the lists of added and removed files at a lower level (manifestrevlog.add()) for consistency. It also seems safer to do it there, just before we write them to the store. That way other callers won't be able to create invalid commits (or whatever the consequence is) by passing in unsorted lists. Also, alternative storages may not care that the lists are sorted. I don't think this will be a performance problem (someone should have fixed the sorting in changelog.add() if it were). Differential Revision: https://phab.mercurial-scm.org/D6390

match: drop unnecessary adding of '' to set of dirs This breaks some tests for "rootfilesin:" in a pattern matcher even more, but that just shows how broken that case is. Differential Revision: https://phab.mercurial-scm.org/D6406

narrowcommands: drop unnecessary adding of '' for root directory It's now included by util.dirs(). Differential Revision: https://phab.mercurial-scm.org/D6405

copies: remove hack for adding root dir to util.dirs object Differential Revision: https://phab.mercurial-scm.org/D6404

util: make util.dirs() and util.finddirs() include root directory (API) This changes the behavior of test-origbackup-conflict.t so it no longer errors out when the backup path points to an existing file. Instead, it replaces the file by a directory. That seems reasonable to me. Differential Revision: https://phab.mercurial-scm.org/D6403

dirstate: drop workaround for '.' matching root directory The check was added in 31abcae33b4f (dirstate: do not ignore current directory '.' (issue 1078), 2008-04-05) to fix issue1078. Funnily enough, comment #2 on that issue mentions using '' instead of '.' to represent the root directory, just like my previous patch did. test-hgignore.t fails with this patch without the previous patch. Differential Revision: https://phab.mercurial-scm.org/D6402

match: use '' instead of '.' for root directory (API) I think '' is generally a better value for the root directory than '.' is. For example, os.path.join('', 'foo') => 'foo', while os.path.join('.', 'foo') => './foo'. This patch mostly makes it so we use '' internally in match.py. However, it also affects the API in visitdir(), visitchildrenset() and files(). The two former now also accept '' as input. I've updated the callers of these methods. I've also added a deprecation warning for passing '.' (for external callers). The only caller I could find that was affected by files() returning '' instead of '.' was in dirstate.walk(). I've updated that. The next few patches show some workarounds we can remove by using '' instead of '.'. Differential Revision: https://phab.mercurial-scm.org/D6401

dirstate: move special handling of files==['.'] together I think it makes it a little clearer to have the two conditions for files==['.'] near each other. Differential Revision: https://phab.mercurial-scm.org/D6400

convert: don't include file in "files" list if it's added in p2 If the file is from p2, we should clearly compare the flags to what they were in p2. Also note that manifest.flags('non-existent') unfortunately returns '' instead of erroring out. Differential Revision: https://phab.mercurial-scm.org/D6409

convert: demonstrate broken {files} list in merge commits with file flags When there is a merge in which the flags for a file from p2 is non-empty, `hg convert` will incorrectly include that in the changeset's files list. Differential Revision: https://phab.mercurial-scm.org/D6408

templater: drop support for old style keywords (API) These changes originated from several commits over a period of time, so I'm slightly unsure if this is correct. But the tests pass.

commands: drop support for legacy ^cmd registration (API)

extensions: drop support for extsetup() without `ui` argument (API)

relnotes: mention removed support for mixed log graph lines This adds release notes for 264a2cbb25d0 (graphmod: remove support for graph lines mixing parent/grandparent styles (BC), 2018-10-16). Differential Revision: https://phab.mercurial-scm.org/D6407

tests: fix test-clonebundles on recent openbsd I guess openbsd feels like it needs to stringify this errno in lowercase and omit the "host" part of "hostname. Okay. Reported in a big test diff talking about libressl, see 6122. I'm not flagging this because most of that issue is about a libressl string change, so this doesn't really make a big difference there. Differential Revision: https://phab.mercurial-scm.org/D6399

rust-python3: compatibility fix for integer conversion On python3, `to_py_object()` on the usize gives us a PyLong, whereas it is the generic `PyObject` already on python2, which fits the `py.None()` default value. Upcasting to `PyObject` explicitely in all cases solves the issue. Differential Revision: https://phab.mercurial-scm.org/D6396

rust: sort dependencies entries in Cargo.toml I should probably write a test to enforce this... Differential Revision: https://phab.mercurial-scm.org/D6398

py3: make contrib/testparseutil.py to work on str(unicodes) contrib/check-code work on unicodes and call functions from testparseutil.py which before this patch used to work on bytes. This path removes that inconsistency and make testparseutil.py work on unicodes. This makes test-check-code.t and test-contrib-check-code.t work on Python 3 again. Differential Revision: https://phab.mercurial-scm.org/D6391

rust-filepatterns: call new Rust implementations from Python This change adds the import to the `rust-cpython` bindings and uses them when appropriate. A wrapper function has been defined in the case of `_regex` to keep this patch simple. Differential Revision: https://phab.mercurial-scm.org/D6273

rust-filepatterns: add `rust-cpython` bindings for `filepatterns` This change adds the `rust-cpython` interface for top-level functions and exceptions in the filepatterns module. Contrary to the Python implementation, this tries to have finer-grained exceptions to allow for better readability and flow control down the line. Differential Revision: https://phab.mercurial-scm.org/D6272

rust-filepatterns: add a Rust implementation of pattern-related utils This change introduces Rust implementations of two functions related to pattern handling, all located in `match.py`: - `_regex` - `readpatternfile` These utils are useful in the long-term effort to improve `hg status`'s performance using Rust. Experimental work done by Valentin Gatien-Baron shows very promising improvements, but is too different from the current Mercurial core code structure to be used "as-is". This is the first - albeit very small - step towards the code revamp needed down the line. Two dependencies were added: `regex` and `lazy_static`. Both of them will be useful for a majority of the Rust code that will be written, are well known and maintained either by the Rust core team, or by very frequent contributors. Differential Revision: https://phab.mercurial-scm.org/D6271

exchange: don't take wlock if bookmarks are stored in .hg/store/ If bookmarks are stored in .hg/store/, there is no need for the wlock(). Differential Revision: https://phab.mercurial-scm.org/D6388

bookmarks: keep bookmarks in .hg/store if new config set Bookmarks storage consists of two parts: (1) the set of bookmarks and their positions, and (2) the current bookmark. The former can get updated by exchange, while the latter cannot. However, they are both stored in directly .hg/ and protected by repo.wlock(). As a result, ugly workarounds were needed. This patch introduces a new config option to store the set of bookmarks and their positions in .hg/store/ but still storing the current bookmark directory in .hg/. The config option only takes effect at repo creation time. It results in a new requirement being set. Differential Revision: https://phab.mercurial-scm.org/D6387

log: flag topo-sorted set as such This isn't required right now, but revs.istopo() should return True.

copies: fix duplicatecopies() with overlay context The reasoning for this check is in 78d760aa3607 (duplicatecopies: do not mark items not in the dirstate as copies, 2013-03-28). The check was then moved to workingfilectx in 754b5117622f (context: add workingfilectx.markcopied, 2017-10-15) and no corresponding check was added later when overlayworkingfilectx was added. Rather than adding the check there, this patch adds a more generic check on the callers side and removes the check in workingfilectx.markcopied(). Differential Revision: https://phab.mercurial-scm.org/D6380

tests: demonstrate crash when rebasing across copy with --collapse As reported by timeless. Differential Revision: https://phab.mercurial-scm.org/D6379

exthelper: add some semi-useful trace logs It'd be nice to make the trace functions a little better-named in the output, but I'm not sure how much better we can do without overhead. This at least lets you see if a single reposetup function is eating all the time or if it's spread over all of them. I needed this because Google's uber-extension has a long load time and I wasn't sure where the problem was. Differential Revision: https://phab.mercurial-scm.org/D6381

help: add missing blank line, making "revlog-compression" show up Differential Revision: https://phab.mercurial-scm.org/D6386

tests: fix share test to actually share the repo "repo2" is clearly meant to be a share from "repo1" but without sharing bookmarks. However, `hg unshare` was called in the repo, so it had become completely unrelated and thus not testing what it was supposed to test. Differential Revision: https://phab.mercurial-scm.org/D6385

tests: separate out bookmarks tests from test-share.t Differential Revision: https://phab.mercurial-scm.org/D6384

bookmarks: use vfs.tryread() instead of reimplementing it Differential Revision: https://phab.mercurial-scm.org/D6383

bookmarks: use context manager when writing files Differential Revision: https://phab.mercurial-scm.org/D6382

bisect: do not crash with rewritten commits

log: add config for making `hg log -G` always topo-sorted I (and everyone else at Google) have an log alias that adds graph mode and templating. I have another one that builds on the first and also restricts the set of revisions to only show those I'm most likely to care about. This second alias also adds topological sorting. I still sometimes use the first one. When I do, it very often bothers me that it's not topologically sorted (branches are interleaved). This patch adds a config option for always using topological sorting with graph log. The revision set is sorted eagerly, which seems like a bad idea, but it doesn't seem to make a big difference in the hg repo (150ms). I initially tried to instead wrap the user's revset in sort(...,topo), but that seemed much harder. Differential Revision: https://phab.mercurial-scm.org/D6331

log: remove an unnecessary "and opts.get('rev')" condition As Yuya pointed out, the condition is unnecessary since revs.isdescending() would be true if --follow without --rev. Differential Revision: https://phab.mercurial-scm.org/D6372

graphmod: remove support for graph lines mixing parent/grandparent styles (BC) Currently, if the configuration for a graph edge draw style has multiple bytes (at least on python2), it is interpreted as "this is a request to draw the line partially in the style of the parent, partially in the style of the grandparent". This precludes the configuration handling unicode characters (which trigger the `len > 1` check, at least on python2), and I believe was part of the reason that beautifygraph was written the way it was. Talking with the person who implemented this, it appears to have been to achieve feature parity with the rendering of the smartlog extension. I suspect that this isn't actually used outside of that situation, so I think that we can remove it without much issue. This will make it so that multi-character edges are possible, and render any existing configuration that uses this feature with these multiple characters. This is *not* going to adjust the width of everything to make it line up correctly, please see the test that's being modified in this changeset for an example of how the previous configuration now renders. Note also that the previous configuration seems to have been broken, or at least it was behaving in a really non-obvious way - it was avoiding the grandparent character(s) when it should have been displaying them! This is why so many "!" characters changed to "3."; I don't know if this was intentional. Differential Revision: https://phab.mercurial-scm.org/D5112

py3: add 5 new passing tests Differential Revision: https://phab.mercurial-scm.org/D6378

py3: add a r'' to prevent transformer adding b'' # skip-blame because just r'' prefix Differential Revision: https://phab.mercurial-scm.org/D6377

rust-dirstate: call parse/pack bindings from Python A future patch will need to address the issue of Rust module policy, to avoid having ugly duplicate imports and conditionals all over the place. As the rewrite of dirstate in Rust progresses, we will need fewer of those "contact points". Differential Revision: https://phab.mercurial-scm.org/D6350

rust-dirstate: add rust-cpython bindings to the new parse/pack functions This allows for Python code to call `parse/pack_dirstate` transparently. These bindings are heavy given the relatively simple task, as they are bound to implementation details of both the C and Python code. They will be slimmed down in future patches and eventually completely removed once more of the dirstate code has been refactored/rewritten in Rust. Both functions emulate the mutate-on-loop style of the Python and C implementations by looping over changed items in the compatibility layer, instead of at the core functions. Differential Revision: https://phab.mercurial-scm.org/D6349

rust-dirstate: add rust implementation of `parse_dirstate` and `pack_dirstate` Working towards the goal of having a complete Rust implementation of `hg status`, these two utils are a first step of many to be taken to improve performance and code maintainability. Two dependencies have been added: `memchr` and `byteorder`. Both of them have been written by reputable community members and are very mature crates. The Rust code will often need to use their byte-oriented functions. A few unit tests have been added and may help future development and debugging. In a future patch that uses `parse_dirstate` to stat the working tree in parallel - which neither the Python nor the C implementations do - actual performance improvements will be seen for larger repositories. Differential Revision: https://phab.mercurial-scm.org/D6348

changelog: define changelogrevision.p[12]copies for null revision Looks like I missed these in 5382d8f8530b (changelog: parse copy metadata if available in extras, 2017-12-27). `hg debugp[12]copies -r null` fails before this patch. Differential Revision: https://phab.mercurial-scm.org/D6376

copies: write empty entries in changeset when also writing to filelog When writing to both changeset and filelog (during transition), we don't want the reader to waste time by falling back to reading from the filelog when there is no copy metadata. Let's write out empty copy metadata instead (the read path is already prepared for this case). Thanks to Greg for pointing this out. Differential Revision: https://phab.mercurial-scm.org/D6306

rebase: hide help for revisions.Predicates._destautoorphanrebase

unshelve: add space to help

context: default to using branch from dirstate only in workingctx Same reasoning as previous commits: only the workingctx should know about the dirstate. committablectx now seems free of dirstate references. Differential Revision: https://phab.mercurial-scm.org/D6374

context: let caller pass in branch to committablectx.__init__() committablectx.__init__() currently looks up the branch from the dirstate unless it's passed in the extras. memctx.__init__() has a branch argument, but since committablectx.__init__() doesn't accept it, it lets that constructor look up the branch from the dirstate before it overwrites it, which seems awkward. Differential Revision: https://phab.mercurial-scm.org/D6366

context: move contents of committablectx.markcommitted() to workingctx Same reasoning as previous commits: this function updates the dirstate. By not updating the dirstate here, we also fix the close-head test. Differential Revision: https://phab.mercurial-scm.org/D6365

tests: demonstrate that close-head command updates working copy The help text for the command says "...it doesn't change the working directory", so I don't think this is intentional. Differential Revision: https://phab.mercurial-scm.org/D6364

context: move walk() and match() overrides from committablectx to workingctx Same reasoning as previous commit: these functions update the dirstate. Differential Revision: https://phab.mercurial-scm.org/D6363

context: move flags overrides from committablectx to workingctx These read from the dirstate, so they shouldn't be used in other subclasses. Differential Revision: https://phab.mercurial-scm.org/D6362

context: reuse changectx._copies() in all but workingctx This moves the dirstate-specific _copies() implementation from committablectx into workingctx where it should be (I think all dirstate-specific stuff should be moved into workingctx). The part of changectx._copies() that is for producing changeset-wide copy dicts from the filectxs is moved into basectx so it's reused by the other subclasses. The part of changectx._copies() that's about reading copy information from the changeset remains there. This fixes in-memory rebase (and makes `hg convert` able to write copies to changesets). Differential Revision: https://phab.mercurial-scm.org/D6219

overlayworkingctx: don't include added-then-deleted files in memctx If a file (such as a .orig file) is temporarily added to the overlayworkingctx and then deleted, it's still going to be in the _cache dict. In tomemctx(), we created the list of files from _cache.keys(), so the memctx.files() would include the temporary file. That was fine because the list of files was only used in localrepo.commitctx() (I think), where there's an extra filtering of incorrectly removed files (annotated with an inaccurate "update manifest" comment). I'd like to call memctx.files() in another case, but first we need to make it accurate. Differential Revision: https://phab.mercurial-scm.org/D6361

tests: demonstrate loss of changeset copy metadata on rebase Differential Revision: https://phab.mercurial-scm.org/D6360

overlaycontext: allow calling copydata() on clean context We should just report no copy if the context is clean. Differential Revision: https://phab.mercurial-scm.org/D6358

tests: demonstrate another failure with in-memory rebase and copies This is a similar to dd1ab72be983 (test: demonstrate crash with in-memory rebase and copies, 2019-03-14). The new failure started with 57203e0210f8 (copies: calculate mergecopies() based on pathcopies(), 2019-04-11). It happens in the call to mergemod.update() on rebase.py:1268 where we call mergemod.update() to graft a node. Since the mergecopies() rewrite, that calls _related() with the filectx from the overlaywctx instead of a filectx from the changectx where the file was last modified. Either should be fine, so I don't think that's a bug. Differential Revision: https://phab.mercurial-scm.org/D6357

commit: fix a typo ("form p1" -> "from p1") Differential Revision: https://phab.mercurial-scm.org/D6375

automation: initial support for running Linux tests Building on top of our Windows automation support, this commit implements support for performing automated tasks on remote Linux machines. Specifically, we implement support for running tests on ephemeral EC2 instances. This seems to be a worthwhile place to start, as building packages on Linux is more or less a solved problem because we already have facilities for building in Docker containers, which provide "good enough" reproducibility guarantees. The new `run-tests-linux` command works similarly to `run-tests-windows`: it ensures an AMI with hg dependencies is available, provisions a temporary EC2 instance with this AMI, pushes local changes to that instance via SSH, then invokes `run-tests.py`. Using this new command, I am able to run the entire test harness substantially faster then I am on my local machine courtesy of access to massive core EC2 instances: wall: 16:20 ./run-tests.py -l (i7-6700K) wall: 14:00 automation.py run-tests-linux --ec2-instance c5.2xlarge wall: 8:30 automation.py run-tests-linux --ec2-instance m5.4xlarge wall: 8:04 automation.py run-tests-linux --ec2-instance c5.4xlarge wall: 4:30 automation.py run-tests-linux --ec2-instance c5.9xlarge wall: 3:57 automation.py run-tests-linux --ec2-instance m5.12xlarge wall: 3:05 automation.py run-tests-linux --ec2-instance m5.24xlarge wall: 3:02 automation.py run-tests-linux --ec2-instance c5.18xlarge ~3 minute wall time to run pretty much the entire test harness is not too bad! The AMIs install multiple versions of Python. And the run-tests-linux command specifies which one to use: automation.py run-tests-linux --python system3 automation.py run-tests-linux --python 3.5 automation.py run-tests-linux --python pypy2.7 By default, the system Python 2.7 is used. Using this functionality, I was able to identity some unexpected test failures on PyPy! Included in the feature is support for running with alternate filesystems. You can simply pass --filesystem to the command to specify the type of filesystem to run tests on. When the ephemeral instance is started, a new filesystem will be created and tests will run from it: wall: 4:30 automation.py run-tests-linux --ec2-instance c5.9xlarge wall: 4:20 automation.py run-tests-linux --ec2-instance c5d.9xlarge --filesystem xfs wall: 4:24 automation.py run-tests-linux --ec2-instance c5d.9xlarge --filesystem tmpfs wall: 4:26 automation.py run-tests-linux --ec2-instance c5d.9xlarge --filesystem ext4 We also support multiple Linux distributions: $ automation.py run-tests-linux --distro debian9 total time: 298.1s; setup: 60.7s; tests: 237.5s; setup overhead: 20.4% $ automation.py run-tests-linux --distro ubuntu18.04 total time: 286.1s; setup: 61.3s; tests: 224.7s; setup overhead: 21.4% $ automation.py run-tests-linux --distro ubuntu18.10 total time: 278.5s; setup: 58.2s; tests: 220.3s; setup overhead: 20.9% $ automation.py run-tests-linux --distro ubuntu19.04 total time: 265.8s; setup: 42.5s; tests: 223.3s; setup overhead: 16.0% Debian and Ubuntu are supported because those are what I use and am most familiar with. It should be easy enough to add support for other distros. Unlike the Windows AMIs, Linux EC2 instances bill per second. So the cost to instantiating an ephemeral instance isn't as severe. That being said, there is some overhead, as it takes several dozen seconds for the instance to boot, push local changes, and build Mercurial. During this time, the instance is largely CPU idle and wasting money. Even with this inefficiency, running tests is relatively cheap: $0.15-$0.25 per full test run. A machine running tests as efficiently as these EC2 instances would cost say $6,000, so you can run the test harness a >20,000 times for the cost of an equivalent machine. Running tests in EC2 is almost certainly cheaper than buying a beefy machine for developers to use :) # no-check-commit because foo_bar function names Differential Revision: https://phab.mercurial-scm.org/D6319

automation: move image operations to own functions An upcoming commit will need this functionality with slightly different values and it is enough code to not want to duplicate. Let's refactor into standalone functions so it can be reused. Differential Revision: https://phab.mercurial-scm.org/D6318

automation: add --version argument to build-all-windows-packages This lets us pass a version string through when building all Windows packages, just like we can do with the individual commands which produce installers. Differential Revision: https://phab.mercurial-scm.org/D6317

automation: do a force push to synchronize We don't know what the state of the remote is. Force pushing will be more resilient. Differential Revision: https://phab.mercurial-scm.org/D6316

automation: add check that hg source directory is a repo Synchronizing from e.g. source distributions is not yet supported. Let's add a check so we fail with an error message indicating such. Differential Revision: https://phab.mercurial-scm.org/D6315

automation: shore up rebooting behavior There was a race condition in the old code. Use instance.stop()/instance.start() to eliminate it. As part of debugging this, I also found another race condition related to PowerShell permissions after the reboot. Unfortunately, I'm not sure the best way to work around it. I've added a comment for now. Differential Revision: https://phab.mercurial-scm.org/D6288

automation: wait longer for WinRM connection I got a few timeouts waiting for only 120s for the WinRM connection to become available. Increasing to 180s seems to fix. I guess AWS isn't as consistent as I would like :( Differential Revision: https://phab.mercurial-scm.org/D6287

automation: wait for instance profiles and roles Otherwise there is a race condition between creating the resources and us attempting to use them / them becoming available. The role waiter API was recently introduced, so we had to upgrade the boto3 package to get it. Other packages were also updated to latest versions just because. Even with this change, I still run into issues with the IAM instance profile not being available when we attempt to create an EC2 instance using a just-created profile. I'm not sure what's going on. Possibly a bug on Amazon's end. But the new behavior is "more correct." Differential Revision: https://phab.mercurial-scm.org/D6286

automation: don't create resources when deleting things Otherwise running these commands can result in resources being created. In the case of `purge-ec2-resources`, we will create resources only to delete them immediately afterwards! With this change, `purge-ec2-resources` now no-ops if no resources exist. # no-check-commit because foo_bar function name Differential Revision: https://phab.mercurial-scm.org/D6285

automation: detach policies before deleting role You can't delete an IAM role that has attached policies. With this change, the purge-ec2-resources command now works. Differential Revision: https://phab.mercurial-scm.org/D6284

automation: only iterate over our AMIs We can't delete AMIs that we don't own. Iterating over other AMIs won't work and slows down execution. Differential Revision: https://phab.mercurial-scm.org/D6283

remotefilelog: move most setup from onetimesetup() to uisetup() All the wrappers moved in this patch check if remotefilelog is enabled before they change behavior, so it's safe to always wrap. Differential Revision: https://phab.mercurial-scm.org/D6334

remotefilelog: move most functions in onetimeclientsetup() to top level This is how most extensions seem to do it. It makes sure we don't accidentally depend on the captured ui instance. Differential Revision: https://phab.mercurial-scm.org/D6333

tests: avoid the word "dirty" to mean "not a descendant of merge base" The term "dirty" is no longer used in the code since 57203e0210f8 (copies: calculate mergecopies() based on pathcopies(), 2019-04-11). Differential Revision: https://phab.mercurial-scm.org/D6373

releasenotes: add a file in which to record release notes I've just spent a few very boring hours going through the changelog for the 5.0 release (829 commits). We only had 5 commits that used the syntax that the release notes extension expects. This commit adds a file in which we can record important changes. The file should preferably be edited in the patch that makes the important change, but it can also be edited after (I think this is an important benefit compared to the release notes extension). I'm thinking that we can rename the file from "next" to "5.1" or something when it's time, and then we'd create a new "next" file on the default branch. I've used the syntax that we use on the our wiki in the template, but I don't care much that we use any valid syntax at all. The idea is mostly to record important changes when they happen. I expect that some copy editing will be needed at release time anyway. Differential Revision: https://phab.mercurial-scm.org/D6332

record: avoid modifying the matcher passed as a method parameter No problem observed, but I remember the previous pattern causing problems with largefiles and/or subrepos. This special matcher was added in 419ac63fe29c, so directly modifying the `fail` callback was probably an oversight in 44611ad4fbd9. Differential Revision: https://phab.mercurial-scm.org/D6371

sslutil: add support for SSLKEYLOGFILE to wrapsocket I recently learned of a Firefox/Chrome feature that allows wiresharking otherwise-TLS'd network connections. Gloriously, there's a pypi module that enables this same feature on Python, so let's add support for it to Mercurial in case we need to wireshark some HTTPs connections. Differential Revision: https://phab.mercurial-scm.org/D6343

phabricator: add custom vcr matcher to match request bodies Currently when the phabricator extension's conduit output changes the tests don't notice since the default vcr matcher only matches on 'method' and 'uri', not the body. Add a custom matcher that checks the same params are in the body (ignoring ordering). vcr's in-built body matcher can't be used since it fails under py3 with a "UnicodeEncodeError" on the "€ in commit message" tests. The DREV ids have decreased since the recordings were generated against a different phabricator instance to avoid spamming mercurial-devel. Differential Revision: https://phab.mercurial-scm.org/D6347

merge with stable

absorb: be more specific when erroring out on merge commit When you have a merge commit checked out and run `hg absorb`, it would tell you abort: no mutable changeset to change That makes it sound like the problem is public commits when isn't really. Let's be more specific in this case. There was already a test case that test this, so that now prints the new message. I added a new test case that shows the old message (when a public commit is checked out). Differential Revision: https://phab.mercurial-scm.org/D6354

remotefilelog: log when we're about to fetch files I'm debugging a slow client situation and knowing how many files are in the batch request would be a nice thing. Differential Revision: https://phab.mercurial-scm.org/D6353

revset: populate wdir() by its hash or revision number It belongs to the same category as the null hash/revision, and we do handle these virtual identifiers in id()/rev() predicates. Let's do that more consistently.

revset: extract private constant of {nullrev, wdirrev} set I'll add a few more users of this constant to get around wdir identifiers.

help: suggest merge() revset instead of -m/--only-merges Suggested by Dr Rainer Woitok.

tests: update annotate tests to work around simplemerge bug test-annotate.t and test-fastannotate.hg were failing with --pure since 57203e0210f8 (copies: calculate mergecopies() based on pathcopies(), 2019-04-11). It turned out to be because the pure file merge code behaved differently. I'm guessing it's the mdiff.get_matching_blocks() that behaves differently, but I haven't confirmed that. With this content in the base: a a a And this on the local side: a z a And this on the other side: a a a b4 c b6 It produced this conflict: a z a <<<<<<< working copy: b80e3e32f75a - test: c ||||||| base a ======= a b4 c b5 >>>>>>> merge rev: 64afcdf8e29e - test: mergeb I don't care enough about the pure Python code to fix it, so this patch just updates the tests to manually resolve the conflict. Differential Revision: https://phab.mercurial-scm.org/D6351

copies: delete misplaced comment The comment was added in 78d760aa3607 (duplicatecopies: do not mark items not in the dirstate as copies, 2013-03-28). It became misplaced in 3666331164bb (cmdutil: add copy-filtering support to duplicatecopies, 2014-06-07). Then the relevant code was moved far away in 754b5117622f (context: add workingfilectx.markcopied, 2017-10-15). Differential Revision: https://phab.mercurial-scm.org/D6352

phabricator: include branch in the phabread output Depends on D6301 Differential Revision: https://phab.mercurial-scm.org/D6302

phabricator: fallback to reading metadata from diff for phabread Differential Revision: https://phab.mercurial-scm.org/D6301

phabricator: include commit (node) and parent in the local:commits metadata Differential Revision: https://phab.mercurial-scm.org/D6298

copies: remove redundant filtering of ping-pong renames in _chain() We already handle the ping-pong rename case in the filtering step, so there's very little point in doing it in the chaining loop (ping-pong renames are very rare, so I'm not worried about the cost of adding it and then removing it again). Differential Revision: https://phab.mercurial-scm.org/D6344

repair: reword comments that I noticed while working on source formatting I think this is clearer, and one will also keep us from upsetting check-code when other formatting cleanups happen. Differential Revision: https://phab.mercurial-scm.org/D6339

gendoc: nest command headers under category headers Differential Revision: https://phab.mercurial-scm.org/D6329

minirst: support subsubsubsubsections (header level 5) with marker '''' Differential Revision: https://phab.mercurial-scm.org/D6328

gendoc: guarantee that all commands were processed The new logic renders the commands belonging to each category in turn. Commands with an unregistered category are at risk of getting skipped because their category is not in the list. By comparing the list of all commands to a log of processed commands, we can detect commands with unregistered categories and fail with an error message. Differential Revision: https://phab.mercurial-scm.org/D6327

gendoc: group commands by category in man page and HTML help Make Mercurial's man page and HTML help group commands by category, and present the categories in a helpful order. `hg help` already does this; this patch uses the same metadata. This patch uses the same header level for command categories and for commands. A subsequent patch will push the command headers down one level. Differential Revision: https://phab.mercurial-scm.org/D6326

gendoc: indent loop to make next patch more legible Differential Revision: https://phab.mercurial-scm.org/D6325

contrib: have byteify-strings explode if run in Python 2 Differential Revision: https://phab.mercurial-scm.org/D6341

repair: reword comment about bookmarks logic Again, this will help auto-formatting shortly. Differential Revision: https://phab.mercurial-scm.org/D6340

monotone: fix a bogus _() wrapper that was caught when formatting code There was a spurious space after `debug`, which hid the _() inside ui.debug() from check-code. Sigh. While here, wrap things more concisely. Differential Revision: https://phab.mercurial-scm.org/D6338

commit: add ability to print file status after each successful invocation When commands.commit.post-status is enabled, `hg commit` will effectively run `hg status -mardu` after committing. It can help catch mistakes like not committing all needed files or not adding unknown files that should've been part of the just created commit.

tests: flatten repo structure in test-commit.t Let's move to parent directory before `hg init` repos, since they don't need to be nested. It makes amend/strip messages that include full path to the backup bundle shorter, for instance.

lfs: add a TODO file This is a cleaned up and reorganized list of items I sent out about a year ago. But tracking this in the repo (like the narrow extension) gives more visibility in case anyone wants to help out.

copies: make "limit" argument to _tracefile() mandatory We always pass a limit. I think the fact that it was optional was also the reason we checked ">=limit" before we used it. So now we can remove that condition too. Differential Revision: https://phab.mercurial-scm.org/D6335

localrepo: don't use defaults arguments that will never be overridden The commithook() callback will be called when the lock is released. lock.release() calls the callback without arguments, so it was quite confusing to me that this function declared extra arguments. We can just close on the variables in the outer scope instead. Differential Revision: https://phab.mercurial-scm.org/D6336

tags: avoid double-reversing a list Differential Revision: https://phab.mercurial-scm.org/D6337

updatecaches: also warm hgtagsfnodescache Now that a full update of this cache run in a reasonable amount of time, we can warm everything when during a full update.

hgtagsfnodescache: inherit fnode from parent when possible If a changeset does not update the content of `.hgtags`, it means it will use the same file-node (for `.hgtags`) as its parents. In this case we can directly reuse the parent's file-node. We use this property when updating the `hgtagsfnodescache` taking a faster path if we already have a cached value for the parents of the node we are looking at. Doing so provides a large performance boost when looking at a lot of fnodes, especially on repository with very large manifest: timing for `tagsmod.fnoderevs(ui, repo, repo.changelog.revs())` mercurial: (41907 revisions, 1923 files) before: 6.9 seconds after: 2.7 seconds (-54%) pypy: (96266 revisions, 5198 files) before: 80 seconds after: 20 seconds (-75%) mozilla-central: (463411 revisions, 272080 files) before: 7166.4 seconds after: 47.8 seconds (-99%, x150 speedup) On a copy of mozilla-try with about 35K heads ans 1.7M changesets, this moves the computation from many hours to a couple of minutes, making it more interesting to do a full warm up of this cache before computing tags (from a cold cache). There seems to be other performance low hanging fruits, like avoiding the use of changectx or a more revision centric logic. However, the new code is fast enough for my needs right now.

hgtagsfnodescache: handle nullid lookup The null revision is empty, so it `.hgtags` content is `nullid` in regards with the `hgtagsfnodescache`. Dealing with `nullid` will help with the next changeset. Before this change, feeding `nullid` to `hgtagsfnodescache.getfnode` would return a wrong result (fnode for tip).

help: register the 'gpg' command category and give it a description help.py expects extensions to register their command category in the CATEGORY_ORDER and CATEGORY_NAMES variables. Once gendoc.py orders commands by category, in the next patch, it'll assume this registration (and raise an exception on encountering any unregistered categories). Luckily, gpg is the only bundled extension with an unregistered custom category, so let's fix it. Differential Revision: https://phab.mercurial-scm.org/D6324

histedit: Speed up scrolling in patch view mode Store patchcontents into the mode state, avoiding the expensive call to ui for computing the patchcontents. Before this change in large repos histedit patch view mode can be very irresponsive.

histedit: Show file names in multiple line format

repoview: introduce a `experimental.extra-filter-revs` config The option define revisions to additionally filter out of all repository "view". The end goal is to provide and easy to way to serve multiple subset of the same repository using multiple "shares". The simplest use case of this feature is to have one view serving the public changesets and one view also serving the draft. This is currently achievable using the new `server.view` option introduced recently by Joerg Sonnenberger. However, more advanced use cases need more advanced definitions. For example some needs a view dedicated to some release branches, or view that hides security fixes to be released. Joerg Sonnenberger and I discussed this topic at the recent mini-sprint and the both of us have seen real life use cases for this. (This series got written during the same mini-sprint). The feature is fully functional, and use similar cache-fallback mechanism to ensure decent performance. However,there remaining room to ensure each share caches and hooks collaborate with each others. This will come at a later time once users start to actually test this feature on real usecase.

copies: filter out copies from non-existent source later in _chain() _changesetforwardcopies() repeatedly calls _chain(). That is very expensive because _chain() does lookups in the manifest. I hope to split up the function in two parts: 1) simple chaining, not considering end points, and 2) filter out files that don't exist in the end points (and ping-pong copies/renames). This patches gets us closer to that by moving the check for non-existent source later in the function. Now there are no more checks for "src" and "dst" in the first loop; all the filtering of invalid copies is done in the second loop. The code also looks much more consistent now. No measureable impact on `hg debugpathcopies 4.0 4.8`. That shouldn't be surprising since the only case we're doing more checks now is in case of chained copies/renames, which are quire rare in practice. Differential Revision: https://phab.mercurial-scm.org/D6277

copies: clarify mutually exclusive cases in _chain() with a s/if/elif/ If the 'b' dict has a rename from 'x' to 'y', it shouldn't be possible for 'x' to be both (a key) in 'a' and in 'src'. That would mean that 'x' is a file in the source commit and also a rename destination in the intermediate commit. But we currently don't allow renaming files onto existing files, so that shouldn't happen. So let's clarify that by using an "elif" instead of an "if". And if we did allow renaming files onto existing files, we should prefer to use the rename destination in the intermediate commit as source anyway. Differential Revision: https://phab.mercurial-scm.org/D6276

copies: delete a redundant cleanup step in _chain() The check is redundant since d5edb5d3a337 (copies: filter out copies when target is not in destination manifest, 2019-02-14). To test that hypothesis, I made this change in the commit that commit, but all tests still passed. I think the case was necessary before then, we just didn't have tests for it. Differential Revision: https://phab.mercurial-scm.org/D6275

copies: document cases in _chain() Differential Revision: https://phab.mercurial-scm.org/D6274

copies: ignore heuristics copytracing when using changeset-centric algos Differential Revision: https://phab.mercurial-scm.org/D6269

copies: move check for experimental.copytrace==<falsy> earlier I'm going to ignore experimental.copytrace when changeset-centric algorithms are required. This little refactoring makes that easier to add. Differential Revision: https://phab.mercurial-scm.org/D6268

copies: replace .items() by .values() where appropriate As pointed out by Pierre-Yves. Differential Revision: https://phab.mercurial-scm.org/D6266

copies: inline _computenonoverlap() in mergecopies() We now call pathcopies() from the base to each of the commits, and that calls _computeforwardmissing(), which does file prefetching (in the remotefilelog override). So the call to _computenonoverlap() is now pointless (the sets of files from _computenonoverlap() are subsets of the sets of files from _computeforwardmissing()). This somehow also fixes a broken remotefilelog test. Differential Revision: https://phab.mercurial-scm.org/D6256

copies: calculate mergecopies() based on pathcopies() When copies are stored in changesets, we need a changeset-centric version of mergecopies() just like we have a changeset-centric version of pathcopies(). I think the natural way of thinking about mergecopies() is in terms of pathcopies() from the base to each of the commits. So if we can rewrite mergecopies() based on two such pathcopies() calls, we'll get the changeset-centric version for free. That's what this patch does. A nice bonus is that it ends up being a lot simpler. mergecopies() has accumulated a lot of technical debt over time. One good example is the code for dealing with grafts (the "partial/incomplete/dirty" stuff). Since pathcopies() already deals with backwards renames and ping-pong renames, we get that for free. I've run tests with hard-coded debug logging for "fullcopy" and while I haven't looked at every difference it produces, all the ones I have looked at seemed reasonable to me. I'm a little surprised that no more tests fail when run with '--extra-config-opt experimental.copies.read-from=compatibility' compared to before this patch. This patch also fixes the broken cases in test-annotate.t and test-fastannotate.t. It also enables the part of test-copies.t that was previously disabled exactly because mergecopies() needed to get a changeset-centric version. One drawback of the rewritten code is that we may now make remotefilelog prefetch more files. We used to prefetch files that were unique to either side of the merge compared to the other. We now prefetch files that are unique to either side of the merge compared to the base. This means that if you added the same file to each side, we would not prefetch it before, but we would now. Such cases are probably quite rare, but one likely scenario where they happen is when moving from a commit to its successor (or the other way around). The user will probably already have the files in the cache in such cases, so it's probably not a big deal. Some timings for calculating mergecopies between two revisions (revisions shown on each line, all using the common ancestor as base): In the hg repo: 4.8 4.9: 0.21s -> 0.21s 4.0 4.8: 0.35s -> 0.63s In and old copy of the mozilla-unified repo: FIREFOX_BETA_60_BASE^ FIREFOX_BETA_60_BASE: 0.82s -> 0.82s FIREFOX_NIGHTLY_59_END FIREFOX_BETA_60_BASE: 2.5s -> 2.6s FIREFOX_BETA_59_END FIREFOX_BETA_60_BASE: 3.9s -> 4.1s FIREFOX_AURORA_50_BASE FIREFOX_BETA_60_BASE: 31s -> 33s So it's measurably slower in most cases. The most significant difference is in the hg repo between revisions 4.0 and 4.8. In that case it seems to come from the fact that pathcopies() uses fctx.isintroducedafter() (in _tracefile), while the old mergecopies() used fctx.linkrev() (in _checkcopies()). That results in a single call to filectx._adjustlinkrev(), which is responsible for the entire difference in time (in my repo). So we pay a performance penalty but we get more correct code (see change in test-mv-cp-st-diff.t). Deleting the "== f.filenode()" in _tracefile() recovers the lost performance in the hg repo. There were are few other optimizations in _checkcopies() that I could not measure any impact from. One was from the "seen" set. Another was from a "continue" when the file was not in the destination manifest (corresponding to "am" in _tracefile). Also note that merge copies are not calculated when updating with a clean working copy, which is probably the most common case. I therefore think the much simpler code is worth the slowdown. Differential Revision: https://phab.mercurial-scm.org/D6255

tests: add test where copy source is deleted and added back This shows another difference between pathcopies() and mergecopies(): mergecopies() considers files that have been deleted and then added back as different files, but pathcopies() does not. Differential Revision: https://phab.mercurial-scm.org/D6330

merge with stable

obsolete: drop the legacy `_enabled` variable Evolve 8.5.0 stopped setting this, and it would have been easier to figure out why TortoiseHg stopped allowing amends if it would have crashed on the missing variable.

discovery: only calculate closed branches if required The number of new closed branches is required for printing in error message. So let's only calculate them if we need to print error about new branches. Differential Revision: https://phab.mercurial-scm.org/D6314

branchcache: store the maximum tip in a variable inside for loop Instead of assigning self.tiprev multiple times in the for loop, and calling cl.node() on it, let's store that in a temporary variable and assign it in the end of loop. Differential Revision: https://phab.mercurial-scm.org/D6311

tests: demonstrate that rename is followed to wrong parent from merge This test case shows another way that copies are handled differently between `hg st` (pathcopies()) and `hg co -m` (mergecopies). The reason is that pathcopies() calls _tracefiles(), which checks that the file nodeid of an ancestor matches the file nodeid in the base commit. mergecopies() should probably be doing the same. Differential Revision: https://phab.mercurial-scm.org/D6323

test: demonstrate failure to follow rename with shadowed linkrev This shows a difference in handling of copies between `hg st` (pathcopies()) and `hg co -m`. The issue here is that mergecopies() uses the unadjusted linkrev() for determining when to stop walking ancestors. Differential Revision: https://phab.mercurial-scm.org/D6322

tests: slightly modify a linkrev test to prepare for expanding it The test case checks that the copy tracing code doesn't get confused by linkrevs when walking a file's ancestors. This patch chnages the test slightly so a second commit is grafted, thus producing a second "bad" linkrev. I'll use this in the next patch to demonstrate a bug. Differential Revision: https://phab.mercurial-scm.org/D6321

copies: process files in deterministic order for stable tests I also fixed a typo while at it. Differential Revision: https://phab.mercurial-scm.org/D6320

py3: properly reject non-encoded strings given to hgweb

py3: handle meta-path finders that only use pre-python3.4 API

remotefilelog: add missing argument to hg.verify wrapper Differential Revision: https://phab.mercurial-scm.org/D6313

revsetbenchmark: track some simple use of "only" The only revset is quite useful and has various possible optimisation. tracking its timing seems useful.

push: added clear warning message when pushing closed branches(issue6080) Differential Revision: https://phab.mercurial-scm.org/D6038

branch: abort if closing branch from a non-branchhead cset This patch make sure that we abort if the user is trying to close a branch from a cset which is not a branch head. Changes in test file reflect the fixed behaviour. Differential Revision: https://phab.mercurial-scm.org/D6282

branch: add tests which shows branch can be closed from a non-branchhead cset This patch shows that we can close a branch even from a cset which is not a branch head. It was supposed to abort this operation. Next patch will be fixing the issue. Differential Revision: https://phab.mercurial-scm.org/D6281

phabricator: read more metadata from local:commits local:commits metadata can contain branch info, and 'rev' has been superseded by 'commit', see: https://github.com/phacility/arcanist/blob/83661809e532c3fe444a8bf7c7d6936e6377691b/src/repository/api/ArcanistMercurialAPI.php#L281 Differential Revision: https://phab.mercurial-scm.org/D6300

phabricator: don't assume the existence of properties of local:commits Not all the properties are guaranteed to be there, so if we don't check first we could die with a KeyError. Differential Revision: https://phab.mercurial-scm.org/D6299

phabricator: include branch in the diffproperty metadata This does not make Phabricator display the branch in web UI anywhere as that still need us to use creatediff API for that. However a future patch will make phabread use this to include the branch in its `hg import`-able output. Differential Revision: https://phab.mercurial-scm.org/D6297

tests: demonstrate `hg log -r . <file>` linkrev bug Differential Revision: https://phab.mercurial-scm.org/D6309

unionrepo: sync with repository API Differential Revision: https://phab.mercurial-scm.org/D6289

match: remove unused match.__iter__ implementation (API) Differential Revision: https://phab.mercurial-scm.org/D6305

fix: allow fixer tools to return metadata in addition to the file content With this change, fixer tools can be configured to output a JSON object that will be parsed and passed to hooks that can be used to print summaries of what code was formatted or perform other post-fixing work. The motivation for this change is to allow parallel executions of a "meta-formatter" tool to report back statistics, which are then aggregated and processed after all formatting has completed. Providing an extensible mechanism inside fix.py is far simpler, and more portable, than trying to make a tool like this communicate through some other channel. Differential Revision: https://phab.mercurial-scm.org/D6167

merge with stable

phabricator: set local:commits time metadata as an int, not a string Same as arcanist does Differential Revision: https://phab.mercurial-scm.org/D6296

phabricator: use templatefilters.json in writediffproperties Instead of json.dumps, since it makes the code simpler and more readable. This would have been the better option for 8fd19a7b4ed6 but I wasn't aware of it at the time. Differential Revision: https://phab.mercurial-scm.org/D6295

commands: use byteskwargs() in verify() Otherwise Python 3 complains about the missing key. Differential Revision: https://phab.mercurial-scm.org/D6294

match: use raw strings to avoid illegal baskslash escape Python 3.8 was complaining about the invalid escape sequences. Let's use raw strings to avoid the warning and double baskslashes. Differential Revision: https://phab.mercurial-scm.org/D6293

revbranchcache: use context manager in _writerevs() to write to file The other _writenames() is a bit complicated to use context manager. Differential Revision: https://phab.mercurial-scm.org/D6292

revbranchcache: factor logic to write names and revs in separate functions Before this patch, the write function was so populated with upto 4 level of indentation, it was hard to understand what's going on. Differential Revision: https://phab.mercurial-scm.org/D6291

tests: make log style a little easier to read in test-copytrace-heuristics.t Revision numbers are much shorter and easier to read (especially compared to the full nodeids that were used here), so I switched to that. That's also what almost all the commands used (e.g. `hg rebase -s . -d 1`). I updated the two instances that used nodeids. I also made some other little cleanups to the log templates. Differential Revision: https://phab.mercurial-scm.org/D6279

tests: avoid cryptic nodeids in tests/test-rename-merge1.t These two nodeids had not been part of any output before, so one can't know which revision they refer to without adding something like `hg log` before them. It turned out that '.^' was equivalent for both of them, so that's what I replaced them with. Differential Revision: https://phab.mercurial-scm.org/D6280

tests: defines aliases for `hg log` calls in test-copytrace-heuristics.t This also makes the test cases more consistent since a few had missed the ":" in "changeset:" that the others used. Differential Revision: https://phab.mercurial-scm.org/D6278

rust-discovery: implementing and exposing stats() This time, it's simple enough that we can do it in all layers in one shot. Differential Revision: https://phab.mercurial-scm.org/D6233

rust-discovery: cpython bindings for the core logic As previously done with the ancestors submodule, testing for the bindings is provided from Python on a trivial case. Differential Revision: https://phab.mercurial-scm.org/D6232

rust-discovery: starting core implementation Once exposed to the Python side, this core object will avoid costly roundtrips with potentially big sets of revisions. This changeset implements the core logic of the object only, i.e., manipulation of the missing, common and undefined set-like revision attributes. Differential Revision: https://phab.mercurial-scm.org/D6231

rust-dagops: roots Unsuprisingly, the algorithm is much easier than for heads, provided we work on a set in the first place. To improve the signature, a trait for set-likes object would be useful, but that's not an immediate concern. Differential Revision: https://phab.mercurial-scm.org/D6230

rust-dagops: range of revisions This is a Rust implementation for what reachableroots2() does if includepath is True. The algorithmic details and performance notes are included in the documentation comment. Our main use case for now is a Rust counterpart of the partialdiscovery object, so we don't really need bindings yet. Differential Revision: https://phab.mercurial-scm.org/D6229

narrow: also warn when not deleting untracked or ignored files Differential Revision: https://phab.mercurial-scm.org/D6265

setdiscovery: fix a few typos Differential Revision: https://phab.mercurial-scm.org/D6263

copies: delete debug message about "unmatched files new in both" Same reasoning as previous patch. Differential Revision: https://phab.mercurial-scm.org/D6251

copies: delete debug message about changes since common ancestor Same reasoning as previous patch. Differential Revision: https://phab.mercurial-scm.org/D6250

copies: delete debug message about search limit I'm about to rewrite mergecopies() and this message will no longer be emitted then. Let's remove the message now to remove a distraction from that patch. Differential Revision: https://phab.mercurial-scm.org/D6249

copies: move early return for "no copies" case a little earlier We can return before the block that prints debug messages. That block will not be run anyway when there are no copies. Differential Revision: https://phab.mercurial-scm.org/D6248

copies: fix up "fullcopy" with missing entries from "diverge" Similar to the previous patch, but this doesn't even affect tests. It does affect tests if you change them to turn on debug logging. I'm fixing it here so reviewers of the later rewrite patch can hard-code debug logging to be on and more easily compare test results. Differential Revision: https://phab.mercurial-scm.org/D6247

copies: fix up "fullcopy" with missing entries from "copy" This is just a workaround similar to the previous one. It will make it easier to follow later patches. Differential Revision: https://phab.mercurial-scm.org/D6246

merge: remove workaround for issue5020 As I explained in the previous commit, I think the filtering added there is a better fix for the issue, so the workaround from 41f6af50c0d8 (merge: fix crash on criss cross merge with dir move and delete (issue5020), 2017-01-31) should no longer be needed. Differential Revision: https://phab.mercurial-scm.org/D6245

copies: don't include copies that are not in source in directory move I've been working on a rewrite of mergecopies(). I compared the output of the rewritten version with the current version. I noticed that between FIREFOX_NIGHTLY_59_END and FIREFOX_BETA_60_BASE in the mozilla-unified repo, there were many copies that the current version detected that the rewritten version did not. One example was js/src/gc/Iteration.h -> js/src/gc/PublicIterators.h. Then I realized that js/src/gc/Iteration.h doesn't even exist in FIREFOX_NIGHTLY_59_END. This patch adds a filtering step for the "fullcopy" dict. It turns out that that change also affects the test for issue5020 in test-merge-criss-cross.t. The 'dm' action no longer happens there. At first I thought that the test case change meant that this patch was broken, but I think it's actually correct tha the 'dm' action should not happen there. The result of the bid merge is still the same. I suspect this filtering is a better solution for the issue than 41f6af50c0d8 (merge: fix crash on criss cross merge with dir move and delete (issue5020), 2017-01-31). I also suspect that it was broken just a few months earlier by a005c33d0bd7 (mergecopies: add logic to process incomplete data, 2016-10-04). Note that bid merge had been enabled for a few years at that point, since 19903277f035 (merge: use bid merge by default (BC), 2014-10-01). This patch is still just a workaround. It will be cleaned up soon (with the rewrite of mergecopies()). But doing this in a separate patch makes later patches easier to understand and gives a place to explain why this is changing. Differential Revision: https://phab.mercurial-scm.org/D6244

tests: add test for issue5343 (grafting with copies) It seems that issue5353 resulted in a lot of tests in test-graft.t, but the bug actually reported in that issue didn't get a test case. This patch adds one for the "move" and one for the "copy" version of it. I also added a "copy+modify" case, to show what should be a merge conflict. I didn't add one for the "backwards" version of it since the comment says that that was already covered by previous work. The tests added by this patch show the broken behavior (the bug is still open). I suspect the results returned from mergecopies() are not expressive enough to fix this issue: it has a dict for copies to merge with, but that can only give one more filename, but here we need two (one for the path on the remote side and one for the path in the merge base). I want to have it tested anyway since I'm about to refactor mergecopies(). Differential Revision: https://phab.mercurial-scm.org/D6242

chistedit: use context manager to set verbose ui I'm still not exactly sure why this is necessary -- perhaps setting it unconditionally would leak this setting in chg invocations. Regardless, this would have looked very out of place as compared to how this setting is done everywhere else, so at least for the sake of style, let's be consistent with the rest of the codebase.

setdiscovery: stop limiting the number of local head we initially send In our testing this limitation provides now real gain and instead triggers pathological discovery timing for some repository with many heads. See inline documentation for details. Some timing below: Mozilla try repository, (~1M revs, ~35K heads), discovery between 2 clones with 100 head missing on each side before: ! wall 1.492111 comb 1.490000 user 1.450000 sys 0.040000 (best of 20) ! wall 1.813992 comb 1.820000 user 1.700000 sys 0.120000 (max of 20) ! wall 1.574326 comb 1.573500 user 1.522000 sys 0.051500 (avg of 20) ! wall 1.572583 comb 1.570000 user 1.520000 sys 0.050000 (median of 20) after: ! wall 1.147834 comb 1.150000 user 1.090000 sys 0.060000 (best of 20) ! wall 1.449144 comb 1.450000 user 1.330000 sys 0.120000 (max of 20) ! wall 1.204618 comb 1.202500 user 1.146500 sys 0.056000 (avg of 20) ! wall 1.194407 comb 1.190000 user 1.140000 sys 0.050000 (median of 20) pypy (~100 heads, 317 heads) discovery between clones with only 42 common heads before: ! wall 0.031653 comb 0.030000 user 0.030000 sys 0.000000 (best of 25) ! wall 0.055719 comb 0.050000 user 0.040000 sys 0.010000 (max of 25) ! wall 0.038939 comb 0.039600 user 0.038400 sys 0.001200 (avg of 25) ! wall 0.038660 comb 0.050000 user 0.040000 sys 0.010000 (median of 25) after: ! wall 0.018754 comb 0.020000 user 0.020000 sys 0.000000 (best of 49) ! wall 0.034505 comb 0.040000 user 0.030000 sys 0.010000 (max of 49) ! wall 0.019631 comb 0.019796 user 0.018367 sys 0.001429 (avg of 49) ! wall 0.019132 comb 0.020000 user 0.020000 sys 0.000000 (median of 49) Private repository (~1M revs, ~3K heads), discovery from a strip subset, about 100 changesets to be pulled. before: ! wall 1.837729 comb 1.840000 user 1.790000 sys 0.050000 (best of 20) ! wall 2.203468 comb 2.200000 user 2.100000 sys 0.100000 (max of 20) ! wall 2.049355 comb 2.048500 user 2.002500 sys 0.046000 (avg of 20) ! wall 2.035315 comb 2.040000 user 2.000000 sys 0.040000 (median of 20) after: ! wall 0.136598 comb 0.130000 user 0.110000 sys 0.020000 (best of 20) ! wall 0.330519 comb 0.330000 user 0.260000 sys 0.070000 (max of 20) ! wall 0.157254 comb 0.155500 user 0.123000 sys 0.032500 (avg of 20) ! wall 0.149870 comb 0.140000 user 0.110000 sys 0.030000 (median of 20) Same private repo, discovery between two clone with 500 different heads on each side: before: ! wall 2.372919 comb 2.370000 user 2.320000 sys 0.050000 (best of 20) ! wall 2.622422 comb 2.610000 user 2.510000 sys 0.100000 (max of 20) ! wall 2.450135 comb 2.450000 user 2.402000 sys 0.048000 (avg of 20) ! wall 2.443896 comb 2.450000 user 2.410000 sys 0.040000 (median of 20) after: ! wall 0.625497 comb 0.620000 user 0.570000 sys 0.050000 (best of 20) ! wall 0.834723 comb 0.820000 user 0.730000 sys 0.090000 (max of 20) ! wall 0.675725 comb 0.675500 user 0.628000 sys 0.047500 (avg of 20) ! wall 0.671614 comb 0.680000 user 0.640000 sys 0.040000 (median of 20)

peer: introduce a limitedarguments attributes When set to True, it signal that the peer cannot receive too larges arguments and that algorithm must adapt. This should only be True for http peer that does not support argument passed as "post". This will be useful to unlock better discovery performance in the next changesets. I am using a dedicated argument because this is not really a usual "capabilities" things. An alternative approach would be to adds a "large-arguments" to all peer, but the http peers. That seemed a bit too hacky to me.

verify: also check full manifest validity during verify runs Before this changes, `hg verify` only checked if a manifest revision existed and referenced the proper files. However it never checked the manifest revision content itself. Mercurial is expecting manifest entries to be sorted and will crash otherwise. Since `hg verify` did not attempted a full restoration of manifest entry, it could ignore this kind of corruption. This new check significantly increases the cost of a `hg verify` run. This especially affects large repository not using `sparse-revlog`. For now, this is hidden behind the `--full` experimental flag.

verify: introduce an experimental --full flag The flag currently has no effect, see next changeset for details. We introduce the flag as experimental to keep the freedom of changing our mind on the final UI. Note: this patch highlight a small but in `hg help`. An option section is generated even if no option are visible.

verify: introduce a notion of "level" Some checks are slower than others, to help the user to run the checks he needs, we are about to introduce new flag to select faster vs deeper runs. This put the scaffolding in place to do this.

tests: split out separate test for issue5020 The test was added to the existing setup in 41f6af50c0d8 (merge: fix crash on criss cross merge with dir move and delete (issue5020), 2017-01-31). I'm about to make a change that affects that test and it's much easier to follow if the test case for issue5020 is a separate test case. The separate test case is based on what mpm provided in comment 12 on the issue. `hg diff -r 41f6af50c0d8^ tests/test-merge-criss-cross.t` after this patch is pretty small (besides the added test). It's probably easier for reviewers to look at that than to try to understand the diff itself (I don't understand it). Differential Revision: https://phab.mercurial-scm.org/D6243

tests: avoid a rename/delete conflict when updating in test-narrow-update.t After the upcoming rewrite of mergecopies(), this test would otherwise (accurately) start warning about "inside/f1 was deleted and renamed". Differential Revision: https://phab.mercurial-scm.org/D6254

tests: delete unused function in test-rename-merge2.t Differential Revision: https://phab.mercurial-scm.org/D6253