shelve: removed redundant merge detection method Differential Revision: https://phab.mercurial-scm.org/D6547

rust-dirstate: call new "dirs" rust implementation from Python This is a simple module attribute replacement, will take precedence over the Python and C implementations. Differential Revision: https://phab.mercurial-scm.org/D6395

rust-dirstate: add "dirs" rust-cpython binding There is an obvious performance and memory issue with those bindings on larger repos as it copies and allocates everything at once, round-trip. Like in the previous patch series, this is only temporary and will only get better once we don't have large data structures going to and from Python. Differential Revision: https://phab.mercurial-scm.org/D6394

rust-dirstate: add "dirs" Rust implementation Following the work done in d1786c1d34fa and working towards the goal of a complete Rust implementation of the dirstate, this rewrites the `dirs` class. There is already a C implementation, which relies heavily on CPython hacks and protocol violations for performance, so I don't expect this to perform as well for now, as this is very straight-forward code. The immediate benefits are new high-level documentation and some unit tests. Differential Revision: https://phab.mercurial-scm.org/D6393

relnotes: added description about statemod._statecheck Differential Revision: https://phab.mercurial-scm.org/D6557

statecheck: shifted defaults to addunfinished() This shifts the definitions and defaults of `_statecheck()` class to `addunfinished()` registration method. Differential Revision: https://phab.mercurial-scm.org/D6583

statecheck: added support for cmdutil.afterresolvedstates This removes `afterresolvedstates` from `cmdutil` and adds support for it in `_statecheck` class. A new flag `continueflag` is added to the class to check whether an operation supports `--continue` option or not. Tests remain unchanged. Differential Revision: https://phab.mercurial-scm.org/D6551

statecheck: added support for STATES This removes `STATES` from `state.py` and adds support to `statecheck` class to handle its features. `getrepostate()` function is modified accordingly. This adds a method 'cmdutil.addunfinished()' for appending to the unfinishedstate list so as to keep 'merge' and 'bisect' at the last. This also makes two separate message formats for `checkunfinished()` and `getrepostate()` as there were previously present. Results of test changed are shown. Differential Revision: https://phab.mercurial-scm.org/D6503

state: moved cmdutil.STATES and utilities to state.py This commit moves `cmdutil.STATES` and adjoining functions to `state.py`. The existing users are updated accordingly. Tests remain unchanged. Differential Revision: https://phab.mercurial-scm.org/D6502

state: created new class statecheck to handle unfinishedstates For the purpose of handling states for various multistep operations like `hg graft`, `hg histedit`, `hg bisect` et al a new class called statecheck is created .This will help in having a unified approach towards these commands and handle them with ease. The class takes in 4 basic arguments which include the name of the command, the name of the state file associated with it , clearable flag , allowcommit flag. This also also adds the support of`checkunfinished()` and `clearunfinished()` to the class. Tests remain unchanged. Differential Revision: https://phab.mercurial-scm.org/D6501

states: moved cmdutil.unfinishedstates to state.py This moves `cmdutil.unfinishedstates`, `checkunfinished()`,`clearunfinished()` to `state.py`. the already existing users of this module are updated accordingly. Test results remain unchanged. Differential Revision: https://phab.mercurial-scm.org/D6484

rebase: fix in-memory rebasing of copy of empty file Classic Python mistake of unintentionally treating None and empty string the same. Differential Revision: https://phab.mercurial-scm.org/D6570

tests: demonstrate broken in-memory rebase of copy to empty file Differential Revision: https://phab.mercurial-scm.org/D6569

zsh: enable completion support for chg as well When verifying this change, you may need to clear/rebuild the completion cache; I did this by deleting the ~/.zcompdump file and then starting a new shell. Differential Revision: https://phab.mercurial-scm.org/D6574

py3: make catapult usable from the test runner in py3 Differential Revision: https://phab.mercurial-scm.org/D6577

py3: use integer division for the value passed to xrange Differential Revision: https://phab.mercurial-scm.org/D6576

pycompat: make fewer assumptions about sys.executable There are many Python "bundlers" which create an archive to run a Python binary from, and they may not set sys.executable at all - handle that case properly, especially to run tests. Differential Revision: https://phab.mercurial-scm.org/D6575

update: fix spurious unclean status bug shown by previous commit The crux of the problem is: - the dirstate is corrupted (the sizes/dates are assigned to the wrong files) - because when worker.worker is used with a return value (batchget in merge.py here), the return value when worker.worker effectively parallelizes is permuted - this is because worker.worker's partition of input and combination of output values are not inverses of one another: it split [1,2,3,4,5,6] into [[1,3,5],[2,4,6]], but combines that into [1,3,5,2,4,6]. Given that worker.worker doesn't call its function argument on contiguous chunks on the input arguments, sticking with lists means we'd need to know the relation between the inputs of worker.worker function argument (for instance, requiring that every input element is mapped to exactly one output element). It seems better to instead switch return values to dicts, which can combined reliably with a straighforward restriction. Differential Revision: https://phab.mercurial-scm.org/D6581

tests: show bug in update introduced in 87a34c767384 As reported by Martin at https://phab.mercurial-scm.org/D6475. Differential Revision: https://phab.mercurial-scm.org/D6580

copies: document how 'copies' dict instances are reused We avoid copying these instances as much as we can, so it's not obvious what's safe to do with them. This patch tries to explain what is safe and what is not. Differential Revision: https://phab.mercurial-scm.org/D6578

copies: simplify merging of copy dicts on merge commits After we removed some filtering in 35d674a3d5db (copies: don't filter out copy targets created on other side of merge commit, 2019-04-18), we will always include all entries from "copies1", so we can simplify the code based on that. Differential Revision: https://phab.mercurial-scm.org/D6561

copies: remove a redundant matcher filtering in _changesetforwardcopies() We filter before pushing items on the queue, so we don't need to filter after popping. Differential Revision: https://phab.mercurial-scm.org/D6560

copies: delete obsolete comment in _changesetforwardcopies() IIRC, the comment applied to the filtering we did before 35d674a3d5db (copies: don't filter out copy targets created on other side of merge commit, 2019-04-18). Differential Revision: https://phab.mercurial-scm.org/D6559

merge with stable

copies: avoid reusing the same variable for two different copy dicts "childcopies" is initally the copies the current changeset to one of its children and then we reassign it with the copies from the start of the chain to the child. Let's use different names for these two things. Differential Revision: https://phab.mercurial-scm.org/D6564

drawdag: don't crash when writing copy info to changesets When writing copies to the changeset, localrepo.commitctx() will call ctx.p1copies() and ctx.p2copies(). These crashed on simplecommitctx because they ended up trying to access the manifest. drawdag doesn't support copies at all, so we can simply override the methods to return empty dicts. Differential Revision: https://phab.mercurial-scm.org/D6565

merge with stable

log: pass getcopies() function instead of getrenamed() to displayer (API) This reduces the duplication between the two displayer functions (and between them and scmutil.getcopiesfn()). It's still more code than two patches ago, but there's less duplication. Differential Revision: https://phab.mercurial-scm.org/D6546

copies: create helper for getting all copies for changeset There are a few places where we get all the copies for a changeset (at least the {file_copies} template and in two places in `hg log --copies` code). These places currently call scmutil.getrenamedfn() to get a caching "getrenamed" function. They all use it in a similar way. We will be able to reuse more code by having a function for getting all the copies for a changeset. This patch introduces such a function. It uses it in the {file_copies} template to show that it works. It relies on the existing scmutil.getrenamedfn() for caching in the filelog-centric case. Differential Revision: https://phab.mercurial-scm.org/D6545

logcmdutil: also check for copies in null revision and working copy It's safe (and fast) to look for copies in the null revision, and it's incorrect not to look for them in the working copy, so let's look in both places. Differential Revision: https://phab.mercurial-scm.org/D6544

tests: demonstrate missing copy information in working copy with graphlog Differential Revision: https://phab.mercurial-scm.org/D6543

remotefilelog: handle copies in changesets in getrenamedfn() override E.g. the {file_copies} template keyword didn't work with copies in changesets before this patch because remotefilelog overrides the getrenamedfn() and didn't handle the changeset-centric case. Differential Revision: https://phab.mercurial-scm.org/D6542

remotefilelog: check if RFL is enabled in getrenamedfn() override In 8a0e03f7baf4 (remotefilelog: move most setup from onetimesetup() to uisetup(), 2019-05-01), I said: All the wrappers moved in this patch check if remotefilelog is enabled before they change behavior, so it's safe to always wrap. That was clearly a lie, because getrenamedfn() didn't. That made e.g. `hg log -T {file_copies}` unbearably slow. This patch fixes that. Differential Revision: https://phab.mercurial-scm.org/D6541

relnotes: document template support for `hg root` Differential Revision: https://phab.mercurial-scm.org/D6540

remotefilelog: tell runbgcommand to not block on child process startup These two invocations will always find a binary because they're re-running hg. As a result, we can skip waiting for the subprocess to start running and save a little bit of wall-time. Differential Revision: https://phab.mercurial-scm.org/D6539

procutil: allow callers of runbgcommand to assume the process starts Experimentally starting the subprocess can take as much as 40ms, and for some of our use cases that's frivolous: we know the binary will start, and if it doesn't we'd only ever ignore it and continue anyway. This lets those use cases be faster. Differential Revision: https://phab.mercurial-scm.org/D6537

shallowrepo: remove backwards compat code that predates in-tree remotefilelog Differential Revision: https://phab.mercurial-scm.org/D6538

commit: make the error message more specific while aborting branch closing Differential Revision: https://phab.mercurial-scm.org/D6493

commit: add a check if it is trying to close an already closed branch head It would check if the revision we are going to close is already a closed branch head and print the error message accordingly. Differential Revision: https://phab.mercurial-scm.org/D6491

strip: move checksubstate() to mq (its only caller) Differential Revision: https://phab.mercurial-scm.org/D6536

strip: use bailifchanged() instead of reimplementing it This also means that we get the standard error messages (see changed test cases). Differential Revision: https://phab.mercurial-scm.org/D6535

strip: remove unused excsuffix argument from checklocalchanges() It was only used by mq, and mq now has its own copy of the function. Differential Revision: https://phab.mercurial-scm.org/D6534

mq: remove dependency on strip's checklocalchanges() Some of the functionality in strip.checklocalchanges() was only used by mq, so let's move it to mq so we can clean up strip. Differential Revision: https://phab.mercurial-scm.org/D6533

copies: avoid calling matcher if matcher.always() When storing copy information in the changesets (experimental.copies.read-from=changeset-only), this patch speeds up hg debugpathcopies FENNEC_58_0_2_BUILD1 FIREFOX_59_0b8_BUILD2 from 5.9s to 4.7s. At the start of this series (b162229e), that command took 18min. Differential Revision: https://phab.mercurial-scm.org/D6422

copies: avoid unnecessary copying of copy dict When storing copy information in the changesets, this patch speeds up hg debugpathcopies FENNEC_58_0_2_BUILD1 FIREFOX_59_0b8_BUILD2 from 11s to 5.9s. That command takes 6.2s when storing copy information in filelogs. Differential Revision: https://phab.mercurial-scm.org/D6421

copies: don't filter out copy targets created on other side of merge commit If file X is copied to Y on one side of merge and the other side creates Y (no copy), we would not mark that as copy. In the changeset-centric pathcopies() version, that was done by checking if the copy target existed on the other branch. Even though merge commits are pretty uncommon, it still turned out to be too expensive to load the manifest of the parents of merge commits. In a repo of mozilla-unified converted to storing copies in changesets, about 2m30s of `hg debugpathcopies FIREFOX_BETA_59_END FIREFOX_BETA_60_BASE` is spent on this check of merge commits. I tried to think of a way of storing more information in the changesets in order to cheaply detect these cases, but I couldn't think of a solution. So this patch simply removes those checks. For reference, these extra copies are reported from the aforementioned command after this patch: browser/base/content/sanitize.js -> browser/modules/Sanitizer.jsm testing/mozbase/mozprocess/tests/process_normal_finish_python.ini -> testing/mozbase/mozprocess/tests/process_normal_finish.ini testing/mozbase/mozprocess/tests/process_waittimeout_python.ini -> testing/mozbase/mozprocess/tests/process_waittimeout.ini testing/mozbase/mozprocess/tests/process_waittimeout_10s_python.ini -> testing/mozbase/mozprocess/tests/process_waittimeout_10s.ini Since these copies were created on one side of some merge, it still seems reasonable to include them, so I'm not even sure it's worse than filelog pathcopies(), just different. Differential Revision: https://phab.mercurial-scm.org/D6420

copies: do full filtering at end of _changesetforwardcopies() As mentioned earlier, pathcopies() is very slow when copies are stored in the changeset. Most of the cost comes from calling _chain() for every changeset, which is slow because it needs to read manifests. It needs to read manifests to be able to filter out copies that are were created in one commit and then deleted. (It also filters out copies that were created from a file that didn't exist in the starting revision, but that's a fixed revision across calls to _chain(), so it's much cheaper.) This patch changes from _chainandfilter() to just _chain() in the main loop in _changesetforwardcopies(). It instead removes copies that have subsequently been removed by using ctx.filesremoved(). We thus rely on that to be fast. It timed this command in mozilla-unified: hg debugpathcopies FIREFOX_59_0b3_BUILD2 FIREFOX_BETA_59_END It took 18s before and 1.1s after. It's still faster when copy information is stored in filelogs: 0.70s. It also still gets slow when there are merge commits involved, because we read manifests there too. We'll deal with that later. Differential Revision: https://phab.mercurial-scm.org/D6419

rust-filepatterns: add comment about Windows path handling As I replied to the Phabricator message, this is wrong. And I even suspect it wouldn't compile because of multiple type mismatches. I think, in Rust where type system is rock solid, we can live with UTF-8 strings except for the bottom storage layer and the top UI/command layer. We'll still have to get around undecodable characters not to be lost, but I think it's okay to drop such filenames from match result if they don't match in UTF-8 world, not in Latin-1 world.

rust-filepatterns: silence warning of non_upper_case_globals

rust: update Cargo.lock to include @generated comment cargo 1.34.0 of Debian sid inserts this comment, and I'm tired of reverting the change every time I do make local. https://github.com/rust-lang/cargo/commit/bd0e4a08471b8bc7957829b4fd294b8985d4fa2d

merge with stable

lfs: correct an error in the TODO file

cat: don't prefetch files unless the output requires it It's a waste to cache lfs blobs when cat'ing the raw data at best, but a hassle debugging when the blob is missing. I'm not sure if there are other commands that have '{data}' for output, and if there's a general way to prefetch on that keyword. It's interesting that the verbose output seems to leak into the JSON output, but that seems like an existing bug.

tracing: add support for emitting counters Differential Revision: https://phab.mercurial-scm.org/D6526

tracing: extract tracing-active logic to separate function I'm about to add support for counters, and want to avoid duplicating this logic. Differential Revision: https://phab.mercurial-scm.org/D6525

catapipe: add support for COUNTER events Differential Revision: https://phab.mercurial-scm.org/D6524