changing-files: add a debug command display changed files The binary output from sidedata is useful to verify the underlying data do not get corrupted. However having a human readable version is much simpler for debuging the changed files data itself. So we add a debug command to dump this information and we use it in the tests. Differential Revision: https://phab.mercurial-scm.org/D9125

side-data: add a test to check sidedata upgrade We want to make sure that sidedata computed at commit time and during upgrade are the same. To do so, we upgrade the repository in `test-copies-chain-merge.t` since the file is checking all the special case that we need to catch. The sidedata content in both case should be the same. Differential Revision: https://phab.mercurial-scm.org/D9124

changing-files: split the changing files computation from encoding The side data encoding is already in its own function, now we move the changing files computation in it own function, it will receive more updates in the coming changesets. Differential Revision: https://phab.mercurial-scm.org/D9123

salvaged: record salvaged in ChangingFiles at commit time The new code is a simple but effective way to detect this information. We might be able to move it inside the various conditionnal above, but I want to focus on simplicity until we have a full working stack. It is worth noting that if we record the information in the ChangingFiles object, it is not persisted yet. This will comes with later changesets. Differential Revision: https://phab.mercurial-scm.org/D9120

salvaged: track removal-candidates in more cases If we want to use this information for copy tracing, then we need to be able to record it. First we unlock the recording of deletion candidates, and we will actually record the data in the next changeset. Differential Revision: https://phab.mercurial-scm.org/D9119

salvaged: explicitly skip salvaged file while encoding This is a temporary measure to avoid diverging test cases (between commit time sidedata and upgraded sidedata) while we implement salvaged tracking in the different cases. This will be dropped soon. Differential Revision: https://phab.mercurial-scm.org/D9122

changing-files: add a "salvaged" set to track file that were not removed We need this set for the copy tracing algorithm. See documentation for details about this set. Differential Revision: https://phab.mercurial-scm.org/D9118

tests: back out accidentally queued changeset 0627cd03b1e9 I accidentally queued other people's patches on top of my own and pushed mine along with theirs.

tests: fix test-check-module-imports.t broken by D9150 Differential Revision: https://phab.mercurial-scm.org/D9177

tests: make test-convert-git.t work across changed default branch name Recent git versions default to "main" instead of "master", which breaks our test. Differential Revision: https://phab.mercurial-scm.org/D9174

tests: update test-copies-chain-merge.t to not use empty files Merging empty files is not very interesting or realistic.

tests: only diff the `Cargo.lock` in `test-check-cargo-lock.t` While not perfect (you could be running tests while developing with a justified modification to `Cargo.lock`), that use-case is small enough that I don't think it's really going to be an issue. This stops the test from failing for unrelated changes when running it locally. Differential Revision: https://phab.mercurial-scm.org/D9170

tests: update/remove outdated comments in test-copies-chain-merge.t 1b8fd4af3318 fixed the test case but missed updating the comments mentioning the bug. Thanks for @martinvonz who spotted these outdated comments. Differential Revision: https://phab.mercurial-scm.org/D9168

churn: leverage logcmdutil to filter revisions by --date

cmdutil: remove remainder of old walkchangerevs() implementation

cmdutil: rewrite walkchangerevs() by using logcmdutil functions cmdutil.walkchangerevs() now takes (revs, makefilematcher) in place of (match, opts), and only provides the "windowing" functionality. Unused classes and functions will be removed by the next patch. "hg grep --follow" (--all-files) is still broken since there is no logic to follow copies while traversing changelog, but at least, it does follow the DAG.

grep: filter target files by matcher Prepares for the migration to logcmdutil's logic, where cmdutil.walkchangerevs() won't always build an exact set of paths to be scanned.

cmdutil: make walkchangerevs() call prepare with matcher instead of filenames Prepares for migrating walkchangerevs() to logcmdutil's logic, which provides matcher-based interface.

grep: add option for logcmdutil.makewalker() to not filter revs by file pats This is needed to implement "grep --all-files".

rust-parsers: use in-place mutation instead of allocating a new `Vec` This is not done for the `dirstate-tree` feature, since it lacks `iter_mut`. Differential Revision: https://phab.mercurial-scm.org/D9136

revlog: pre-compute null revlog item for pure version The dynamically created tuple shows up in memory profiles and the use of nullid prevents the normal constant building to work. Differential Revision: https://phab.mercurial-scm.org/D9154

packaging: normalize - to _ in WiX Id values - is not a valid identifier character in WiX Ids. So let's normalize accordingly. I discovered this issue after a subsequent change which introduces a directory with a - in its name. Differential Revision: https://phab.mercurial-scm.org/D9147

contrib: install Rust 1.46.0 Let's use the latest/greatest version of Rust in the automation environment. Differential Revision: https://phab.mercurial-scm.org/D9146

contrib: install latest versions of Python Let's keep our build environment modern. We upgrade 3.5-3.8 on Linux. Just 3.7 and 3.8 on Windows because we don't use <3.7 on Windows. Differential Revision: https://phab.mercurial-scm.org/D9145

help: extract logic for listing commands and topics Differential Revision: https://phab.mercurial-scm.org/D9134

copies: directly pass a changes object to the copy tracing code The object contains all the data we need. For example, the `is_merged` callback can now use the associated precomputed data. This will be useful again soon when the `salvaged` set will be introduce to solve the issue with delete file reverted during a merge. See 4b582a93316a and 14be07d5603c for details. Differential Revision: https://phab.mercurial-scm.org/D9117

copies: no longer change the sidedata flag With the new sidedata storage that include data about all file changes, every revision has one, so the sidedata flag is not longer a good way to spot changeset with copy information. So we drop this check to simplify the code We optimisation itself provided an interesting speedup, so we will likely reintroduce something similar, with a dedicated flag, in the future. Differential Revision: https://phab.mercurial-scm.org/D9116

copies: use dedicated `_revinfo_getter` function and call We want to return data in a different form, so we need different revinfo function. At that point it make sense to have different getter. Differential Revision: https://phab.mercurial-scm.org/D9115

copies: make two version of the changeset centric algorithm They are two main ways to run the changeset-centric copy-tracing algorithm. One fed from data stored in side-data and still in development, and one based on data stored in extra (with a "compatibility" mode). The `extra` based is used in production at Google, but still experimental in code. It is mostly unsuitable for other users because it affects the hash. The side-data based storage and algorithm have been evolving to store more data, cover more cases (mostly around merge, that Google do not really care about) and use lower level storage for efficiency. All this changes make is increasingly hard to maintain de common code base, without impacting code complexity and performance. For example, the compatibility mode requires to keep things at different level than what we need for side-data. So, I am duplicating the involved functions. The newly added `_extra` variants will be kept as today, while I will do some deeper rework of the side data versions. Long terms, the side-data version should be more featureful and performant than the extra based version, so I expect the duplicated `_extra` functions to eventually get dropped. Differential Revision: https://phab.mercurial-scm.org/D9114

changing-files: retrieve changelogrevision.files from the sidedata block The `files` field is know to have issue, using a list with fixed, and fixable, computation can only help. For example, using a fixes `files` field would be enough to fix issue6219 once this feature get usable in production. We focus on having thing working for now, we will deal with performance later. Right now we have a ironic situation were we parse sorted value from disk to turn them into a set and then having to sort it again. Differential Revision: https://phab.mercurial-scm.org/D9092