branchmap-v3: introduce a "stop_rev" argument to `headsrevs` The `headsrevs` method of the revlog already have a `revs` argument to compute the headrevs of a limited set of heads. However, it disable the use of the native compiled code to compute the heads, which slows down the branchmap v3 code a lot. The branchmap v3 usage is actually quite constrained as we will always only ignores a part at the top of the graph. So we could be significantly faster. We start by making small change to the python side to improve the situation and introduce the new API. More collaboration with the native code are coming later. This massively speedup operation and close most of the remaining gaps between branchmap-v3 and branchmap-v2. especially on repository with many revs like mozilla-try. A small overhead remains mostly because the `headrevs` logic currently has some inefficiently. We will look into them from there. ### benchmark.name = hg.command.unbundle # bin-env-vars.hg.py-re2-module = default # benchmark.variants.issue6528 = disabled # benchmark.variants.resource-usage = default # benchmark.variants.reuse-external-delta-parent = yes # benchmark.variants.revs = any-1-extra-rev # benchmark.variants.source = unbundle # benchmark.variants.validate = default # benchmark.variants.verbosity = quiet ## data-env-vars.name = netbeans-2018-08-01-zstd-sparse-revlog # bin-env-vars.hg.flavor = default branch-v2: 0.233711 ~~~~~ branch-v3 before: 0.368769 (+57.79%, +0.14) branch-v3 after: 0.239857 (+2.63%, +0.01) # bin-env-vars.hg.flavor = rust branch-v2: 0.235230 ~~~~~ branch-v3 before: 0.372460 (+58.34%, +0.14) branch-v3 after: 0.240972 (+2.44%, +0.01) ## data-env-vars.name = netbeans-2018-08-01-ds2-pnm # bin-env-vars.hg.flavor = rust branch-v2: 0.255586 ~~~~~ branch-v3 before: 0.318907 (+24.78%, +0.06) branch-v3 after: 0.268560 (+5.08%, +0.01) ## data-env-vars.name = mozilla-central-2024-03-22-zstd-sparse-revlog # bin-env-vars.hg.flavor = default branch-v2: 0.339010 ~~~~~ branch-v3 before: 0.349752 (+3.17%, +0.01) branch-v3 after: 0.349389 (+3.06%, +0.01) # bin-env-vars.hg.flavor = rust branch-v2: 0.346525 ~~~~~ branch-v3 before: 0.354300 (+2.24%, +0.01) branch-v3 after: 0.355661 (+2.64%, +0.01) ## data-env-vars.name = mozilla-central-2024-03-22-ds2-pnm # bin-env-vars.hg.flavor = rust branch-v2: 0.380202 ~~~~~ branch-v3 before: 0.396293 (+4.23%, +0.02) branch-v3 after: 0.408851 (+7.54%, +0.03) ## data-env-vars.name = mozilla-unified-2024-03-22-zstd-sparse-revlog # bin-env-vars.hg.flavor = default branch-v2: 0.412165 ~~~~~ branch-v3 before: 0.424769 (+3.06%, +0.01) branch-v3 after: 0.427782 (+3.79%, +0.02) # bin-env-vars.hg.flavor = rust branch-v2: 0.412397 ~~~~~ branch-v3 before: 0.421796 (+2.28%, +0.01) branch-v3 after: 0.422354 (+2.41%, +0.01) ## data-env-vars.name = mozilla-unified-2024-03-22-ds2-pnm # bin-env-vars.hg.flavor = rust branch-v2: 0.429501 ~~~~~ branch-v3 before: 0.443849 (+3.34%, +0.01) branch-v3 after: 0.443197 (+3.19%, +0.01) ## data-env-vars.name = mozilla-try-2024-03-26-zstd-sparse-revlog # bin-env-vars.hg.flavor = default branch-v2: 3.403171 ~~~~~ branch-v3 before: 6.234055 (+83.18%, +2.83) branch-v3 after: 3.819477 (+12.23%, +0.42) # bin-env-vars.hg.flavor = rust branch-v2: 3.454876 ~~~~~ branch-v3 before: 6.307813 (+82.58%, +2.85) branch-v3 after: 3.590284 (+3.92%, +0.14) ## data-env-vars.name = mozilla-try-2024-03-26-ds2-pnm # bin-env-vars.hg.flavor = rust branch-v2: 3.465435 ~~~~~ branch-v3 before: 5.176076 (+49.36%, +1.71) branch-v3 after: 3.633278 (+4.84%, +0.17)

branchmap-v3: filter topo heads using node for performance reason The branchmap currently contains heads as nodeid. If we build a set of revnum with the topological heads, we need to turn the nodeid in the branchmap to revnum to be able to check if they are topo-heads. That nodeid → revnum lookup is "expensive" and adds up to something noticeable if you do it hundreds of thousand of time. Instead we turn all the topo-heads revnums into nodes and build a set. So we can directly test membership of the nodeids stored in the branchmap. That is much faster. Ideally we would have revnum in the branchmap and could directly test revnum against a revnum set and that would be even faster. However that's an adventure for another time. Without this change, the branchmap format "v3" was significantly slower than the "v2" format. With this changes, some of that gap is recovered With rust + persistent nodemap, this overhead was smaller because the extra lookup did not had to to build the nodemap from scratch. In addition the mozilla-unified repository is able to use the "pure_top" mode of branchmap v3, so it was not really affected by this. Future changeset will work of the remaining of the performance gap. ### benchmark.name = hg.command.unbundle # bin-env-vars.hg.py-re2-module = default # benchmark.variants.issue6528 = disabled # benchmark.variants.resource-usage = default # benchmark.variants.reuse-external-delta-parent = yes # benchmark.variants.revs = any-1-extra-rev # benchmark.variants.source = unbundle # benchmark.variants.validate = default # benchmark.variants.verbosity = quiet ## data-env-vars.name = netbeans-2018-08-01-zstd-sparse-revlog # bin-env-vars.hg.flavor = default branch-v2: 0.233711 ~~~~~ branch-v3 before: 0.380994 (+63.02%, +0.15) branch-v3 after: 0.368769 (+57.79%, +0.14) # bin-env-vars.hg.flavor = rust branch-v2: 0.235230 ~~~~~ branch-v3 before: 0.385060 (+63.70%, +0.15) branch-v3 after: 0.372460 (+58.34%, +0.14) ## data-env-vars.name = netbeans-2018-08-01-ds2-pnm # bin-env-vars.hg.flavor = rust branch-v2: 0.255586 ~~~~~ branch-v3 before: 0.317524 (+24.23%, +0.06) branch-v3 after: 0.318907 (+24.78%, +0.06) ## data-env-vars.name = mozilla-central-2024-03-22-zstd-sparse-revlog # bin-env-vars.hg.flavor = default branch-v2: 0.339010 ~~~~~ branch-v3 before: 0.410007 (+20.94%, +0.07) branch-v3 after: 0.349752 (+3.17%, +0.01) # bin-env-vars.hg.flavor = rust branch-v2: 0.346525 ~~~~~ branch-v3 before: 0.410428 (+18.44%, +0.06) branch-v3 after: 0.354300 (+2.24%, +0.01) ## data-env-vars.name = mozilla-central-2024-03-22-ds2-pnm # bin-env-vars.hg.flavor = rust branch-v2: 0.380202 ~~~~~ branch-v3 before: 0.393871 (+3.60%, +0.01) branch-v3 after: 0.396293 (+4.23%, +0.02) ## data-env-vars.name = mozilla-unified-2024-03-22-zstd-sparse-revlog # bin-env-vars.hg.flavor = default branch-v2: 0.412165 ~~~~~ branch-v3 before: 0.438105 (+6.29%, +0.03) branch-v3 after: 0.424769 (+3.06%, +0.01) # bin-env-vars.hg.flavor = rust branch-v2: 0.412397 ~~~~~ branch-v3 before: 0.438405 (+6.31%, +0.03) branch-v3 after: 0.421796 (+2.28%, +0.01) ## data-env-vars.name = mozilla-unified-2024-03-22-ds2-pnm # bin-env-vars.hg.flavor = rust branch-v2: 0.429501 ~~~~~ branch-v3 before: 0.452692 (+5.40%, +0.02) branch-v3 after: 0.443849 (+3.34%, +0.01) ## data-env-vars.name = mozilla-try-2024-03-26-zstd-sparse-revlog # bin-env-vars.hg.flavor = default branch-v2: 3.403171 ~~~~~ branch-v3 before: 6.562345 (+92.83%, +3.16) branch-v3 after: 6.234055 (+83.18%, +2.83) # bin-env-vars.hg.flavor = rust branch-v2: 3.454876 ~~~~~ branch-v3 before: 6.160248 (+78.31%, +2.71) branch-v3 after: 6.307813 (+82.58%, +2.85) ## data-env-vars.name = mozilla-try-2024-03-26-ds2-pnm # bin-env-vars.hg.flavor = rust branch-v2: 3.465435 ~~~~~ branch-v3 before: 5.381648 (+55.30%, +1.92) branch-v3 after: 5.176076 (+49.36%, +1.71)

rev-branch-cache: properly ignores unaligned trailing data Previously, trailing data could lead to crash and would be written back to disk, disaligning all new data… This is no longer the cases. This was detected while playing with branchmap-v3 that access the rev-branch-cache much more aggressively.

rev-branch-cache: stop pretending we will overwrite data when we don't We were issuing a message about overwriting data even when we were about to write 0 bytes in pratice. This is silly. Instead we point at the extra data remaining in the file (in case someone is using debug to debug something).

rev-branch-cache: fix message about overwritten data If there is data at the end we ignore, we should not count them as overwritten. This reveal that we something don't overwrite anything, this will be taken car of in the next changeset.

branchmap-v3: detect invalid headerline and raise error Otherwise, broken cache file would be accept as valid, but empty.

head-revs: merge the two inner_headrevs… variants Now that there is only one method, it does not make sense to have two different "inner" method. This is especially true as we are about to add another parameter to the method. So we clean up before that.

head-revs: move hg-cpython's inner_headrevsfiltered closer to inner_headrevs This is pure code movement, it make a coming changesets significantly clearer.

head-revs: teach the pure indexes about the `headrevs` method Having this computation done at the index level unify the API and remove revlog side complexity. It might also be a front runner of handing more responsability to the index.

head-revs: remove the `headrevsfiltered` method on the index This method is no longer called and can be removed.

head-revs: stop calling headrevsfiltered The `headrevs` method have been accepting filter since Mercurial 3.2¹. I guess we can rely on it for now. (except the Rust extension that just gained this capability, but it has it now)

rust-cpython: also accept the `filteredrevs` argument in index.headrevs The C version have been accepting this argument since Mercurial 3.2, lets align the Rust index here. This will make it possible to simplify the code in later changesets.

branchmap: use the proper experimental name in cacheutil Otherwise they are not properly copied around.

interfaces: introduce and use a protocol class for the `mpatch` module See f2832de2a46c for details when this was done for the `bdiff` module. Two things worth pointing out- 1) The `cffi` module "inherits" the `pure` implementation of `patchedsize()` because of its wildcard import. 2) It's odd that the `mpatchError` lives in both `pure` and `cext` modules. I initially thought to move the exception into the new class, and make the existing class name an alias to the class in the new location, but the exception is created in C code by the `cext` module, so that won't work. I don't think a protocol class is approriate, because there's nothing special about the class to distinguish from any other `Exception`. Fortunately, nobody is catching this exception in core, so we can kick the can down the road.

branching: merge stable into default

doctest: use the system hg to find the list of file to tests Same as what we do for the `.t` tests, but more manually.

relnotes: skip the test if the source repository is not readable The test want to run the relnot extension, with the tested mercurial, on the original repository. This is not always possible (e.g. when running with --pure and the repository use zstd for example). So we skip the test in this case.

check-code: document reason and suggest alternative to exit code negation The check was introduced in f48b075ff088. We trust the explanation from there.

ci: drop the HGMODULEPOLICY overwrite The test runner have been setting them properly for a while now. So we don't need to do it manually anymore.

test: preserve the existing module policy config for testrepohg resuse The test runner is, righfully, setting HGMODULEPOLICY to match the flavor requests to run the tests. However this mess with the `testrepohg` ability to run properly, as the newly set policy might be incompatible with its installation, or its ability to read the test repo. So we preserve the initial value in a dedicated variable and use it in the `testrepohg` helper.

module-policy: ignore empty module policy This make the variable easier to work with, the empty value is not ambiguous about not wanting to get in the way.

tests: provide an alternate fake lock for filesystems without symlink support

tests: disable `worker.backgroundclose` to stabilize a test on Windows TIL that `worker.enabled=0` doesn't prevent these workers from spinning up. At any rate, there's already a whole lot of conditionalized output following `cat client.log`, the placement of the "starting 4 threads for background file closing" message seems unstable, and we don't care about those worker threads here. Preventing the message is better for test maintenance.

tests: fix lock file path mangling in `test-racy-mutations.t` on Windows I guess `$TESTTMP_FORWARD_SLASH` gets translated by MSYS. This was in the `.foo_commit_out` file: sh: C;C:\\MinGW\\msys\\1.0\\Users\\Matt\\AppData\\Local\\Temp\\hgtests.1qc8jmdl\\child2\\test-racy-mutations.t-skip-detection\\waitlock_editor.sh: $ENOENT

tests: stabilize `test-status-eacces.t` on Windows As noted earlier, `chmod` doesn't complain in MSYS, but also doesn't alter the file permissions such that they are unreadable. I'm guessing the other lines of output in this area that are gated on `rhg` (or not) will also need this, but I don't want to dig too deeply into something that is apparently working well enough.

run-tests: bump the default timeout on Windows to 4x the normal value There are a ridiculous number of tests that timeout on Windows with the 360 sec default (~60). And because of the bug where timed out tests still run to completion before the results are thrown away[1], the timeout does nothing but waste time, so there's no reason to try to find a lower value that still works. For reference on my system: # Ran 909 tests, 116 skipped, 119 failed. python hash seed: 2052473208 real 151m44.322s user 0m0.077s sys 0m0.046s [1] I thought that I wrote a bug for this, but search isn't finding it.

run-tests: bump the minimum python to 3.8 Presumably this was an oversight when hg was updated to 3.8.

tests: stabilize `test-sparse.t` on Windows One of the reserved characters for path values is '*', so it can't be used. Fortunately, missing this seems to not get in the way of any other tests, and it is removed shortly after with `rm -r foo*bar`, and the extant 'foo-bar' matches the pattern.

tests: fix a test hang on Windows when setting a debuglock I have no idea why, but running the `hg -R auto-upgrade debuglock --set-lock` command near the end of `test-upgrade-repo.t` hangs the test. It does background the process and `killdaemons.py` runs without error, but control doesn't return to `run-tests.py` until the process is manually killed. I did notice that `$!` in MSYS is *not* the PID of the process that got backgrounded, even when a simple `sleep 60 &` is run in MSYS without the *.t file. When `killdaemons.py` is run manually with the PID in ProcessExplorer, the backgrounded process terminates immediately, and returns control to `run-tests.py`. This looks like it would be a race, but the test waits 10s for the lock file to appear before attempting to kill the process, so there's time. `hg serve` has a `--pid-file` option to write the pid to the file, but this is only a debug command, so I'm not bothering with cluttering the command line.

tests: conditionalize `chmod` usage in `test-upgrade-repo.t` While the command itself doesn't error out on Windows, it also doesn't make the filesystem readonly. Therefore the repo gets altered to drop dirstate-v2, and puts it out of sync with that happens on Linux.