ci: shard the test run on mac os X This should comes with some benefit: - spread the load across more runner, - reduce the real-time CI run, - reduce the "retry" run when we need them. We start with the Mac jobs, but that would be tremendously useful for Windows too. For linux, we need to reduce the startup overhead for this to be worth it. Building smaller image and speeding up clone should help with that.

run-tests: implement crude sharding support It will help to spread the testing load across more CI runners.

ci: have the mac test run if you trigger building the mac wheel The mac test job now depends on the wheel building. And the wheel building is manual. So if the mac test job is set to "on_success" if will be "skipped" by default, and automatically run if the wheel are build. That is especially handy as we are about to shard that test and that the UI for manual sharded test sucks.

ci: adjust the starting port range to runner concurrency If multiple job runs on the same runner, they should not use the port range.

ci: move the "tempory work dir" to "concurrency-safe" location Lets not use a global location and move at the root of the directory dedicated to the job.

ci: abstract the of absolute /tmp/ path We now have a TMP_WORK_DIR directory that we can update to a more sensible value in the next changesets.

ci: rationalize variable usage The usage of "extends" allow to skip a lot of duplication. We also introduce more fine grained variables to help finer override.

ci: move some variables closer to their usage These were defined globaly because we had trouble making them inherited. This is now fixed, so we can get them were they belong.

ci: use extends instead of <<: *x The old form is a yaml construct that make it hard to share variable definition. The "extends:" key is a gitlab specific that preserve the variable definition and just add the new ones. This will help us to reduce duplication. This has the effect of fixing some of variants definition we though we set while we did actually not. Most notably, the "rust" variant for 3.12 and 3.13 seems fully broken in the CI (possibly because some rust-cpython version issue?). This changeset only reveal such breackage and does not introduce them.

ci: use the macos wheel to run tests

wheel: build mac os wheel through the CI Let's start building wheel for mac os X too.

pycompat: drop some now useless workaround for makedirs This `exists_ok` flag was added in Python 3.2

run-tests: install wheel using --prefix instead of --user --user does not work if a venv is enabled when calling run-tests.py

ci: automatically compute the python tag we use to identify tag This make the determination more automatic and less error prone. In addition, this will make it possible to run on a runner without a pre-determined Python version, like what we do for the macos and windows workers.

pytype: add relative timestamp to the output if `ts` is available This should help to identify the module that are the slower to analyze.

ci: build (and use) wheel for all supported version We test wheel building for all supported version and use them where applicable The usage is more verbose than I wish because .gitlab-ci is not that great.

ci: use a pre-setup many-linux image to build wheel This produce wheel that are more universal and identical to the one we want to publish.

ci: use smaller VM to build wheel There is no need for large machine for this job, Python will be mostly singled threaded anyway.

setup: add a way to force the setup to translate (or fail) we add the `MERCURIAL_SETUP_FORCE_TRANSLATIONS` variable that is intended to make sure we don't stop building the translation silently.

ci: pre-adjust some identation We adjust a couple of job to consistently use double space identation. This will make the next changesets clearer.

test-install: glob instance of "python" in warning If run with something else (e.g. python3.12) the message says "python3.12" so we just glob that away.

test-install: use the global hg for the install step This prevent error in some cases.

tests: bump the wait timeouts in test-racy-mutations.t This was done to try to figure out why there's output differences in the previous two commits- low timeouts have been a cause of a lot of problems on Windows. That doesn't seem to be the case here, but I'm leaving it in anyway to avoid sporadic failures.

tests: add a "missing" tests for manifest content in test-racy-mutations.t Trying to figure out why the divergence in behavior on Windows. The first test shows everything is the same on all platforms; the second shows that the other #testcase also diverges. The difference might be that `00manifest.i` doesn't get updated (and copied over) after `pre-race` is created, but I've no idea why that would be either.

tests: conditionalize missing output in test-racy-mutations.t on Windows No idea why this is only missing on Windows. I verified that the line marked as missing directly above this for the changelog is, in fact, missing on both Windows and Linux. So there's probably work to do in this area on all platforms. It would be nice to figure out what is going on, but this appeases CI in the meantime.

tests: disable a section of `test-paths.t` that may hit a zeroconf bug This effectively re-disables the same test as cce9e7d2fb92, but unconditionally because it's not a pyoxidizer-specific problem. 74e16d8ca5f3 disabled the other related test for the same reason- this one was missed because it wasn't failing for the few runs when that was tested.

rust-update: make `update_from_null` respect `worker.numcpu` config option This was overlooked in the original series. This is important for tests (because we run many at once), and for the occasional end user that wants to keep their CPU usage in check. A future series should clean up this `worker` parameter tunelling business by rewriting the config in Rust, but doing so on stable would be a very bad idea.

rust-cpython: add a TODO about repo reuse This will need to be done soon-ish to prevent any surprises.

streamclone: disable the volatile file open handle optimization on Windows Leaving files open caused new failures like this, since a47f09da8bd1: diff --git a/tests/test-persistent-nodemap-stream-clone.t b/tests/test-persistent-nodemap-stream-clone.t --- a/tests/test-persistent-nodemap-stream-clone.t +++ b/tests/test-persistent-nodemap-stream-clone.t @@ -115,7 +115,12 @@ Do a mix of clone and commit at the same $ (hg clone -U --stream ssh://user@dummy/test-repo stream-clone-race-1 --debug 2>> clone-output | grep -E '00(changelog|manifest)' >> clone-output; touch $HG_TEST_STREAM_WALKED_FILE_3) & $ $RUNTESTDIR/testlib/wait-on-file 10 $HG_TEST_STREAM_WALKED_FILE_1 $ hg -R test-repo/ commit -m foo - created new head + transaction abort! + failed to recover 00changelog.n ([WinError 32] The process cannot access the file because it is being used by another process: b'$STR_REPR_TESTTMP\\test-repo/.hg/store/00changelog.n' -> b'$STR_REPR_TESTTMP\\test-repo/.hg/store/00changelog.n-f418dcd6') + rollback failed - please run hg recover + (failure reason: [WinError 32] The process cannot access the file because it is being used by another process: b'$STR_REPR_TESTTMP\\test-repo/.hg/store/00changelog.n' -> b'$STR_REPR_TESTTMP\\test-repo/.hg/store/00changelog.n-f418dcd6') + abort: The process cannot access the file because it is being used by another process: '$TESTTMP\test-repo\.hg\store\00changelog.n' + [255] $ touch $HG_TEST_STREAM_WALKED_FILE_2 $ $RUNTESTDIR/testlib/wait-on-file 10 $HG_TEST_STREAM_WALKED_FILE_3 $ cat clone-output Since the `VolatileManager` falls back to the old copy method when the open file threshold is exceeded, this just drops the threshold so that only 1 file is open. The actual value used (2) is unexpected, and explained inline. I'd like to have a config option for this so that we can test both ways (in theory, it could resort to copies on non-Windows systems too), but I don't see a `uimod.ui` handy. Alternately, I tried replacing the 3 `open()` calls in the `VolatileManager` with `util.posixfile()`, but that simply hung the test on Windows for some reason, I think on the same line that's indicated as failing above. (There was a `grep` command hanging around, as well as `hg -R test-repo serve --stdio`.)

tests: treat `select` as a built-in module on Windows This fixes: --- C:/Users/Matt/hg/tests/test-check-module-imports.t +++ C:/Users/Matt/hg/tests/test-check-module-imports.t.err @@ -43,3 +43,15 @@ > -X tests/test-verify-repo-operations.py \ > -X tests/test-extension.t \ > | sed 's-\\-/-g' | "$PYTHON" "$import_checker" - + hgext/zeroconf/Zeroconf.py:86: stdlib import "socket" follows local import: select\r (esc) + hgext/zeroconf/Zeroconf.py:87: stdlib import "struct" follows local import: select\r (esc) + hgext/zeroconf/Zeroconf.py:88: stdlib import "threading" follows local import: select\r (esc) + hgext/zeroconf/Zeroconf.py:89: stdlib import "time" follows local import: select\r (esc) + hgext/zeroconf/Zeroconf.py:90: stdlib import "traceback" follows local import: select\r (esc) + mercurial/posix.py:18: stdlib import "stat" follows local import: select\r (esc) + mercurial/posix.py:19: stdlib import "sys" follows local import: select\r (esc) + mercurial/posix.py:20: stdlib import "tempfile" follows local import: select\r (esc) + mercurial/posix.py:21: stdlib import "typing" follows local import: select\r (esc) + tests/tinyproxy.py:19: stdlib import "socket" follows local import: select\r (esc) + tests/tinyproxy.py:20: stdlib import "sys" follows local import: select\r (esc) + [1] ERROR: test-check-module-imports.t output changed

rust-vfs: add docstrings to all VFS methods on the trait

rust-vfs: support checkambig This was missing from the Rust code, which means worse caching. See https://wiki.mercurial-scm.org/ExactCacheValidationPlan. Explanations on what ambiguity means inline.

rust-vfs: add tests to `AtomicFile` This also makes it more usable from Rust by separating `from_file` and `new`.

rust-vfs: delete the temp file and not the target on drop Oops. This never affected anything since `close()` is correct and no code has dropped an `AtomicFile` without a close first yet. Next patch will add tests.

rust: don't star export from the `revlog` module This made a lot of the imports confusing because they didn't make sense at the top level (so, outside of `revlog`), and they hide the more common types when autocompleting.

rust: populate mmaps in a separate thread if possible Same rationale as b619ba39d10a.

rust-revlog: build an in-memory nodemap if a given revlog gets queried a lot This will help with non-persistent nodemap repos that would benefit from one, and mirrors what the C implementation does.

rust-revlog: generalize an error message This is used for more than the nodemap data.

rust-revlog: don't create an in-memory nodemap for filelogs from Python Explanations inline. Benchmarks from this change affect positively the only repo that showed this being a problem: ``` ### data-env-vars.name = mozilla-try-2024-03-26-ds2-pnm # benchmark.name = hg.command.cat # bin-env-vars.hg.flavor = rust # bin-env-vars.hg.py-re2-module = default # benchmark.variants.files = all-root # benchmark.variants.output = plain # benchmark.variants.rev = tip default: 62.848869 ~~~~~ before-this-patch: 58.113051 (-7.54%, -4.74) this-patch: 57.407533 (-8.66%, -5.44) ### data-env-vars.name = mozilla-try-2024-03-26-ds2-pnm # benchmark.name = hg.command.log # bin-env-vars.hg.flavor = rust # bin-env-vars.hg.py-re2-module = default # benchmark.variants.limit-rev = 10 # benchmark.variants.patch = yes # benchmark.variants.rev = none default: 3.173532 ~~~~~ before-this-patch: 3.543591 (+11.66%, +0.37) this-patch: 3.297235 (+3.90%, +0.12) ```

rust-revlog: move non-persistent-nodemap rev lookup to the index It only uses index features and does not need to be on the revlog. A later patch will make use of this function from a different context.

revlog: add glue to use a pure-Rust VFS This will save us a lot of calling back into Python, which is always horribly expensive. We are now faster in all benchmarked cases except for `log --patch` specifically on mozilla-try. Fixing this will happen in a later patch. ``` ### data-env-vars.name = mercurial-devel-2024-03-22-ds2-pnm # benchmark.name = hg.command.cat # bin-env-vars.hg.flavor = rust # bin-env-vars.hg.py-re2-module = default # benchmark.variants.files = all-root # benchmark.variants.output = plain # benchmark.variants.rev = tip e679697a6ca4: 1.760765 ~~~~~ 5559d7e63ec3: 1.555513 (-11.66%, -0.21) ### data-env-vars.name = mozilla-try-2024-03-26-ds2-pnm # benchmark.name = hg.command.cat # bin-env-vars.hg.flavor = rust # bin-env-vars.hg.py-re2-module = default # benchmark.variants.files = all-root # benchmark.variants.output = plain # benchmark.variants.rev = tip e679697a6ca4: 62.848869 ~~~~~ 5559d7e63ec3: 58.113051 (-7.54%, -4.74) ### data-env-vars.name = mozilla-try-2024-03-26-ds2-pnm # benchmark.name = hg.command.log # bin-env-vars.hg.flavor = rust # bin-env-vars.hg.py-re2-module = default # benchmark.variants.limit-rev = 10 # benchmark.variants.patch = yes # benchmark.variants.rev = none e679697a6ca4: 3.173532 ~~~~~ 5559d7e63ec3: 3.543591 (+11.66%, +0.37) ### data-env-vars.name = mozilla-try-2024-03-26-ds2-pnm # benchmark.name = hg.command.log # bin-env-vars.hg.flavor = rust # bin-env-vars.hg.py-re2-module = default # benchmark.variants.limit-rev = 1000 # benchmark.variants.patch = no # benchmark.variants.rev = none e679697a6ca4: 1.214698 ~~~~~ 5559d7e63ec3: 1.192478 (-1.83%, -0.02) ### data-env-vars.name = mozilla-unified-2024-03-22-ds2-pnm # benchmark.name = hg.command.cat # bin-env-vars.hg.flavor = rust # bin-env-vars.hg.py-re2-module = default # benchmark.variants.files = all-root # benchmark.variants.output = plain # benchmark.variants.rev = tip e679697a6ca4: 56.205474 ~~~~~ 5559d7e63ec3: 51.520074 (-8.34%, -4.69) ### data-env-vars.name = mozilla-unified-2024-03-22-ds2-pnm # benchmark.name = hg.command.log # bin-env-vars.hg.flavor = rust # bin-env-vars.hg.py-re2-module = default # benchmark.variants.limit-rev = 10 # benchmark.variants.patch = yes # benchmark.variants.rev = none e679697a6ca4: 2.105419 ~~~~~ 5559d7e63ec3: 2.051849 (-2.54%, -0.05) ### data-env-vars.name = mozilla-unified-2024-03-22-ds2-pnm # benchmark.name = hg.command.log # bin-env-vars.hg.flavor = rust # bin-env-vars.hg.py-re2-module = default # benchmark.variants.limit-rev = 1000 # benchmark.variants.patch = no # benchmark.variants.rev = none e679697a6ca4: 0.309960 ~~~~~ 5559d7e63ec3: 0.299035 (-3.52%, -0.01) ### data-env-vars.name = tryton-public-2024-03-22-ds2-pnm # benchmark.name = hg.command.cat # bin-env-vars.hg.flavor = rust # bin-env-vars.hg.py-re2-module = default # benchmark.variants.files = all-root # benchmark.variants.output = plain # benchmark.variants.rev = tip e679697a6ca4: 1.849832 ~~~~~ 5559d7e63ec3: 1.805076 (-2.42%, -0.04) ### data-env-vars.name = tryton-public-2024-03-22-ds2-pnm # benchmark.name = hg.command.log # bin-env-vars.hg.flavor = rust # bin-env-vars.hg.py-re2-module = default # benchmark.variants.limit-rev = 10 # benchmark.variants.patch = yes # benchmark.variants.rev = none e679697a6ca4: 0.289521 ~~~~~ 5559d7e63ec3: 0.279889 (-3.33%, -0.01) ### data-env-vars.name = tryton-public-2024-03-22-ds2-pnm # benchmark.name = hg.command.log # bin-env-vars.hg.flavor = rust # bin-env-vars.hg.py-re2-module = default # benchmark.variants.limit-rev = 1000 # benchmark.variants.patch = no # benchmark.variants.rev = none e679697a6ca4: 0.332270 ~~~~~ 5559d7e63ec3: 0.323324 (-2.69%, -0.01) ```

fncache: add attribute to check whether we're using dotencode This will make it easy to know if we can use the Rust implementation that doesn't support older forms of encoding.

fncachestore: add typing information This helps with autocomplete.

fncache: refactor load check into a property This makes the intent more obvious new callers less prone to error.

hg-core: add FnCacheVFS This will allow us to only call back into Python to add items to the fncache, which should save us a lot of FFI overhead. This is also of course a stepping stone for more pure Rust work.

hg-core: add a complete VFS This will be used from Python in a later change. More changes are needed in hg-core and rhg to properly clean up the APIs of the old VFS implementation but it can be done when the dust settles and we start adding more functionality to the pure Rust VFS.

hg-core: add fncache module For now it's only a super simple trait. It will be used for calling back into Python soon, and later will be fleshed out into a full fncache.

rust: populate mmap by default if available See 522b4d729e89edc76544fa549ed36de4aea0b7fb for more details. Background population to follow in a later patch.

rust-changelog: switch away from deprecated APIs for datetime use This was caught by clippy, nothing was changed aside from some light API changes.

revlog: add the glue to use the Rust `InnerRevlog` from Python The performance of this has been looked at for quite some time, and some workflows are actually quite a bit faster than with the Python + C code. However, we are still (up to 20%) slower in some crucial places like cloning certain repos, log, cat, which makes this an incomplete rewrite. This is mostly due to the high amount of overhead in Python <-> Rust FFI, especially around the VFS code. A future patch series will rewrite the VFS code in pure Rust, which should hopefully get us up to par with current perfomance, if not better in all important cases. This is a "save state" of sorts, as this is a ton of code, and I don't want to pile up even more things in a single review. Continuing to try to match the current performance will take an extremely long time, if it's not impossible, without the aforementioned VFS work.

changelog: also set the general delta config flag in the data config This duplication is dubious, but it's a decision to be made at a later date, this is the fix.

rust-index: use `IndexEntry::offset` to compute read segments This only matters for inline revlogs where the impact is debatable, but this is what the C index does.

rust-revlog: add a Rust-only `InnerRevlog` This mirrors the Python `InnerRevlog` and will be used in a future patch to replace said Python implementation. This allows us to start doing more things in pure Rust, in particular reading and writing operations. A lot of changes have to be introduced all at once, it wouldn't be very useful to separate this patch IMO since all of them are either interlocked or only useful with the rest.

rust-index: fix the computation of data start This was falling into place instead of being correct, we clean up the logic by differenciating the on-disk offset and the actual start of the data more cleanly.

rust-index: return an error on a bad index header This is more idiomatic and allows us to better handle the problem later.

rust-vfs: add a TODO to remember a decision taken about naming Explanations inline.