revlog: add glue to use a pure-Rust VFS This will save us a lot of calling back into Python, which is always horribly expensive. We are now faster in all benchmarked cases except for `log --patch` specifically on mozilla-try. Fixing this will happen in a later patch. ``` ### data-env-vars.name = mercurial-devel-2024-03-22-ds2-pnm # benchmark.name = hg.command.cat # bin-env-vars.hg.flavor = rust # bin-env-vars.hg.py-re2-module = default # benchmark.variants.files = all-root # benchmark.variants.output = plain # benchmark.variants.rev = tip e679697a6ca4: 1.760765 ~~~~~ 5559d7e63ec3: 1.555513 (-11.66%, -0.21) ### data-env-vars.name = mozilla-try-2024-03-26-ds2-pnm # benchmark.name = hg.command.cat # bin-env-vars.hg.flavor = rust # bin-env-vars.hg.py-re2-module = default # benchmark.variants.files = all-root # benchmark.variants.output = plain # benchmark.variants.rev = tip e679697a6ca4: 62.848869 ~~~~~ 5559d7e63ec3: 58.113051 (-7.54%, -4.74) ### data-env-vars.name = mozilla-try-2024-03-26-ds2-pnm # benchmark.name = hg.command.log # bin-env-vars.hg.flavor = rust # bin-env-vars.hg.py-re2-module = default # benchmark.variants.limit-rev = 10 # benchmark.variants.patch = yes # benchmark.variants.rev = none e679697a6ca4: 3.173532 ~~~~~ 5559d7e63ec3: 3.543591 (+11.66%, +0.37) ### data-env-vars.name = mozilla-try-2024-03-26-ds2-pnm # benchmark.name = hg.command.log # bin-env-vars.hg.flavor = rust # bin-env-vars.hg.py-re2-module = default # benchmark.variants.limit-rev = 1000 # benchmark.variants.patch = no # benchmark.variants.rev = none e679697a6ca4: 1.214698 ~~~~~ 5559d7e63ec3: 1.192478 (-1.83%, -0.02) ### data-env-vars.name = mozilla-unified-2024-03-22-ds2-pnm # benchmark.name = hg.command.cat # bin-env-vars.hg.flavor = rust # bin-env-vars.hg.py-re2-module = default # benchmark.variants.files = all-root # benchmark.variants.output = plain # benchmark.variants.rev = tip e679697a6ca4: 56.205474 ~~~~~ 5559d7e63ec3: 51.520074 (-8.34%, -4.69) ### data-env-vars.name = mozilla-unified-2024-03-22-ds2-pnm # benchmark.name = hg.command.log # bin-env-vars.hg.flavor = rust # bin-env-vars.hg.py-re2-module = default # benchmark.variants.limit-rev = 10 # benchmark.variants.patch = yes # benchmark.variants.rev = none e679697a6ca4: 2.105419 ~~~~~ 5559d7e63ec3: 2.051849 (-2.54%, -0.05) ### data-env-vars.name = mozilla-unified-2024-03-22-ds2-pnm # benchmark.name = hg.command.log # bin-env-vars.hg.flavor = rust # bin-env-vars.hg.py-re2-module = default # benchmark.variants.limit-rev = 1000 # benchmark.variants.patch = no # benchmark.variants.rev = none e679697a6ca4: 0.309960 ~~~~~ 5559d7e63ec3: 0.299035 (-3.52%, -0.01) ### data-env-vars.name = tryton-public-2024-03-22-ds2-pnm # benchmark.name = hg.command.cat # bin-env-vars.hg.flavor = rust # bin-env-vars.hg.py-re2-module = default # benchmark.variants.files = all-root # benchmark.variants.output = plain # benchmark.variants.rev = tip e679697a6ca4: 1.849832 ~~~~~ 5559d7e63ec3: 1.805076 (-2.42%, -0.04) ### data-env-vars.name = tryton-public-2024-03-22-ds2-pnm # benchmark.name = hg.command.log # bin-env-vars.hg.flavor = rust # bin-env-vars.hg.py-re2-module = default # benchmark.variants.limit-rev = 10 # benchmark.variants.patch = yes # benchmark.variants.rev = none e679697a6ca4: 0.289521 ~~~~~ 5559d7e63ec3: 0.279889 (-3.33%, -0.01) ### data-env-vars.name = tryton-public-2024-03-22-ds2-pnm # benchmark.name = hg.command.log # bin-env-vars.hg.flavor = rust # bin-env-vars.hg.py-re2-module = default # benchmark.variants.limit-rev = 1000 # benchmark.variants.patch = no # benchmark.variants.rev = none e679697a6ca4: 0.332270 ~~~~~ 5559d7e63ec3: 0.323324 (-2.69%, -0.01) ```

fncache: add attribute to check whether we're using dotencode This will make it easy to know if we can use the Rust implementation that doesn't support older forms of encoding.

fncachestore: add typing information This helps with autocomplete.

fncache: refactor load check into a property This makes the intent more obvious new callers less prone to error.

hg-core: add FnCacheVFS This will allow us to only call back into Python to add items to the fncache, which should save us a lot of FFI overhead. This is also of course a stepping stone for more pure Rust work.

hg-core: add a complete VFS This will be used from Python in a later change. More changes are needed in hg-core and rhg to properly clean up the APIs of the old VFS implementation but it can be done when the dust settles and we start adding more functionality to the pure Rust VFS.

hg-core: add fncache module For now it's only a super simple trait. It will be used for calling back into Python soon, and later will be fleshed out into a full fncache.

rust: populate mmap by default if available See 522b4d729e89edc76544fa549ed36de4aea0b7fb for more details. Background population to follow in a later patch.

rust-changelog: switch away from deprecated APIs for datetime use This was caught by clippy, nothing was changed aside from some light API changes.

revlog: add the glue to use the Rust `InnerRevlog` from Python The performance of this has been looked at for quite some time, and some workflows are actually quite a bit faster than with the Python + C code. However, we are still (up to 20%) slower in some crucial places like cloning certain repos, log, cat, which makes this an incomplete rewrite. This is mostly due to the high amount of overhead in Python <-> Rust FFI, especially around the VFS code. A future patch series will rewrite the VFS code in pure Rust, which should hopefully get us up to par with current perfomance, if not better in all important cases. This is a "save state" of sorts, as this is a ton of code, and I don't want to pile up even more things in a single review. Continuing to try to match the current performance will take an extremely long time, if it's not impossible, without the aforementioned VFS work.

changelog: also set the general delta config flag in the data config This duplication is dubious, but it's a decision to be made at a later date, this is the fix.

rust-index: use `IndexEntry::offset` to compute read segments This only matters for inline revlogs where the impact is debatable, but this is what the C index does.

rust-revlog: add a Rust-only `InnerRevlog` This mirrors the Python `InnerRevlog` and will be used in a future patch to replace said Python implementation. This allows us to start doing more things in pure Rust, in particular reading and writing operations. A lot of changes have to be introduced all at once, it wouldn't be very useful to separate this patch IMO since all of them are either interlocked or only useful with the rest.

rust-index: fix the computation of data start This was falling into place instead of being correct, we clean up the logic by differenciating the on-disk offset and the actual start of the data more cleanly.

rust-index: return an error on a bad index header This is more idiomatic and allows us to better handle the problem later.

rust-vfs: add a TODO to remember a decision taken about naming Explanations inline.

rust-revlog: introduce an `options` module This helps group all the relevant revlog options code and makes the `mod.rs` more readable.

rust-revlog: add file IO helpers This will be useful for the upcoming `InnerRevlog`.

rust-revlog: add compression helpers This will be used in the upcoming `InnerRevlog` when reading/writing data.

hgweb: skip logging ConnectionAbortedError Not stacktracing on `ConnectionResetError` was added in 6bbb12cba5a8 (though it was spelled differently for py2 support), but for some reason Windows occasionally triggers a `ConnectionAbortedError` here across various *.t files (notably `test-archive.t` and `test-lfs-serve-access.t`, but there are others). The payload that fails to send seems to be the html that describes the error to the client, so I suspect some code is seeing the error status code and closing the connection before the server gets to write this html. So don't log it, for test stability- nothing we can do anyway. FWIW, the CPython implementation of wsgihander specifically ignores these two errors, plus `BrokenPipeError`, with a comment that "we expect the client to close the connection abruptly from time to time"[1]. The `BrokenPipeError` is swallowed a level up in `do_write()`, and avoids writing the response following this stacktrace. I'm puzzled why a response is being written after these connection errors are detected- the CPython code referenced doesn't, and the connection is now broken at this point. Perhaps these errors should both be handled with the `BrokenPipeError` after the freeze. (The refactoring away from py2 compat may not be desireable in the freeze, but this is much easier to read, and obviously correct given the referenced CPython code.) I suspect this is what 6bceecb28806 was attempting to fix, but it wasn't specific about the sporadic errors it was seeing. [1] https://github.com/python/cpython/blob/b2eaa75b176e07730215d76d8dce4d63fb493391/Lib/wsgiref/handlers.py#L139

ci: add a runner for Windows 10 This is currently only manually invoked, and allows for failure because we only have a single runner that takes over 2h for a full run, and there are a handful of flakey tests, plus 3 known failing tests. The system being used here is running MSYS, Python, Visual Studio, etc, as installed by `install-windows-dependencies.ps1`. This script installs everything to a specific directory instead of using the defaults, so we adjust the MinGW shell path to compensate. Additionally, the script doesn't install the launcher `py.exe`. It is possible to adjust the script to install it, but it's an option to an existing python install (instead of a standalone installer), and I've had the whole python install fail and rollback when requested to install the launcher if it detects a newer one is already installed. In short, it is a point of failure for a feature we don't (yet?) need. Unlike other systems where the intepreter name includes the version, everything here is `python.exe`, so they can't all exist on `PATH` and let the script choose the desired one. (The `py.exe` launcher would accomplish, using the registry instead of `PATH`, but that wouldn't allow for venv installs.) Because of this, switch to the absolute path of the python interpreter to be used (in this case a venv created from the py39 install, which is old, but what both pyoxidizer and TortoiseHg currently use). The `RUNTEST_ARGS` hardcodes `-j8` because this system has 4 cores, and therefore runs 4 parallel tests by default. However on Windows, using more parallel tests than cores results in better performance for whatever reason. I don't have an optimal value yet (ideally the runner itself can make the adjustment on Windows), but this results in saving ~15m on a full run that otherwise takes ~2.5h. I'm also not concerned about how it would affect other Windows machines, because we don't have any at this point, and I have no idea when we can get more. As far as system setup goes, the CI is run by a dedicated user that lacks admin rights. The install script was run by an admin user, and then the standard user was configured to use it. If I set this up again, I'd probably give the dedicated user admin rights to run the install script, and reset to standard user rights when done. The python intepreter failed in weird ways when run by the standard user until it was manually reinstalled by the standard user: Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding Additionally, changing the environment through the Windows UI prompts to escalate to an admin user, and then setting the user level environment variables like `TEMP` and `PATH` (to try to avoid exceeding the 260 character path limit) didn't actually change the user's environment. (Likely it changed the admin user's environment, but I didn't confirm that.) I ended up having to use the registry editor for the standard user to make those changes.

tests: disable a section of `test-hgrc.t` that may hit a zeroconf bug This effectively re-disables the same test as cce9e7d2fb92, but unconditionally because it's not a pyoxidizer-specific problem (see below and 997c9b2069d1). I can run the test locally fine, with the same venv as CI is using, and have had multiple CI runs that don't hit this. But one failed with this: --- /private/tmp/mercurial-ci/tests/test-hgrc.t +++ /private/tmp/mercurial-ci/tests/test-hgrc.t.err @@ -305,5 +305,17 @@ [255] $ HGRCSKIPREPO=1 hg paths --config extensions.zeroconf= + Traceback (most recent call last): + File "/private/tmp/hgtests.7idf706t/install/lib/python/hgext/zeroconf/Zeroconf.py", line 966, in run + self.readers[sock].handle_read() + File "/private/tmp/hgtests.7idf706t/install/lib/python/hgext/zeroconf/Zeroconf.py", line 1020, in handle_read + msg = DNSIncoming(data) + File "/private/tmp/hgtests.7idf706t/install/lib/python/hgext/zeroconf/Zeroconf.py", line 537, in __init__ + self.readOthers() + File "/private/tmp/hgtests.7idf706t/install/lib/python/hgext/zeroconf/Zeroconf.py", line 650, in readOthers + self.readCharacterString(), + File "/private/tmp/hgtests.7idf706t/install/lib/python/hgext/zeroconf/Zeroconf.py", line 584, in readCharacterString + length = ord(self.data[self.offset]) + TypeError: ord() expected string of length 1, but int found foo = $TESTTMP/bar The zeroconf extension has bytes vs str problems that are obvious from inspection alone, and nobody has complained, so I'm not going to let this block getting CI for macOS up and running. Given that it's in the packet read code, I suspect that this 1) requires something on the network to speak mDNS, and 2) it is a timing issue if this is seen or not. (The bytes vs str issue itself is real, but only happen if a response is received quickly.)

tests: disable `test-git-interop.t` with a requirements directive Note that the failures in this test affect all platforms. I don't like this, but the test has been broken for awhile because of dirstate API changes, and nobody noticed because the required `pygit2` package isn't installed on the CI systems. I did install it on the mac CI system, which triggers this failure. Disabling it is no worse than not running it due to the missing package, but at least this way the CI systems can get the package installed, and the test can be enabled and fixed eventually, without needing to alter the CI systems. The feature here is kind of abused. I thought about adding one specifically to test for CI, but didn't feel like doing it at this point. Maybe if we need to disable things to get the Windows CI off the ground (but that likely requires testing for CI + platform).

tests: stabilize `test-extdiff.t` on macOS The recent change in the extdiff extension to take into account whether the GUI is accessible in d1b54c152673 started triggering this. I was able to run the test cleanly without this change at the console, but somewhere along the line, I read that the CI runner isn't able to access the GUI when not run as the root user. This is causing CI failures, so we conditionalize these tests out where `DISPLAY` is set to a non empty value to force `procutil.isgui()` to be True, when it in fact doesn't have GUI access.

branching: merge stable into default

ci: build a wheel and use it to run c tests First step into building and testing wheel automatically.

ci: split the jobs on more stage We start to have a lot of job, grouping them help to clarifying the pipeline. We don't actually create dependency between each stage, so everything still run concurrently. However we are about to introduce some wheel-building job that will be reused by some tests. So some dependencies are coming.

ci: unify the way `check-pytype` inherit the common setting All the other job use this syntax, so lets us it there too.

run-tests: add a --hg-wheel options to test a pre-built wheel This will be useful to test the wheel we intend to publish. A future changeset will integrate this in the CI.

run-tests: don't use shell call for subprocess This part of the test runner seems to comes for some ages ago.

branching: merge stable into default

Added signature for changeset dc97e8670dec

Added tag 6.9rc0 for changeset dc97e8670dec

doc: register the `config-doc` rst directive This was making the build fail because the directive was unknown.

relnotes: add 6.9rc0

branching: merge default into stable We will be releasing 6.9rc0 soon.

branching: merge stable into default

Added signature for changeset eae3ec345e5e

Added tag 6.8.2 for changeset eae3ec345e5e

relnotes: add 6.8.2

evolution: stop wrongly flagging unrelated part of a split as divergent Before this change, divergence introduced by successors of a split would "spill" to other unrelated successors of the split that were not ambiguous. This small changes fixes it. Thanks goes to Manuel Jacobs for the discussion leading to this realization that a new simple and correct definition could be found.

tests: skip doctests that use `time.tzset()` on Windows There's no way to conditionally skip the tests for a function (see the inline feature request). That leaves us with the choice to either put the whole `mercurial.utils.dateutil` module in the skip list of this script (but then this script prints out the module as unexpectedly not tested, and misses a bunch of tests that can be run), blacklist the test entirely (but that makes it harder to work with on Windows), or use this hack to look for the statement that is broken, and skip the test currently attached to one function. (It appears that an example in the list of examples corresponds to a single `>>>` block, and the `test` itself corresponds to a single function. So prescan the examples, and skip all of them when the statement is found in any, since the setup of setting the timezone has an effect on subsequent examples.)

tests: hopefully fix `test-doctest.py` on Windows and more 1. Shell syntax understood by `shell=True` depends on the platform. Instead, pass `shell=False` and call `sh` explicitly to interpret the command correctly. 2. Stop setting `HGRCPATH=/dev/null`, so the setting `experimental.evolution=createmarkers` is set correctly. The reason I set HGRCPATH to /dev/null previously is because of misunderstanding where I thought the Python script had no HGRC to edit. As it turns out, there is in fact a valid temporary HGRC pointed to by HGRCPATH in this context so we don't seem to need this. /shrug

hghave: make the description for "clang-format" ascii test-fix-clang-format.t suddenly started failing on Windows by wiping the whole file content, and replacing with an error: $TESTTMP.sh: $TESTTMP.sh: cannot execute binary file Odd, because I don't have `clang-format` installed, so the test should be skipped. The problem started with 73cf8b56c2f5, and I noticed that running `hghave` manually resulted in a `SyntaxError` (so I can't see how this isn't broken everywhere, but maybe it's because I'm using py3.9 on Windows): $ py hghave --list Traceback (most recent call last): File "hghave", line 8, in <module> import hghave File "c:\Users\Matt\hg\tests\hghave.py", line 627 SyntaxError: Non-ASCII character '\xe2' in file c:\Users\Matt\hg\tests\hghave.py on line 627, but no encoding declared; see http://python.org/dev/peps/pep-0263/ for details

branching: merge stable into default Hopefully this will bring the last changes necessary to make the 3.13 tests green (on Linux).

tests: enable pytype checking on `mercurial/wireprotov1peer.py`

typing: suppress bogus pytype errors in `mercurial/wireprotov1peer.py` Fixes: File "/mnt/c/Users/Matt/hg/mercurial/wireprotov1peer.py", line 100, in result: No attribute '_peerexecutor' on unsentfuture [attribute-error] File "/mnt/c/Users/Matt/hg/mercurial/wireprotov1peer.py", line 278, in close: No attribute 'shutdown' on None [attribute-error] Called from (traceback): line 123, in __exit__ File "/mnt/c/Users/Matt/hg/mercurial/wireprotov1peer.py", line 278, in close: No attribute 'shutdown' on None [attribute-error] In Optional[concurrent.futures.thread.ThreadPoolExecutor] We drop the zope decorator on `peerexecutor`, because otherwise it triggers this error: File "/tmp/mercurial-ci/mercurial/wireprotov1peer.py", line 111, in <module>: Invalid type annotation [invalid-annotation] Must be constant Not sure why, because the decorated classes usually get typed as `Any`, which would also be fine here.

tests: enable pytype checking on `mercurial/wireprotoframing.py`

typing: suppress bogus pytype errors in `mercurial/wireprotoframing.py` This fixes: File "/mnt/c/Users/Matt/hg/mercurial/wireprotoframing.py", line 480, in createalternatelocationresponseframe: unsupported operand type(s) for item assignment: bytes [unsupported-operands] No attribute '__setitem__' on bytes File "/mnt/c/Users/Matt/hg/mercurial/wireprotoframing.py", line 510, in createcommanderrorresponse: unsupported operand type(s) for item assignment: bytes [unsupported-operands] No attribute '__setitem__' on bytes File "/mnt/c/Users/Matt/hg/mercurial/wireprotoframing.py", line 776, in __init__: Can't find module 'mercurial.zstd'. [import-error] File "/mnt/c/Users/Matt/hg/mercurial/wireprotoframing.py", line 804, in __init__: Can't find module 'mercurial.zstd'. [import-error] File "/mnt/c/Users/Matt/hg/mercurial/wireprotoframing.py", line 834, in populatestreamencoders: Can't find module 'mercurial.zstd'. [import-error] Using `TypedDict` is tempting here to fix the first two, but requires str keys. The code doing the importing doesn't call the code at the other three locations if the `mercurial.zstd` module fails to import in a place that handles the ImportError.

wireprototypes: make `baseprotocolhandler` methods abstract The documentation says it's an abstract base class, so let's enforce it. The `typing.Protocol` class is already an ABC, but it only prevents instantiation if there are abstract attrs that are missing. For example, from `hg debugshell`: >>> from mercurial import wireprototypes >>> x = wireprototypes.baseprotocolhandler() Traceback (most recent call last): File "<console>", line 1, in <module> TypeError: Can't instantiate abstract class baseprotocolhandler with abstract method name >>> class fake(wireprototypes.baseprotocolhandler): ... pass ... >>> x = fake() Traceback (most recent call last): File "<console>", line 1, in <module> TypeError: Can't instantiate abstract class fake with abstract method name That's great, but it doesn't protect against calling non-abstract methods at runtime, rather it depends on the protocol type hint being added to method signatures or class attrs, and then running a type checker to notice when an instance is assigned that doesn't conform to the protocol. We don't widely use type hints yet, and do have a lot of class hierarchy in the repository area, which could lead to surprises like this: >>> class fake(wireprototypes.baseprotocolhandler): ... @property ... def name(self) -> bytes: ... return b'name' ... >>> z = fake() >>> z.client() >>> print(z.client()) None Oops. That was supposed to return `bytes`. So not only is a bad/unexpected value returned, but it's one that violates the type hints (since the base client() method will be annotated to return bytes). With this change, we get: >>> from mercurial import wireprototypes >>> class fake(wireprototypes.baseprotocolhandler): ... @property ... def name(self) -> bytes: ... return b'name' ... >>> x = fake() Traceback (most recent call last): File "<console>", line 1, in <module> TypeError: Can't instantiate abstract class fake with abstract methods addcapabilities, checkperm, client, getargs, getpayload, getprotocaps, mayberedirectstdio So this looks like a reasonable safety harness to me, and lets us catch problems by running the standard tests while the type hints are being added, and pytype is improved. We should probably do this for all Protocol class methods that don't supply a method implementation.

wireprototypes: convert `baseprotocolhandler.name` to an abstract property PyCharm was flagging the subclasses where this was declared as a `@property` with Type of 'name' is incompatible with 'baseprotocolhandler' But pytype didn't complain. This seems more correct, however. Since `Protocol` is already an `abc.ABCMeta` class, we don't need to mess with the class hierarchy.

wireprotoserver: subclass the new `baseprotocolhandler` Protocol class

wireprototypes: convert `baseprotocolhandler` to a Protocol class The methodology for doing this is now known, and this is limited to two implementing classes, so just make the changes.

test: stabilize `test-audit-path.t` in rust (hopefully) We have been seeing flakiness on the file reported for a bit.

pycompat: drop test involving assigning "foo" to `sys.hexversion` Starting with python 3.13, `sys.hexversion` refuse to be assigned non-hex value like "foo". I don't think I can blame it. It is time to drop that part of the tests.

pycompat: filter more of the traceback in `test-flagproccessor.t` The traceback changes again with 3.13. So we filter it to only keeps the bits we care about. This is actually only reusing the approach from a few line below.

pycompat: ignore the fork + thread warning for now No known issues has been reported and this is breaking the CI quite hard. So for now we have to delay the issue.

wireprototypes: fix exception handling code with a bad pytype suppression This goes back to f5fcf7123a92, and I suspect it was a misread of the error message is describes- the LHS of `.sorted()` is always bytes, and bytes didn't have this method in py2 either. The invalid names were already handled like this a few lines above. PyCharm flagged this, and it stood out after converting the zope interfaces to Protocol classes (which hasn't been published yet).

tests: conditionalize undesired output on Windows for rbc the mmap cases I don't want to lose sight of this issue, and it's useful to be able to turn on mmap support to hack on the underlying problem. As noted in the previous commit, I think the current usage of `mmap` and `memoryview` needs to be reworked for correctness on posix anyway.

rev-branch-cache: disable mmapping by default on Windows See the inline comment for why. The commands work, other than leaving extra files laying around. Perhaps there's some way to get this to work like on posix with some `CreateFile` magic (though it already uses `FILE_SHARE_DELETE`, so I'm not sure offhand what else we can do). However big picture- it seems wrong that the old file is left mmapped, a new one moved into place, and the mapping left over the old file instead of retargeted to the new file. That's got to be a bug on posix too, in a long running process like chg, right? If the memory is read again for some reason, it will be stale data.