branching: merge stable into default

Added signature for changeset eae3ec345e5e

Added tag 6.8.2 for changeset eae3ec345e5e

relnotes: add 6.8.2

evolution: stop wrongly flagging unrelated part of a split as divergent Before this change, divergence introduced by successors of a split would "spill" to other unrelated successors of the split that were not ambiguous. This small changes fixes it. Thanks goes to Manuel Jacobs for the discussion leading to this realization that a new simple and correct definition could be found.

tests: skip doctests that use `time.tzset()` on Windows There's no way to conditionally skip the tests for a function (see the inline feature request). That leaves us with the choice to either put the whole `mercurial.utils.dateutil` module in the skip list of this script (but then this script prints out the module as unexpectedly not tested, and misses a bunch of tests that can be run), blacklist the test entirely (but that makes it harder to work with on Windows), or use this hack to look for the statement that is broken, and skip the test currently attached to one function. (It appears that an example in the list of examples corresponds to a single `>>>` block, and the `test` itself corresponds to a single function. So prescan the examples, and skip all of them when the statement is found in any, since the setup of setting the timezone has an effect on subsequent examples.)

tests: hopefully fix `test-doctest.py` on Windows and more 1. Shell syntax understood by `shell=True` depends on the platform. Instead, pass `shell=False` and call `sh` explicitly to interpret the command correctly. 2. Stop setting `HGRCPATH=/dev/null`, so the setting `experimental.evolution=createmarkers` is set correctly. The reason I set HGRCPATH to /dev/null previously is because of misunderstanding where I thought the Python script had no HGRC to edit. As it turns out, there is in fact a valid temporary HGRC pointed to by HGRCPATH in this context so we don't seem to need this. /shrug

hghave: make the description for "clang-format" ascii test-fix-clang-format.t suddenly started failing on Windows by wiping the whole file content, and replacing with an error: $TESTTMP.sh: $TESTTMP.sh: cannot execute binary file Odd, because I don't have `clang-format` installed, so the test should be skipped. The problem started with 73cf8b56c2f5, and I noticed that running `hghave` manually resulted in a `SyntaxError` (so I can't see how this isn't broken everywhere, but maybe it's because I'm using py3.9 on Windows): $ py hghave --list Traceback (most recent call last): File "hghave", line 8, in <module> import hghave File "c:\Users\Matt\hg\tests\hghave.py", line 627 SyntaxError: Non-ASCII character '\xe2' in file c:\Users\Matt\hg\tests\hghave.py on line 627, but no encoding declared; see http://python.org/dev/peps/pep-0263/ for details

branching: merge stable into default Hopefully this will bring the last changes necessary to make the 3.13 tests green (on Linux).

tests: enable pytype checking on `mercurial/wireprotov1peer.py`

typing: suppress bogus pytype errors in `mercurial/wireprotov1peer.py` Fixes: File "/mnt/c/Users/Matt/hg/mercurial/wireprotov1peer.py", line 100, in result: No attribute '_peerexecutor' on unsentfuture [attribute-error] File "/mnt/c/Users/Matt/hg/mercurial/wireprotov1peer.py", line 278, in close: No attribute 'shutdown' on None [attribute-error] Called from (traceback): line 123, in __exit__ File "/mnt/c/Users/Matt/hg/mercurial/wireprotov1peer.py", line 278, in close: No attribute 'shutdown' on None [attribute-error] In Optional[concurrent.futures.thread.ThreadPoolExecutor] We drop the zope decorator on `peerexecutor`, because otherwise it triggers this error: File "/tmp/mercurial-ci/mercurial/wireprotov1peer.py", line 111, in <module>: Invalid type annotation [invalid-annotation] Must be constant Not sure why, because the decorated classes usually get typed as `Any`, which would also be fine here.

tests: enable pytype checking on `mercurial/wireprotoframing.py`

typing: suppress bogus pytype errors in `mercurial/wireprotoframing.py` This fixes: File "/mnt/c/Users/Matt/hg/mercurial/wireprotoframing.py", line 480, in createalternatelocationresponseframe: unsupported operand type(s) for item assignment: bytes [unsupported-operands] No attribute '__setitem__' on bytes File "/mnt/c/Users/Matt/hg/mercurial/wireprotoframing.py", line 510, in createcommanderrorresponse: unsupported operand type(s) for item assignment: bytes [unsupported-operands] No attribute '__setitem__' on bytes File "/mnt/c/Users/Matt/hg/mercurial/wireprotoframing.py", line 776, in __init__: Can't find module 'mercurial.zstd'. [import-error] File "/mnt/c/Users/Matt/hg/mercurial/wireprotoframing.py", line 804, in __init__: Can't find module 'mercurial.zstd'. [import-error] File "/mnt/c/Users/Matt/hg/mercurial/wireprotoframing.py", line 834, in populatestreamencoders: Can't find module 'mercurial.zstd'. [import-error] Using `TypedDict` is tempting here to fix the first two, but requires str keys. The code doing the importing doesn't call the code at the other three locations if the `mercurial.zstd` module fails to import in a place that handles the ImportError.

wireprototypes: make `baseprotocolhandler` methods abstract The documentation says it's an abstract base class, so let's enforce it. The `typing.Protocol` class is already an ABC, but it only prevents instantiation if there are abstract attrs that are missing. For example, from `hg debugshell`: >>> from mercurial import wireprototypes >>> x = wireprototypes.baseprotocolhandler() Traceback (most recent call last): File "<console>", line 1, in <module> TypeError: Can't instantiate abstract class baseprotocolhandler with abstract method name >>> class fake(wireprototypes.baseprotocolhandler): ... pass ... >>> x = fake() Traceback (most recent call last): File "<console>", line 1, in <module> TypeError: Can't instantiate abstract class fake with abstract method name That's great, but it doesn't protect against calling non-abstract methods at runtime, rather it depends on the protocol type hint being added to method signatures or class attrs, and then running a type checker to notice when an instance is assigned that doesn't conform to the protocol. We don't widely use type hints yet, and do have a lot of class hierarchy in the repository area, which could lead to surprises like this: >>> class fake(wireprototypes.baseprotocolhandler): ... @property ... def name(self) -> bytes: ... return b'name' ... >>> z = fake() >>> z.client() >>> print(z.client()) None Oops. That was supposed to return `bytes`. So not only is a bad/unexpected value returned, but it's one that violates the type hints (since the base client() method will be annotated to return bytes). With this change, we get: >>> from mercurial import wireprototypes >>> class fake(wireprototypes.baseprotocolhandler): ... @property ... def name(self) -> bytes: ... return b'name' ... >>> x = fake() Traceback (most recent call last): File "<console>", line 1, in <module> TypeError: Can't instantiate abstract class fake with abstract methods addcapabilities, checkperm, client, getargs, getpayload, getprotocaps, mayberedirectstdio So this looks like a reasonable safety harness to me, and lets us catch problems by running the standard tests while the type hints are being added, and pytype is improved. We should probably do this for all Protocol class methods that don't supply a method implementation.

wireprototypes: convert `baseprotocolhandler.name` to an abstract property PyCharm was flagging the subclasses where this was declared as a `@property` with Type of 'name' is incompatible with 'baseprotocolhandler' But pytype didn't complain. This seems more correct, however. Since `Protocol` is already an `abc.ABCMeta` class, we don't need to mess with the class hierarchy.

wireprotoserver: subclass the new `baseprotocolhandler` Protocol class

wireprototypes: convert `baseprotocolhandler` to a Protocol class The methodology for doing this is now known, and this is limited to two implementing classes, so just make the changes.

test: stabilize `test-audit-path.t` in rust (hopefully) We have been seeing flakiness on the file reported for a bit.

pycompat: drop test involving assigning "foo" to `sys.hexversion` Starting with python 3.13, `sys.hexversion` refuse to be assigned non-hex value like "foo". I don't think I can blame it. It is time to drop that part of the tests.

pycompat: filter more of the traceback in `test-flagproccessor.t` The traceback changes again with 3.13. So we filter it to only keeps the bits we care about. This is actually only reusing the approach from a few line below.

pycompat: ignore the fork + thread warning for now No known issues has been reported and this is breaking the CI quite hard. So for now we have to delay the issue.

wireprototypes: fix exception handling code with a bad pytype suppression This goes back to f5fcf7123a92, and I suspect it was a misread of the error message is describes- the LHS of `.sorted()` is always bytes, and bytes didn't have this method in py2 either. The invalid names were already handled like this a few lines above. PyCharm flagged this, and it stood out after converting the zope interfaces to Protocol classes (which hasn't been published yet).

tests: conditionalize undesired output on Windows for rbc the mmap cases I don't want to lose sight of this issue, and it's useful to be able to turn on mmap support to hack on the underlying problem. As noted in the previous commit, I think the current usage of `mmap` and `memoryview` needs to be reworked for correctness on posix anyway.

rev-branch-cache: disable mmapping by default on Windows See the inline comment for why. The commands work, other than leaving extra files laying around. Perhaps there's some way to get this to work like on posix with some `CreateFile` magic (though it already uses `FILE_SHARE_DELETE`, so I'm not sure offhand what else we can do). However big picture- it seems wrong that the old file is left mmapped, a new one moved into place, and the mapping left over the old file instead of retargeted to the new file. That's got to be a bug on posix too, in a long running process like chg, right? If the memory is read again for some reason, it will be stale data.

tests: actually test the non-mmap case in `test-branches.t` It looks like 40943970b7ae renamed the config, but also flipped it to 'on' by default, and the test file didn't keep up. I noticed because all 4 test cases failed on Windows due to a mmap problem, and there only should have been 2.

tests: dump the http server log after a clone in `test-static-http.t` The 404 message lines don't match `$LOGDATE$`, because that tests for a pattern from the first "-" through a "(GET|PUT|POST)", so glob the timestamp away manually.

tests: force `dumbhttp.py` to write its log file with '\n' on Windows This wasn't causing obvious test failures, but it's the same fix as dbd2d56224d1 for `dummysmtpd.py`, and there's no sense in leaving this problem lying around. (And upon further review, it might have been causing some non-obviously related failures- see the next commit.)

branching: merge stable into default

rust: address 'error: unnecessarily eager cloning of iterator items' Build failed with a reference to https://rust-lang.github.io/rust-clippy/master/index.html#iter_overeager_cloned which seems reasonable. There doesn't seem to be any reason to not follow the advice.

utils: fix resourceutil use of deprecated importlib.resources Some importlib functionality was deprecated in 3.11 . The documentation on https://docs.python.org/3.12/library/importlib.resources.html recommends using the new .files() API that was introduced in 3.9.

utils: avoid using internal _imp.is_frozen() imp has been deprecated for a long time, and were removed in Python 3.12 . As a workaround, we started using the internal _imp. That is ugly and risky. It seems less risky to get the functionality in some other way. Here, we just inspect if 'origin' of the '__main__' module is set and 'frozen'. That seems to work and do the same, and might be better than using the internal _imp directly. This way of inspecting module attributes seems to work in some test cases, but it is a risky change. This level of importlib doesn't have much documentation, a complicated implementation, and we are dealing with some odd use cases.

cext: use sys.executable instead of deprecated Py_GetProgramFullPath Fix warning with Python 3.13: mercurial/cext/parsers.c: In function 'check_python_version': mercurial/cext/parsers.c:1243:30: warning: 'Py_GetProgramFullPath' is deprecated [-Wdeprecated-declarations] 1243 | Py_GetProgramFullPath()); | ^~~~~~~~~~~~~~~~~~~~~ In file included from /usr/include/python3.13/Python.h:119, from mercurial/cext/parsers.c:11: /usr/include/python3.13/pylifecycle.h:43:43: note: declared here 43 | Py_DEPRECATED(3.13) PyAPI_FUNC(wchar_t *) Py_GetProgramFullPath(void); | ^~~~~~~~~~~~~~~~~~~~~ At this point in time, the PyConfig struct memory has been released and the PyConfig API can't be used. https://docs.python.org/3.13/c-api/init.html#c.Py_GetProgramFullPath recommands using sys.executable instead. Let's assume that will work in all versions. It would perhaps be better to use PySys_GetObject, but I prefer to stay consistent with how the same function is retrieving sys.hexversion.

subrepoutil: pass re.sub 'count' argument by name Python 3.13 started warning: DeprecationWarning: 'count' is passed as positional argument

tests: pass re.MULTILINE to re.sub as 'flags' - not in 'count' position This bug was caught by the new Python 3.13 warning: DeprecationWarning: 'count' is passed as positional argument

tests: use packaging from setuptools instead of deprecated distutils When invoking StrictVersion in 3.12 we got: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. distutils is dead in the standard library, and we have to move towards using `setuptools` as general extern dependency. Instead of also requiring the extern `packaging`, we will just use the packaging that is vendored in setuptools.

tests: drop test-demandimport.py distutils test that failed with warnings The test would fail because warnings: /usr/lib/python3.11/site-packages/_distutils_hack/__init__.py:18: UserWarning: Distutils was imported before Setuptools, but importing Setuptools also replaces the `distutils` module in `sys.modules`. This may lead to undesirable behaviors or errors. To avoid these issues, avoid using distutils directly, ensure that setuptools is installed in the traditional way (e.g. not an editable install), and/or make sure that setuptools is always imported before distutils. warnings.warn( /usr/lib/python3.11/site-packages/_distutils_hack/__init__.py:33: UserWarning: Setuptools is replacing distutils. warnings.warn("Setuptools is replacing distutils.") The test for distutils.msvc9compiler comes from 2205d00b6d2b. But since then, distutils is going away, and this test must change somehow. It is unclear exactly how setuptools depended on msvc9compiler, but setuptools also moved forward, and this exact test no longer seems relevant. It thus seems like a fair solution to remove the test while keeping the demandimport blacklist of distutils.msvc9compiler.

utils: test coverage of makedate Explore the scenario from ae04af1ce78d to avoid future regressions. This was intended to give some coverage of the change in faccec1edc2c.

filecache: use bytes wherever possible in the tests This is closer than the actual usage, so I figured in would not hurt.

cachestat: avoid creating cachestat for http path The statichttprepo repo attemp to create cachestat for content we access through http. We modify the couple of place create cachestat object to detect this situation and avoids it. This is not marvelous, but there is few of them and the freeze is looming. This helps on Windows where calling cachestat on http path might create issues.

filecache: use binary path in the test This was overlooked when converting string. This is needed as we are about to introduce bytes specific code in the filecache code path.

branching: merge stable into default

py-3-13: stabilize the docstring output across all supported Python versions Python 3.13 now trims indents from docstrings at compilation time (to save space in .pyc), so all of our helptext is affected. The indentation has never served a user-facing purpose and was more here because nobody cared enough to remove it: we gain some screen space this way. Rather than undo the transformation (which isn't really possible since the transform also deletes leading/trailing whitespace), we align the behavior of older Python versions with that of 3.13. Unfortunately, this means breaking some of the translations. I've only touched the ones that need to work for some tooling tests to pass, but I do not have the time to fix the rest of them across all languages, since they cannot be done in an automated way. i18n updates have been basically abandonned for a good while now, hopefully someone cares enough to bring them back.

docstring: backed out changeset 51057ab0dffa In retrospect this is too much of a behavior change for stable. So I grafted the same change as 31076a2301f1 on default, and I am backing out its version on stable.

tests: stabilize `test-clonebundles-autogen.t` on Windows The problem was that the commands are spun up with `shell=True`, which uses `cmd.exe`, which doesn't understand `$foo` style variables. The HGCB variable expansion has to be delayed, because it's figured out right before launching the command. We could probably add a conditional for Windows, and rewrite the config to use `%foo%` style variables, but it's more maintainable to just wrap the command in a bash shell invocation. The forward style slashes in the path are needed to avoid accruing double backslashes (when switching between shells- the url template seems fine). Also need to strong quote the command so that the double quotes don't get stripped off of `$HGCB_BUNDLE_PATH`, which results in: sh: 1: Syntax error: Unterminated quoted string abort: command returned status 2: sh -c "cp $HGCB_BUNDLE_PATH $TESTTMP/final-upload/"

clonebundles: stop shell quoting `HGCB_BUNDLE_BASENAME` environment variable This causes problems in `test-clonebundles-autogen.t` on Windows, because the quoted path ends up being passed to the `cp` command, which fails, because quote characters are not a legal part of a file name. I don't see any quoting in environment variables on either MSYS or WSL, even with weird ones that appear to have escape sequences like `PS1=\[\033]0;$MSYSTEM:\w\007` (in MSYS). The quoting was added back in 5ae30ff79c76, and as shown here, was causing problems even on posix when a quote was slipped into the path. (The other obvious problem is that the command is spun up shell style, which invokes `cmd.exe`, which doesn't know about `$foo` style variables. That will be addressed next, but that change didn't work without this too.)

tests: add coverage to for `HGCB_BUNDLE_BASENAME` with special characters Per request on IRC, to show the behavior of dropping the quoting of `HGCB_BUNDLE_BASENAME` in the next commit. This current failure is basically the same error and output that currently happens on Windows with any path (even without the embedded quote). The only difference is Windows doesn't print the `cp: cannot stat ...` line.

tests: stabilize `test-eol-update.t` on Windows Perhaps it's better if this doesn't happen, but there are a bunch of tests that spew this, and we already have a conditional match for this in the block prior to the comment right above this section. So accept it as a possibility, and reduce the noise in the Windows tests.

tests: force `dummysmtpd.py` to write its log file with '\n' on Windows The log files were being `cat'd` in `test-patchbomb-tls.t`, and causing gratuitous failures. Since `sys.stdout` is being written to with `str` instead of `bytes`, use a `io.TextIOWrapper` to change the EOL, like 2924676d4728.

tests: raise the default value for the various `devel.sync.*-timeout` configs These are used in `mercurial.testing.wait_file()` to stall for a file to appear in the filesystem, and raise an error if the file doesn't show up before the timeout expires. The default of 2s was way too low on Windows, especially when running tests in parallel, and resulted in various timeouts in `test-dirstate-read-race.t`, `test-dirstate-status-write-race.t`, and `test-clone-stream-revlog-split.t`. The various `wait-on-file` invocations in the tests are inconsistent, and wait anywhere from 5s - 20s. I'm using 20s here because if everything is working, the timeout won't matter. Also with the default timeout being raised on Windows in f4c038081561, both `HGTEST_TIMEOUT_DEFAULT` and `HGTEST_TIMEOUT` are 1440 in the default case where the timeout is not specified on the command line of the test runner, so the timing factor that is multipled with the value is 1, resulting in no changes. (But if someone specified a lower value on the command line, that would *lower* the timeout period used.)

tests: remove deprecated test-check-py3-compat.t If our current source files were not compatible with Python 3, we would know by now. This check has not been relevant for a couple of years now and we can safely remove it.

zope-interface: add compatibility with 3.13 compiler attributes We could follow-up with an actual vendoring update from the newest version of zope-interface in the new cycle since we're dropping 3.7 and down. However we are also in the process of replacing zope-interface with Protocol, so hopefully we can simply drop the zope-interface vendoring.

py-3-13: fix traceback matching for the new Python version

py-3-13: stabilize the docstring output across all supported Python versions Python 3.13 now trims indents from docstrings at compilation time (to save space in .pyc), so all of our helptext is affected. The indentation has never served a user-facing purpose and was more here because nobody cared enough to remove it: we gain some screen space this way. Rather than undo the transformation (which isn't really possible since the transform also deletes leading/trailing whitespace), we align the behavior of older Python versions with that of 3.13. Unfortunately, this means breaking some of the translations. I've only touched the ones that need to work for some tooling tests to pass, but I do not have the time to fix the rest of them across all languages, since they cannot be done in an automated way. i18n updates have been basically abandonned for a good while now, hopefully someone cares enough to bring them back.

test-lfs-serve-access: vastly simplify the error output check The traceback differences between all supported Python versions is about to become insane to handle, so let's check what we're actually looking to check.

subrepo: propagate non-default path on outgoing There was already a fix made in 5dbff89cf107 for pull and push commands. I did the same for the outgoing command. The problem I identified is that when the parent repository has multiple paths, the outgoing command was not respecting the parent path used and was always using the default path for subrepositories.

outgoing: pre-indent some code This will make the next changeset clearer.

outgoing: move sorting in the display function This make the core code simpler.

outgoing: extract changeset display in its own function This will clarify future patches.

subrepo: move code around

merge: add a config to allow conflict-free merge of changes on adjacent lines This change adds a config to make it no longer a conflict to merge changes made on adjacent lines. The reason these changes are considered a conflict is that there's no region of text at the relevant position (sync region) that's kept unchanged by both sides of the merge. The problem can be solved by making the sync regions being a bit more powerful: we can keep a 0-length sync region if we find that a block unchanged by one side is ajacent to a block unchanged by the other side. Since these 0-length sync regions are emitted using the ~same algorithm as the normal non-empty sync regions, this change involves no arbitrary decisions and I expect it to work pretty well. 0-length sync regions do create an ambiguity in a special case where two pairs of adjacent regions "meet" at the same point. This corresponds to an insertion made at the same place by the two sides of the merge, and this still results in a conflict.

localrepo: drop the CamelCase name for `localrepo.ilocalrepositorymain` See 61557734c0ae for the reasoning. This one is slightly different, however, because the `localrepository` class already subclasses the interface class in the type checking phase.

localrepo: drop the CamelCase name for `localrepo.revlognarrowfilestorage` See 61557734c0ae for the reasoning.

localrepo: drop the CamelCase name for `localrepo.revlogfilestorage` See 61557734c0ae for the reasoning.

localrepo: drop the CamelCase name for `localrepo.locallegacypeer` See 61557734c0ae for the reasoning.

localrepo: drop the CamelCase name for `localrepo.localpeer` See 61557734c0ae for the reasoning.

localrepo: drop the CamelCase name for `localrepo.localcommandexecutor` See 61557734c0ae for the reasoning.

manifest: drop the CamelCase name for `manifest.treemanifestctx` See 61557734c0ae for the reasoning.

manifest: drop the CamelCase name for `manifest.memtreemanifestctx` See 61557734c0ae for the reasoning.

manifest: drop the CamelCase name for `manifest.manifestctx` See 61557734c0ae for the reasoning.

manifest: drop the CamelCase name for `manifest.memmanifestctx` See 61557734c0ae for the reasoning.

manifest: drop the CamelCase name for `manifest.manifestlog` See 61557734c0ae for the reasoning.

manifest: drop the CamelCase name for `manifest.manifestrevlog` See 61557734c0ae for the reasoning.

manifest: drop the CamelCase name for `manifest.treemanifest` See 61557734c0ae for the reasoning.

manifest: drop the CamelCase name for `manifest.manifestdict` See 61557734c0ae for the reasoning.

filelog: drop the CamelCase name for `filelog.filelog` See 61557734c0ae for the reasoning.

revlog: drop the CamelCase name for `revlog.revlogproblem` See 61557734c0ae for the reasoning.

revlog: drop the CamelCase name for `revlog.revlogrevisiondelta` These CamelCase names popped up this cycle because pytype was getting confused by the zope decorator, and so the decoration was decoupled from the class declaration. We're in the process of switching all of the zope interfaces to `typing.Protocol` classes, but we're up against the code freeze. It would be nice to use CamelCase for protocol classes (or classes in general), but let's reset to a consistent state and buy some more time to think about this. Since we're moving to Protocol classes and I disabled the interface tests back in ef7d85089952, I'm not bothering to re-add the decorator, and re-confuse pytype. But do place the interface next to the class, so that these places can be found with the `grep` when the time comes to subclass.

tests: use pyflakes as a tool, not a python module The usage of pyflakes as a Python module was introduced in e397c6d74652, to work around issue between Python 2 and Python 3. This issues are long behind us now and we can get beck to using pyflakes as a tool, giving us more flexibility about how we install it. The `hghave` requirements is modified to check that we have a tool available, instead of a python module.

clang-format: do not accept version above 19.x This gives me different formatting on latest version available in Debian. So disabling for now.

rust-update: add a config item to disable the Rust update fastpath Explanations inline.

update: add a Rust fast-path when updating from null (and clean) This case is easy to detect and we have all we need to generate a valid working copy and dirstate entirely in Rust, which speeds things up considerably: On my machine updating a repo of ~300k files goes from 10.00s down to 4.2s, all while consuming 50% less system time, with all caches hot. Something to note is that further improvements will probably happen with the upcoming `InnerRevlog` series that does smarter mmap hanlding, especially for filelogs. Here are benchmark numbers on a machine with only 4 cores (and no SMT enabled) ``` ### data-env-vars.name = heptapod-public-2024-03-25-ds2-pnm # benchmark.name = hg.command.update # bin-env-vars.hg.py-re2-module = default # bin-env-vars.hg.changeset.node = <this change> # benchmark.variants.atomic-update = no # benchmark.variants.scenario = null-to-tip # benchmark.variants.worker = default default: 5.328762 ~~~~~ rust: 1.308654 (-75.44%, -4.02) ### data-env-vars.name = mercurial-devel-2024-03-22-ds2-pnm # benchmark.name = hg.command.update # bin-env-vars.hg.py-re2-module = default # bin-env-vars.hg.changeset.node = <this change> # benchmark.variants.atomic-update = no # benchmark.variants.scenario = null-to-tip # benchmark.variants.worker = default default: 1.693271 ~~~~~ rust: 1.151053 (-32.02%, -0.54) ### data-env-vars.name = mozilla-unified-2024-03-22-ds2-pnm # benchmark.name = hg.command.update # bin-env-vars.hg.py-re2-module = default # bin-env-vars.hg.changeset.node = <this change> # benchmark.variants.atomic-update = no # benchmark.variants.scenario = null-to-tip # benchmark.variants.worker = default default: 38.901613 ~~~~~ rust: 11.637880 (-70.08%, -27.26) ### data-env-vars.name = netbsd-xsrc-public-2024-09-19-ds2-pnm # benchmark.name = hg.command.update # bin-env-vars.hg.py-re2-module = default # bin-env-vars.hg.changeset.node = <this change> # benchmark.variants.atomic-update = no # benchmark.variants.scenario = null-to-tip # benchmark.variants.worker = default default: 4.793727 ~~~~~ rust: 1.505905 (-68.59%, -3.29) ```

merge: improve working-copy mtime race handling Explanations inline. This also makes use of `make_mtime_reliable`, which unifies our mtime raciness logic from the status. On top of this, this fixes the handling of the pure dirstate status to better catch racy status, as we've been doing in Rust for a long time now.

merge: move the filtering of ambiguous files to a dedicated function I have multiple reasons: - The body of `_update` is way too long - This adds typing which will help our tooling and brains understand this code more easily - This function will get more nested and complex in the next patch I've taken the liberty of rewrapping and typo-passing the docstring.

timestamp: make the reliable comparison more usable from outside This is going to be used with pre-computed times unlike in status.

timestamp: add type information to the module This is easy to do and helps both Pytype and developpers understand what objects they are dealing with.

rust-dirstate: make the reliable timestamp comparison more usable from outside This is going to be used with pre-computed times unlike in status.

rust-dirstate-map: use a more precise identity This is closer to the behavior of what Python does. So far, we were checking only the inode, but this might not be good enough for the v1 case.

dirstate-map: add a missing debug wait point when accessing the v2 docket fc8e37c380d3 added synchronization points to the dirstate to allow for race condition testing without actually requiring a time-based race condition to happen. This changes adds the `pre-read-file` wait point before we read the docket, since callers might ask for the parents before anything else is read, leading to the first read being done before the wait point. This removes some differences in test output which were presumed to be speed related, but weren't.

rust-parsers: use the same error message as with the higher-level code This can happen at two places, but it's not really enough time to justify it being refactored. Let's ensure we have the same error message, the newer one being slightly more helpful.

rust-pathauditor: make sure we actually test the nested repo case This covers the *on-disk* case, where the path itself does not have a `.hg` component.

rust-pathauditor: match more of Python's behavior and display messages We will make use of the path auditor when running our update fast-path, and we want to output of it to be close enough.

rust-dirstate: pass dirstate tracked key from the requirements We will make use of this with the `hg update` fastpath

rust-files: check for empty manifests caused by narrow Explanations inline

rust-repo: don't use on-disk dirstate parents in v1 This mistake was not causing any problems yet since we were never updating parents from Rust code. This is about to change, so let's fix it.

rust-cpython: add a util to get a `Repo` from a python path I suspect this will not be the last time we need to do something like this.

rust: improve `InvalidRevision` error message I encountered this when debugging earlier and felt like we were losing some information along the way, which we were!

rust-dirstate: use a struct as arguments for the high-level `reset_state` This makes the interface a lot clearer at the call site and prevents silly mistakes, as an API with a bunch of booleans is prone to errors. This refactor adds a `from_empty` parameter for a fast-path when resetting and entry we're sure does not exist. It will be used in the upcoming update Rust fastpath, and was not split to prevent more churn.

rust: implement `From<SparseConfigWarning>` for `HgError` This will be useful in a future patch to avoid a lot of boilerplate.

rust-hg-cpython: add an `HgProgressBar` util This will be the entry point for all progress bars from a Python context in upcoming patches. Like the `Progress` trait, this is subject to change once we have more use cases, but this is good enough for now.

rust: add `Progress` trait for progress bars This will be used in the next few changes to introduce a progress bar for the `hg update` fastpath.

rust-files: separate the listing of files from a revset and a revision We won't need to parse a revset all the time, and an upcoming patch will make use of this new util.

rust-files: add a `Sync` bound to the matcher This enables us to use this code in multithreaded environements, which we very much want to do.

rust-files: also return filenode and flags when listing a revision's files This is going to be useful when implementing parts of `update` and makes it so we don't have to fetch the manifest and each entry twice.

rust-manifest: encode flags as `Option<NonZeroU8>` This makes the compiler use the niche optimization for all flags: since 0 is not a valid representation of any flags, we can use 0 as a replacement for `None`, which reduces memory footprint and could yield a little performance improvement over many iterations.

rust-repo: add a method to set the current parents This will be useful when we start writing an `hg update` fastpath.

rust-repo: add a method to get a `Node` from a `Revision` to the `Repo` This is going to be more and more useful as we start writing higher-level code

rust-utils: move the `filesystem_now` function to a util This is going to be useful for an upcoming `hg update` fastpath.

rust-dirstate: actually remember the identity This was an oversight that likely had no influence on anything since we have only been writing the dirstate for status. We will start writing the dirstate for more operations now, so we'll need this fixed.

mergestate: reduce the number of attribute lookups This code is called a lot during updates, this is a very small but also very easy thing to do.

test-lfs-serve-access: account for output differences in Python 3.12 This test is quite noisy, but the fix was very easy.

patchbomb: don't test ambiguous address This is a bug in Python's `email` package and shouldn't be relied on. Python 3.12 has fixed this problem¹ and raises an exception. We keep the multiple `-t` because this is still relevant for testing. [1] https://github.com/python/cpython/issues/102988 [2] https://docs.python.org/3/whatsnew/changelog.html

contrib: install pip 24.2 in the Windows dependency installer script One of the modern pythons was crying about the old version importing `distutils`.

contrib: install the latest py 3.11-3.13 in the Windows dependency script Note that python 3.11 adds an installer for arm64, which we ignore here because I don't have a Windows system running arm64, and it's likely we will be dropping support for some platforms before I get my hands on one.

contrib: update to the latest py3.9 and 3.10 in the Windows dependency script Python3.8 was already at its last version with an installer, FWIW.

contrib: drop python 3.7 from the Windows dependency installer script I'm tempted to drop 3.8 too, since we use 3.9 on Windows.

contrib: drop python 3.5 requirements file for Linux automation The new minimum this cycle is py3.8.

windows: implement `util.cachestat` to fix numerous dirstate problems I got here by bisecting the issue1790 related failure on Windows to keep an entry from being marked "unset" in `test-dirstate.t` back to eedbf8256263. There were a handful of other tests failing with an unexpected dirstate entry state like this, as well as numerous "skip updating dirstate: identity mismatch" messages added to various tests, as well as an issue with dirstate wrapping with the largefiles extension[1], all of which appear to be fixed by this. In total, ~25 tests are fully fixed on Windows with this change on default. This is basically a copy/paste of the posix implementation, but we drop the `st_mode` comparison- I think the only reason we care about the mode on posix is to detect +/-x mode changes, but the executable bits on Windows are synthesized based on the name of the file[2]. None of the other parts of the codebase are equipped to handle executable bits in the filesystem on Windows anyway, so it doesn't make sense to worry about them here. Note that `st_uid` and `st_gid` seem to always be 0 on Windows (and I can't find them being initialized), so they can probably be dropped from the comparison. But I doubt they matter any more on posix, since we don't track ownership. The `st_ino`, `st_dev`, and `st_nlink` attributes all seem to have reasonable values for comparing like on posix[3]. Also note that `st_ctime` is apparently deprecated in 3.12+ (for reasons I haven't explored)[4]. [1] https://foss.heptapod.net/mercurial/mercurial-devel/-/merge_requests/884 [2] https://github.com/python/cpython/blob/aab3210271136ad8e8fecd927b806602c463e1f2/Modules/posixmodule.c#L1948 [3] https://github.com/python/cpython/blob/aab3210271136ad8e8fecd927b806602c463e1f2/Python/fileutils.c#L1158 [4] https://github.com/python/cpython/blob/aab3210271136ad8e8fecd927b806602c463e1f2/Modules/posixmodule.c#L2200

extdiff: don't run gui programs when in a cli-only environment In order to provide a useful error message to override the behavior, we also need to slightly change the way that tool.gui is found in the config. Before, it had to be where tool.diffargs is located, which might not exist. Now, tool.isgui can exist on its own. A test is added for the new error message. We also need to force procutil.isgui() to return true, so we set $DISPLAY to a non-empty value before running any test expecting to have a gui.

bookflow: fix bullet list indentation in docstring

doc: generate separate commands/topics/extension pages This change modifies gendoc.py and Makefile so that individual pages for commands, help topics, and extensions can be generated. A new index page is also generated with links to all these pages. This makes it easier to look up and search the help text of a given command or topic, instead of having to deal with the giant hg.1 "all-in-one" page. Since the list of individual pages varies based on the source code, we generate a dynamic Makefile that contains this list of files as individual targets. This gives us fine-grained control over output files. However, it greatly increases the time spent generating all help pages. It's recommended to run make with -j to make use of multi-core archs. Individual man pages are produced in doc/man, and HTML ones are in doc/html

doc: refactor gendoc for better reusability This change separates the gathering of commands/topics/etc from the logic of printing their documentation out.

stream: prefer keeping an open file handle to volatile file instead of copy We will still do copy if too many file handle are open. Currently, have less than 10 volatile files in typical usage, so we should be fine. See inline documentation for details.

stream: remove __getitem__ from the VolatileManager It is no longer used and that API will get in the way of the new order.

stream: open volatile file through the manager To do more subtle things, we need more control.

stream: rename TempCopyManager to VolatileManager We are going to be more subtle in our management of volatile file, so use a more semantic name.

ci: only use the macOS runner if manually invoked Right now, we only have a single machine with a single runner, so don't create a bottleneck by default.

ci: add a runner for macos This is running from a venv, created with the packages from the requirements file in the `test-setup-future-work` topic being discussed separately. We can basically reuse the existing `test_c` configuration as-is, but there was one small issue- if the clone from the previous CI run is still present in `/tmp`, the clone for the current run fails. It's not a problem for the current setup because a new container is instantiated, so the easy fix is to simply delete the clone first. (The Windows CI avoids this by not cloning in the first place.) It looks like it takes ~80m to run the whole test suite on the current system.

branching: merge stable into default

tests: use shlex.quote instead of pipes.quote The pipes module got removed in python 3.13. https://bugs.debian.org/1084553

branchmap-v3: make sure we write the cache after detecting pure-topo mode We were properly detecting the pure topo-case but in some case the one disk cache file was never updated with that information.

demande-import-test: use `wsgiref` instead of `telnetlib` for testing The `telnetlib` module has been dropped in 3.13. The `wsgiref` module fit the same purpose of being obscure and not imported yet.

hgdemandimport: add collections.abc to the ignore list because Python 3.13 rc3 It worked in Python 3.13 rc2, but something changed for rc3. Now, when i18n.py imports typing and it touches collections.abc.Hashable , we get: ValueError: module object for collections.abc substituted in sys.modules during a lazy load This seems to be a general problem, released in the final Python 3.13 . I have not analyzed the problem in details. We *could* work around the problem by disabling demand import of typing. But that would effectively disable demand import of collections.abc too. Instead, just disable demand import for collections.abc .

run-tests: include non-activated venv packages in `PYTHONPATH` If a venv is activated since afa9d73780e1, `hghave` would see the packages installed in it, and enable related tests. If the python interpreter was launched directly however, none of them were seen. In addition to getting consistent behavior, it's also easier to manage in CI if the venv doesn't need activation.

tests: allow optional output when `test-lfs-server` is sent SIGTERM I guess this has only ever been tested on Windows, but on macOS, the test was failing with changes like: +++ /private/tmp/mercurial-ci/tests/test-lfs-test-server.t#git-server.err @@ -858,6 +858,7 @@ (Restart the server in a different location so it no longer has the content) $ "$PYTHON" $RUNTESTDIR/killdaemons.py $DAEMON_PIDS + $TESTTMP.sh: line 153: 38639 Terminated: 15 lfs-test-server > lfs-server.log 2>&1 #if hg-server $ cat $TESTTMP/access.log $TESTTMP/errors.log

tests: stabilize `test-http-bad-server.t` on macOS I'm not sure what's going on here, because I got one test run where this worked, but two where it gave this error instead. It's connection related as the comment says it should be, so good enough.

upgrade: disable using the parallel workers optimization on macOS It crashes `test-copies-chain-merge.t` for some reason[1]. It's only experimental, and already hard-disabled on Windows. [1] https://foss.heptapod.net/mercurial/mercurial-devel/-/jobs/2591045#L108

format: fix few black issues

install: static data moved from setup.py to pyproject.toml

ci: display tool version more selectively The goal here is to avoid showing tool version in all case as this create issue for non linux/docker runner that might not have some of them installed. This is currently relevant for the coming mac runner that does not have clang-format installed and does not needs its. This might also be useful in the future to use narrower docker image more specialized for each jobs.

zeroconf: fix a warning about a signature mismatch in a method override Caught by PyCharm. It looks like the `hdr` arg is mostly unused (thus why it was missing in some cases), so pass along an empty string where needed.

zeroconf: use str instead of bytes when indexing `globals()` I suppose since we set the key to bytes during init that it won't raise a KeyError, but this was very likely an oversight when mass-byteifying, rather than purposeful.

zeroconf: fix an invalid argument error on Windows The idea that pyoxidizer was triggering the problem when standing up the previous incarnation of CI for Windows was misleading- it was a Windows problem in general. See the inline bug link. Unfortunately, there's no commit referenced there, and it looks like OP closed the report himself with the suggested workaround. IOW, very modern python may not work, but it's extremely unlikely that there are any users of this extension, especially on Windows.

tests: fix hooks in `test-transaction-rollback-on-revlog-split.t` for Windows A few problems with external hooks here: 1) `cmd.exe` is blissfully unaware of the meaning of single quotes, and passes them along as part of the revision arg 2) `cmd.exe` doesn't know how to run the python script `f` with the shebang line, so it needs to be invoked with the intepreter explicitly. Then for some reason it was trying to open `$TESTTMP\troffset-computation-hooks\f`, so make it an absolute path. 3) Likewise, the shell script `wait-on-file` cannot be invoked directly by `cmd.exe`. 4) Windows python doesn't understand whatever `$TESTTMP` unrolls to (probably it has the double backslashes), and silently failed to write the lock file. I'm assuming that `pretxnclose.03-abort` is also bad (there's no `false` in `cmd.exe`), but it's not currently causing problems. Also note that the paths in this are very long, and typically exceed the `MAX_PATH` limit in Windows. It can be run with `run-tests.py --tmpdir /c/t`.

tests: use the cross platform `SIGKILL` function This partially fixes `test-transaction-rollback-on-revlog-split.t` (there are still problems related to waiting on the lock files), and completely fixes `test-fncache.t`.

tests: add a module that can perform the equivalent of `SIGKILL` on any OS I started with this being Windows specific, but let's push all of the decision making into this function so that it can just be called by the tests. The tradeoff is that this is very specific to sending `SIGKILL`- since `signal.SIGKILL` doesn't exist on Windows, the desired signal can't be passed from the caller. Maybe there's a way, but let's wait until there's a need. We don't use `killdaemons.py` unconditionally because it starts with a more graceful `SIGTERM` on posix.

tests: teach `killdaemons` on Windows to use an exit code provided by a caller Right now, there are several tests that use `signal.SIGKILL`, which isn't a thing on Windows. The `killdaemons` script approximates this by forcibly terminating the process. There's a minor difference in that `signal.SIGKILL` results in the test capturing an exit code of 137, and the `killdaemons` victim doesn't record a code (as though it exited with 0). Since the exit code line couldn't be conditionalized the last time I checked, let's just allow the caller to simulate the same exit code, and avoid conditionalizing the tests.

ci: also offer tests with Python 3.13 Python3.13 is du to be released soon. We better make sure we work with it.

run-test: add regular line break and test count in the dot output This will make it simpler to follow the actual progress. The value "75" have been picked as it is smaller or equal to 80 and give a nice round number.

clonebundle-digest: add recursion guards for Python 3.8 For Python 3.8 and 3.9, the read/readinto pair can recurse, so make sure the data is only hashed once.

tests: replace inline `waitlock()` with `wait-on-file` script The latter scales up the timeout based on the timeout value provided to the test runner, and I was seeing timeouts on Windows when running all of the tests using all CPU cores.

tests: stabilize `test-split-legacy-inline-changelog.t` on Windows The `tar` command is unable to process "C:\path\to\foo.tar" style paths, which is how `$TESTDIR` is constructed. It also didn't work with `$TESTDIR_FORWARD_SLASH`- both failed with: tar: Cannot connect to C: resolve failed [128] But `cat` can handle it if the path is quoted, and `tar` can read from stdin.

tests: stabilize `test-largefiles-cache.t` on Windows The `undo.backup.dirstate.bck` was created by `hg commit -m 'add largefile'` at line 18, and deleted in the conditional block by `hg push -q http://localhost:$HGPORT/` at line 138. That's... surprising, but probably not harmful, and can be debugged from Linux if it is a problem. This was showing up in `find src/.hg/largefiles/* | grep -E "(dirstate|$hash)"`.

tests: stabilize `test-journal.t` on Windows The file in these lines are double quoted, but single quoted on other platforms. Not sure why, other than `cmd.exe` doesn't recognize single quotes. But it's a cosmetic difference, so glob over it and move on.

tests: stabilize `test-clonebundles.t` on Windows The `remote: {foo,bar}` lines were different because `echo` in MSYS uses `\r\n`. I couldn't make it work with the previous echoing of individual lines, changing the internal `echo` to `printf "foo\n"`, because that output as "foon". This works on Linux and Windows, so I'm not thinking too hard about it.

tests: cleanup some special casing of `seq` emitting '\r\n' My guess is these predate the commit referenced in the previous commit.

tests: force `seq` to print with '\n' EOL It looks like consistent EOL is the reason for 0605726179a0, but now on py3, `print()` uses the platform EOL without regard to binary mode. The tests mostly use this to loop over a sequence of number in the shell, but there are a handful that redirect output to a file. Specifically, this fixes Windows runs of `test-bundle2-multiple-changegroups.t`, but there may be other tests this fixes. Some other `tests/*.py` files also set binary mode on stdout, but they also write bytes directly to `sys.stdout.buffer`. I'm not doing that here because PyCharm flags these write calls for passing bytes instead of str (PyCharm is likely wrong, but possibly confused because the code falls back to `sys.stdout` if there is no `.buffer` attribute), and it's annoying.

tests: stabilize `test-bundle-phase-internal.t` on Windows This is a silly difference, and we control the Windows side of things from mercurial/windows.py:195. I'll swap and quote the values to be like `no-windows` at some point, but I suspect this exception output would appear elsewhere, and don't feel like waiting for a 2h+ test run to find all of them.

tests: skip `test-wsgicgi.t` on MSYS The test is attempting to set `PATH_INFO="/rev/\xe2\x80\x94"` into the environment, which it does. The problem is that when MSYS sees a leading '/' in an environment variable, it thinks it's a unix filesystem path, so it "helpfully" prepends the Windows path to the MSYS root directory before running a non-MSYS process. hgweb would then split this value on '/', so it would get 'C:' instead of 'rev', and return a 400 since that isn't a valid web command. I tried generating a *.bat file, but had trouble running that via `cmd.exe` inside the test. I also tried generating an equivalent *.py launcher that would set the environment variables itself. But there is no `os.environb` on Windows, and the value was getting mangled when put into the script. So, I give up. If it's encoding stuff on Windows, it's probably broken.

clonebundles: allow manifest to specify sha256 digest of bundles

install: add long_description_content_type

headrevs: replace a boolean match with a if/else I missed that while doing a previous cleanup.

head-revs: add a native implementation of the `stop_rev` parameter This does not add too much complexity to the native code and help with branchmap v3 performance. Note that the final conversion of the heads from native-code to Python is still too costly, especially in Rust. In addition the current caching around headrevs is too simple and fragile. However these are an unrelated problem. ### benchmark.name = hg.command.unbundle # bin-env-vars.hg.py-re2-module = default # benchmark.variants.issue6528 = disabled # benchmark.variants.resource-usage = default # benchmark.variants.reuse-external-delta-parent = yes # benchmark.variants.revs = any-1-extra-rev # benchmark.variants.source = unbundle # benchmark.variants.validate = default # benchmark.variants.verbosity = quiet ## data-env-vars.name = netbeans-2018-08-01-zstd-sparse-revlog # bin-env-vars.hg.flavor = default branch-v2: 0.233711 ~~~~~ branch-v3 before: 0.239857 (+2.63%, +0.01) branch-v3 after: 0.239558 (+2.50%, +0.01) # bin-env-vars.hg.flavor = rust branch-v2: 0.235230 ~~~~~ branch-v3 before: 0.240972 (+2.44%, +0.01) branch-v3 after: 0.239917 (+1.99%, +0.00) ## data-env-vars.name = netbeans-2018-08-01-ds2-pnm # bin-env-vars.hg.flavor = rust branch-v2: 0.255586 ~~~~~ branch-v3 before: 0.268560 (+5.08%, +0.01) branch-v3 after: 0.262261 (+2.61%, +0.01) ## data-env-vars.name = mozilla-central-2024-03-22-zstd-sparse-revlog # bin-env-vars.hg.flavor = default branch-v2: 0.339010 ~~~~~ branch-v3 before: 0.349389 (+3.06%, +0.01) branch-v3 after: 0.348247 (+2.72%, +0.01) # bin-env-vars.hg.flavor = rust branch-v2: 0.346525 ~~~~~ branch-v3 before: 0.355661 (+2.64%, +0.01) branch-v3 after: 0.350906 (+1.26%, +0.00) ## data-env-vars.name = mozilla-central-2024-03-22-ds2-pnm # bin-env-vars.hg.flavor = rust branch-v2: 0.380202 ~~~~~ branch-v3 before: 0.408851 (+7.54%, +0.03) branch-v3 after: 0.406511 (+6.92%, +0.03) ## data-env-vars.name = mozilla-unified-2024-03-22-zstd-sparse-revlog # bin-env-vars.hg.flavor = default branch-v2: 0.412165 ~~~~~ branch-v3 before: 0.427782 (+3.79%, +0.02) branch-v3 after: 0.422595 (+2.53%, +0.01) # bin-env-vars.hg.flavor = rust branch-v2: 0.412397 ~~~~~ branch-v3 before: 0.422354 (+2.41%, +0.01) branch-v3 after: 0.421079 (+2.11%, +0.01) ## data-env-vars.name = mozilla-unified-2024-03-22-ds2-pnm # bin-env-vars.hg.flavor = rust branch-v2: 0.429501 ~~~~~ branch-v3 before: 0.443197 (+3.19%, +0.01) branch-v3 after: 0.449432 (+4.64%, +0.02) ## data-env-vars.name = mozilla-try-2024-03-26-zstd-sparse-revlog # bin-env-vars.hg.flavor = default branch-v2: 3.403171 ~~~~~ branch-v3 before: 3.819477 (+12.23%, +0.42) branch-v3 after: 3.658482 (+7.50%, +0.26) # bin-env-vars.hg.flavor = rust branch-v2: 3.454876 ~~~~~ branch-v3 before: 3.590284 (+3.92%, +0.14) branch-v3 after: 3.545843 (+2.63%, +0.09) ## data-env-vars.name = mozilla-try-2024-03-26-ds2-pnm # bin-env-vars.hg.flavor = rust branch-v2: 3.465435 ~~~~~ branch-v3 before: 3.633278 (+4.84%, +0.17) branch-v3 after: 3.556074 (+2.62%, +0.09)

head-revs: move hg-core's inner_headrevsfiltered closer to inner_headrevs This is pure code movement, it make a coming changesets significantly clearer.

branchmap-v3: introduce a "stop_rev" argument to `headsrevs` The `headsrevs` method of the revlog already have a `revs` argument to compute the headrevs of a limited set of heads. However, it disable the use of the native compiled code to compute the heads, which slows down the branchmap v3 code a lot. The branchmap v3 usage is actually quite constrained as we will always only ignores a part at the top of the graph. So we could be significantly faster. We start by making small change to the python side to improve the situation and introduce the new API. More collaboration with the native code are coming later. This massively speedup operation and close most of the remaining gaps between branchmap-v3 and branchmap-v2. especially on repository with many revs like mozilla-try. A small overhead remains mostly because the `headrevs` logic currently has some inefficiently. We will look into them from there. ### benchmark.name = hg.command.unbundle # bin-env-vars.hg.py-re2-module = default # benchmark.variants.issue6528 = disabled # benchmark.variants.resource-usage = default # benchmark.variants.reuse-external-delta-parent = yes # benchmark.variants.revs = any-1-extra-rev # benchmark.variants.source = unbundle # benchmark.variants.validate = default # benchmark.variants.verbosity = quiet ## data-env-vars.name = netbeans-2018-08-01-zstd-sparse-revlog # bin-env-vars.hg.flavor = default branch-v2: 0.233711 ~~~~~ branch-v3 before: 0.368769 (+57.79%, +0.14) branch-v3 after: 0.239857 (+2.63%, +0.01) # bin-env-vars.hg.flavor = rust branch-v2: 0.235230 ~~~~~ branch-v3 before: 0.372460 (+58.34%, +0.14) branch-v3 after: 0.240972 (+2.44%, +0.01) ## data-env-vars.name = netbeans-2018-08-01-ds2-pnm # bin-env-vars.hg.flavor = rust branch-v2: 0.255586 ~~~~~ branch-v3 before: 0.318907 (+24.78%, +0.06) branch-v3 after: 0.268560 (+5.08%, +0.01) ## data-env-vars.name = mozilla-central-2024-03-22-zstd-sparse-revlog # bin-env-vars.hg.flavor = default branch-v2: 0.339010 ~~~~~ branch-v3 before: 0.349752 (+3.17%, +0.01) branch-v3 after: 0.349389 (+3.06%, +0.01) # bin-env-vars.hg.flavor = rust branch-v2: 0.346525 ~~~~~ branch-v3 before: 0.354300 (+2.24%, +0.01) branch-v3 after: 0.355661 (+2.64%, +0.01) ## data-env-vars.name = mozilla-central-2024-03-22-ds2-pnm # bin-env-vars.hg.flavor = rust branch-v2: 0.380202 ~~~~~ branch-v3 before: 0.396293 (+4.23%, +0.02) branch-v3 after: 0.408851 (+7.54%, +0.03) ## data-env-vars.name = mozilla-unified-2024-03-22-zstd-sparse-revlog # bin-env-vars.hg.flavor = default branch-v2: 0.412165 ~~~~~ branch-v3 before: 0.424769 (+3.06%, +0.01) branch-v3 after: 0.427782 (+3.79%, +0.02) # bin-env-vars.hg.flavor = rust branch-v2: 0.412397 ~~~~~ branch-v3 before: 0.421796 (+2.28%, +0.01) branch-v3 after: 0.422354 (+2.41%, +0.01) ## data-env-vars.name = mozilla-unified-2024-03-22-ds2-pnm # bin-env-vars.hg.flavor = rust branch-v2: 0.429501 ~~~~~ branch-v3 before: 0.443849 (+3.34%, +0.01) branch-v3 after: 0.443197 (+3.19%, +0.01) ## data-env-vars.name = mozilla-try-2024-03-26-zstd-sparse-revlog # bin-env-vars.hg.flavor = default branch-v2: 3.403171 ~~~~~ branch-v3 before: 6.234055 (+83.18%, +2.83) branch-v3 after: 3.819477 (+12.23%, +0.42) # bin-env-vars.hg.flavor = rust branch-v2: 3.454876 ~~~~~ branch-v3 before: 6.307813 (+82.58%, +2.85) branch-v3 after: 3.590284 (+3.92%, +0.14) ## data-env-vars.name = mozilla-try-2024-03-26-ds2-pnm # bin-env-vars.hg.flavor = rust branch-v2: 3.465435 ~~~~~ branch-v3 before: 5.176076 (+49.36%, +1.71) branch-v3 after: 3.633278 (+4.84%, +0.17)

branchmap-v3: filter topo heads using node for performance reason The branchmap currently contains heads as nodeid. If we build a set of revnum with the topological heads, we need to turn the nodeid in the branchmap to revnum to be able to check if they are topo-heads. That nodeid → revnum lookup is "expensive" and adds up to something noticeable if you do it hundreds of thousand of time. Instead we turn all the topo-heads revnums into nodes and build a set. So we can directly test membership of the nodeids stored in the branchmap. That is much faster. Ideally we would have revnum in the branchmap and could directly test revnum against a revnum set and that would be even faster. However that's an adventure for another time. Without this change, the branchmap format "v3" was significantly slower than the "v2" format. With this changes, some of that gap is recovered With rust + persistent nodemap, this overhead was smaller because the extra lookup did not had to to build the nodemap from scratch. In addition the mozilla-unified repository is able to use the "pure_top" mode of branchmap v3, so it was not really affected by this. Future changeset will work of the remaining of the performance gap. ### benchmark.name = hg.command.unbundle # bin-env-vars.hg.py-re2-module = default # benchmark.variants.issue6528 = disabled # benchmark.variants.resource-usage = default # benchmark.variants.reuse-external-delta-parent = yes # benchmark.variants.revs = any-1-extra-rev # benchmark.variants.source = unbundle # benchmark.variants.validate = default # benchmark.variants.verbosity = quiet ## data-env-vars.name = netbeans-2018-08-01-zstd-sparse-revlog # bin-env-vars.hg.flavor = default branch-v2: 0.233711 ~~~~~ branch-v3 before: 0.380994 (+63.02%, +0.15) branch-v3 after: 0.368769 (+57.79%, +0.14) # bin-env-vars.hg.flavor = rust branch-v2: 0.235230 ~~~~~ branch-v3 before: 0.385060 (+63.70%, +0.15) branch-v3 after: 0.372460 (+58.34%, +0.14) ## data-env-vars.name = netbeans-2018-08-01-ds2-pnm # bin-env-vars.hg.flavor = rust branch-v2: 0.255586 ~~~~~ branch-v3 before: 0.317524 (+24.23%, +0.06) branch-v3 after: 0.318907 (+24.78%, +0.06) ## data-env-vars.name = mozilla-central-2024-03-22-zstd-sparse-revlog # bin-env-vars.hg.flavor = default branch-v2: 0.339010 ~~~~~ branch-v3 before: 0.410007 (+20.94%, +0.07) branch-v3 after: 0.349752 (+3.17%, +0.01) # bin-env-vars.hg.flavor = rust branch-v2: 0.346525 ~~~~~ branch-v3 before: 0.410428 (+18.44%, +0.06) branch-v3 after: 0.354300 (+2.24%, +0.01) ## data-env-vars.name = mozilla-central-2024-03-22-ds2-pnm # bin-env-vars.hg.flavor = rust branch-v2: 0.380202 ~~~~~ branch-v3 before: 0.393871 (+3.60%, +0.01) branch-v3 after: 0.396293 (+4.23%, +0.02) ## data-env-vars.name = mozilla-unified-2024-03-22-zstd-sparse-revlog # bin-env-vars.hg.flavor = default branch-v2: 0.412165 ~~~~~ branch-v3 before: 0.438105 (+6.29%, +0.03) branch-v3 after: 0.424769 (+3.06%, +0.01) # bin-env-vars.hg.flavor = rust branch-v2: 0.412397 ~~~~~ branch-v3 before: 0.438405 (+6.31%, +0.03) branch-v3 after: 0.421796 (+2.28%, +0.01) ## data-env-vars.name = mozilla-unified-2024-03-22-ds2-pnm # bin-env-vars.hg.flavor = rust branch-v2: 0.429501 ~~~~~ branch-v3 before: 0.452692 (+5.40%, +0.02) branch-v3 after: 0.443849 (+3.34%, +0.01) ## data-env-vars.name = mozilla-try-2024-03-26-zstd-sparse-revlog # bin-env-vars.hg.flavor = default branch-v2: 3.403171 ~~~~~ branch-v3 before: 6.562345 (+92.83%, +3.16) branch-v3 after: 6.234055 (+83.18%, +2.83) # bin-env-vars.hg.flavor = rust branch-v2: 3.454876 ~~~~~ branch-v3 before: 6.160248 (+78.31%, +2.71) branch-v3 after: 6.307813 (+82.58%, +2.85) ## data-env-vars.name = mozilla-try-2024-03-26-ds2-pnm # bin-env-vars.hg.flavor = rust branch-v2: 3.465435 ~~~~~ branch-v3 before: 5.381648 (+55.30%, +1.92) branch-v3 after: 5.176076 (+49.36%, +1.71)

rev-branch-cache: properly ignores unaligned trailing data Previously, trailing data could lead to crash and would be written back to disk, disaligning all new data… This is no longer the cases. This was detected while playing with branchmap-v3 that access the rev-branch-cache much more aggressively.

rev-branch-cache: stop pretending we will overwrite data when we don't We were issuing a message about overwriting data even when we were about to write 0 bytes in pratice. This is silly. Instead we point at the extra data remaining in the file (in case someone is using debug to debug something).

rev-branch-cache: fix message about overwritten data If there is data at the end we ignore, we should not count them as overwritten. This reveal that we something don't overwrite anything, this will be taken car of in the next changeset.

branchmap-v3: detect invalid headerline and raise error Otherwise, broken cache file would be accept as valid, but empty.

head-revs: merge the two inner_headrevs… variants Now that there is only one method, it does not make sense to have two different "inner" method. This is especially true as we are about to add another parameter to the method. So we clean up before that.

head-revs: move hg-cpython's inner_headrevsfiltered closer to inner_headrevs This is pure code movement, it make a coming changesets significantly clearer.

head-revs: teach the pure indexes about the `headrevs` method Having this computation done at the index level unify the API and remove revlog side complexity. It might also be a front runner of handing more responsability to the index.

head-revs: remove the `headrevsfiltered` method on the index This method is no longer called and can be removed.

head-revs: stop calling headrevsfiltered The `headrevs` method have been accepting filter since Mercurial 3.2¹. I guess we can rely on it for now. (except the Rust extension that just gained this capability, but it has it now)

rust-cpython: also accept the `filteredrevs` argument in index.headrevs The C version have been accepting this argument since Mercurial 3.2, lets align the Rust index here. This will make it possible to simplify the code in later changesets.

branchmap: use the proper experimental name in cacheutil Otherwise they are not properly copied around.

interfaces: introduce and use a protocol class for the `mpatch` module See f2832de2a46c for details when this was done for the `bdiff` module. Two things worth pointing out- 1) The `cffi` module "inherits" the `pure` implementation of `patchedsize()` because of its wildcard import. 2) It's odd that the `mpatchError` lives in both `pure` and `cext` modules. I initially thought to move the exception into the new class, and make the existing class name an alias to the class in the new location, but the exception is created in C code by the `cext` module, so that won't work. I don't think a protocol class is approriate, because there's nothing special about the class to distinguish from any other `Exception`. Fortunately, nobody is catching this exception in core, so we can kick the can down the road.

branching: merge stable into default

doctest: use the system hg to find the list of file to tests Same as what we do for the `.t` tests, but more manually.

relnotes: skip the test if the source repository is not readable The test want to run the relnot extension, with the tested mercurial, on the original repository. This is not always possible (e.g. when running with --pure and the repository use zstd for example). So we skip the test in this case.

check-code: document reason and suggest alternative to exit code negation The check was introduced in f48b075ff088. We trust the explanation from there.

ci: drop the HGMODULEPOLICY overwrite The test runner have been setting them properly for a while now. So we don't need to do it manually anymore.

test: preserve the existing module policy config for testrepohg resuse The test runner is, righfully, setting HGMODULEPOLICY to match the flavor requests to run the tests. However this mess with the `testrepohg` ability to run properly, as the newly set policy might be incompatible with its installation, or its ability to read the test repo. So we preserve the initial value in a dedicated variable and use it in the `testrepohg` helper.

module-policy: ignore empty module policy This make the variable easier to work with, the empty value is not ambiguous about not wanting to get in the way.

tests: provide an alternate fake lock for filesystems without symlink support

tests: disable `worker.backgroundclose` to stabilize a test on Windows TIL that `worker.enabled=0` doesn't prevent these workers from spinning up. At any rate, there's already a whole lot of conditionalized output following `cat client.log`, the placement of the "starting 4 threads for background file closing" message seems unstable, and we don't care about those worker threads here. Preventing the message is better for test maintenance.

tests: fix lock file path mangling in `test-racy-mutations.t` on Windows I guess `$TESTTMP_FORWARD_SLASH` gets translated by MSYS. This was in the `.foo_commit_out` file: sh: C;C:\\MinGW\\msys\\1.0\\Users\\Matt\\AppData\\Local\\Temp\\hgtests.1qc8jmdl\\child2\\test-racy-mutations.t-skip-detection\\waitlock_editor.sh: $ENOENT

tests: stabilize `test-status-eacces.t` on Windows As noted earlier, `chmod` doesn't complain in MSYS, but also doesn't alter the file permissions such that they are unreadable. I'm guessing the other lines of output in this area that are gated on `rhg` (or not) will also need this, but I don't want to dig too deeply into something that is apparently working well enough.

run-tests: bump the default timeout on Windows to 4x the normal value There are a ridiculous number of tests that timeout on Windows with the 360 sec default (~60). And because of the bug where timed out tests still run to completion before the results are thrown away[1], the timeout does nothing but waste time, so there's no reason to try to find a lower value that still works. For reference on my system: # Ran 909 tests, 116 skipped, 119 failed. python hash seed: 2052473208 real 151m44.322s user 0m0.077s sys 0m0.046s [1] I thought that I wrote a bug for this, but search isn't finding it.

run-tests: bump the minimum python to 3.8 Presumably this was an oversight when hg was updated to 3.8.

tests: stabilize `test-sparse.t` on Windows One of the reserved characters for path values is '*', so it can't be used. Fortunately, missing this seems to not get in the way of any other tests, and it is removed shortly after with `rm -r foo*bar`, and the extant 'foo-bar' matches the pattern.

tests: fix a test hang on Windows when setting a debuglock I have no idea why, but running the `hg -R auto-upgrade debuglock --set-lock` command near the end of `test-upgrade-repo.t` hangs the test. It does background the process and `killdaemons.py` runs without error, but control doesn't return to `run-tests.py` until the process is manually killed. I did notice that `$!` in MSYS is *not* the PID of the process that got backgrounded, even when a simple `sleep 60 &` is run in MSYS without the *.t file. When `killdaemons.py` is run manually with the PID in ProcessExplorer, the backgrounded process terminates immediately, and returns control to `run-tests.py`. This looks like it would be a race, but the test waits 10s for the lock file to appear before attempting to kill the process, so there's time. `hg serve` has a `--pid-file` option to write the pid to the file, but this is only a debug command, so I'm not bothering with cluttering the command line.

tests: conditionalize `chmod` usage in `test-upgrade-repo.t` While the command itself doesn't error out on Windows, it also doesn't make the filesystem readonly. Therefore the repo gets altered to drop dirstate-v2, and puts it out of sync with that happens on Linux.

tests: print the actual timeout value used in `wait-on-file` Previously, it was printing the time passed in, prior to it being scaled up to account for a longer timeout.

tests: stabilize `test-transaction-wc-rollback-race.t` on Windows MSYS has a tendency to munge C:\Dir\SubDir\File into C:DirSubDirFile unless it is quoted, and that's what was happening here- there were a lot of these failures: file not created after 5 seconds: $TESTTMP/transaction-waiting I suspect quoting is only needed in the hook script that is generated (the catting of the log file pointed me in the right direction here), but I missed a spot and trial and error got me here. The quoting elsewhere doesn't harm anything and it was taking 7+ minutes to run this test when things were timing out, so I don't feel like reducing the quoting to the minimum required.

tests: stabilize `test-merge-partial-tool.t` on Windows The test was previously failing because it was opening the shell scripts being used as an executable in a text editor, and problems cascaded from there.

tests: replace `hg id --debug -i` command substitution with non-debug command The censor and convert tests were failing on Windows because the `--debug` flag also prints debug messages, and at least some of these were outputting: skip updating dirstate: identity mismatch ${node} Obviously that causes cascading problems. The other tests were OK, but it's better to use a non debug command for stability.

tests: correct Windows output to account for putting repos in `repo` subdir These were missed in 55c6ebd11cb9, due to being conditionalized and not running in CI.

tests: use pattern matching to mask `ECONNREFUSED` messages The second and third one of these in `test-http-proxy.t` was failing on Windows. The others were found by grep and by failed tests when output was matched and an attempt was made to emit the mask pattern. The first clonebundles failure on Windows emitted: error fetching bundle: [WinError 10061] $ECONNREFUSED$ We should probably stringify that better to get rid of the "[WinError 10061]" part.

typing: add stub functions for `cext/charencoding` I'm not sure if it's better to have a separate file, and currently pytype doesn't really know how to handle these, so it's no help in figuring that out. Technically, these methods are part of the `mercurial.cext.parsers` module, so put them into the existing stub until there's a reason to split it out.

interfaces: introduce and use a protocol class for the `charencoding` module See f2832de2a46c for details when this was done for the `bdiff` module. This lets us dump the hack where the `pure` implementation was imported during the type checking phase to provide signatures for the module methods it provides. Now the protocol classes are starting to shine, because these methods are provided by `pure.charencoding` and `cext.parsers`, and references to `cffi.charencoding` and `cext.charencoding` are forwarded to them as appropriate by the `policy` module. But none of that matters, as long as the module returned provides the listed methods. The interface was copy/pasted from the `pure` module, but `jsonescapeu8fallback` is omitted because it is accessed from the `pure` module directly when the escaping fails in the primary module's `jsonescapeu8()`.

debugantivirusrunning: use bytes when opening a vfs file I noticed this when searching for "base85" to see if anything else in the previous commit needed to be annotated. This was added in 87047efbc6a6, after the mass byteification in 687b865b95ad.

interfaces: introduce and use a protocol class for the `base85` module See f2832de2a46c for details when this was done for the `bdiff` module. It looks like PEP-688 removed the special casing of `bytes` being a standin for any type of `ByteString`, and defines a `typing.Buffer` class (with a backport in `typing_extensions` for Python prior to 3.12). There's been a lot of churn in this area with pytype, but recent versions of pytype and PyCharm recognize this, and e.g. have `mercurial.node.hex()` defined as: from typing_extensions import Buffer def hex(data: Buffer, sep: str | bytes = ..., bytes_per_sep: int = ...) -> bytes This covers `bytes`, `bytearray`, and `memoryview` by default. Both of the C functions here use `y#` to parse the arguments, which means the arg is a byte-like object[2], so the args would appear to be better typed as `Buffer`. However, pytype has a bug that prevents using this from `typing_extensions`[3], and mypy complained `Unsupported left operand type for + ("memoryview")` in the pure module on line 37 (meaning it's only a subset of `Buffer`). So hold off on changing any of that for now. [1] https://peps.python.org/pep-0688/#no-special-meaning-for-bytes [2] https://docs.python.org/3/glossary.html#term-bytes-like-object [3] https://github.com/google/pytype/issues/1772

base85: avoid a spurious use-before-initialized warning in `pure` module The error wasn't possible because the only way for `acc` to not be initialized was if `len(text) == 0`. But then `0 % 5 == 0`, so no attempt at padding was done. It's a simple enough fix to not have PyCharm flag this though. The value needs to be reset on each loop iteration, so it's a line copy, not a line move.

typing: add type annotations to `mercurial/mdiff.py` We'll leave converting `diffopts` to `attrs` as another project.

mdiff: convert a few block definitions from lists to tuples These were flagged by adding type hints. Some places were using a tuple of 4 ints to define a block, and others were using a list of 4. A tuple is better for typing, because we can define the length and the type of each entry. One of the places had to redefine the tuple, since writing to a tuple at an index isn't supported. This change spills out into the tests, and archeology says it was added to the repo in this state. There was no reason given for the divergence, and I suspect it wasn't intentional. It looks like `splitblock()` is completely unused in the codebase.

interfaces: add the optional `bdiff.xdiffblocks()` method PyCharm flagged where this was called on the protocol class in `mdiff.py` in the previous commit, but pytype completely missed it. PyCharm is correct here, but I'm committing this separately to highlight this potential problem- some of the implementations don't implement _all_ of the methods the others do, and there's not a great way to indicate on a protocol class that a method or attribute is optional- that's kinda the opposite of what static typing is about. Making the method an `Optional[Callable]` attribute works here, and keeps both PyCharm and pytype happy, and the generated `mdiff.pyi` and `modules.pyi` look reasonable. We might be getting a little lucky, because the method isn't invoked directly- it is returned from another method that selects which block function to use. Except since it is declared on the protocol class, every module needs this attribute (in theory, but in practice this doesn't seem to be checked), so the check for it on the module has to change from `hasattr()` to `getattr(..., None)`. We defer defining the optional attrs to the type checking phase as an extra precaution- that way it isn't an attr with a `None` value at runtime if someone is still using `hasattr()`. As to why pytype missed this, I have no clue. The generated `mdiff.pyi` even has the global variable typed as `bdiff: intmod.BDiff`, so uses of it really should comply with what is on the class, protocol class or not.

interfaces: introduce and use a protocol class for the `bdiff` module This is allowed by PEP 544[1], and we basically follow the example there. The class here is copied from `mercurial.pure.bdiff`, and the implementation removed. There are several modules that have a few different implementations, and the implementation chosen is controlled by `HGMODULEPOLICY`. The module is loaded via `mercurial/policy.py`, and has been inferred by pytype as `Any` up to this point. Therefore it and PyCharm were blind to all functions on the module, and their signatures. Also, having multiple instances of the same module allows their signatures to get out of sync. Introducing a protocol class allows the loaded module that is stored in a variable to be given type info, which cascades through the various places it is used. This change alters 11 *.pyi files, for example. In theory, this would also allow us to ensure the various implementations of the same module are kept in alignment- simply import the module in a test module, attempt to pass it to a function that uses the corresponding protocol as an argument, and run pytype on it. In practice, this doesn't work (yet). PyCharm (erroneously) flags imported modules being passed where a protocol class is used[2]. Pytype has problems the other way- it fails to detect when a module that doesn't adhere to the protocol is passed to a protocol argument. The good news is that mypy properly detects this case. The bad news is that mypy spews a bunch of other errors when importing even simple modules, like the various `bdiff` modules. Therefore I'm punting on the tests for now because the type info around a loaded module in PyCharm is a clear win by itself. [1] https://peps.python.org/pep-0544/#modules-as-implementations-of-protocols [2] https://youtrack.jetbrains.com/issue/PY-58679/Support-modules-implementing-protocols

mdiff: tweak calls into `bdiff.fixws` to match its type hints It turns out that protocol classes can be used for modules too, which is great because all of the dynamically loaded modules (and their attributes) are currently inferred as `Any`. See the next commit for details. A protocol class for the `bdiff` module detected this (trivial) mismatch, so correct it first. The various implementations of this method are typed as taking a `bool`. The `cext` implementation parses its arguments with `PyArg_ParseTuple(args, "Sb:fixws", &s, &allws)`, which wants an `int`. But experimenting in `hg debugshell` under py38, passing `True` or `False` to `cext.fixws()` also works. We can change the implementation to use "p" (which was introduced in py33) instead of "b", but that's beyond the scope of this.

util: minor copy editing of the documentation for `mmapread()`

util: make `mmapread()` work on Windows again 522b4d729e89 started referencing `mmap.MAP_PRIVATE`, but that's not available on Windows, so `hg version` worked, but `make local` did not. That commit also started calling the constructor with the fine-grained `flags` and `prot` args, but those aren't available on Windows either[1] (though the backing C code doesn't seem conditionalized to disallow usage of them). I assume the change away from from the `access` arg was to provide the same options, plus `MAP_POPULATE`. Looking at the source code[2], they're not quite the same- `ACCESS_READ` is equivalent to `flags = MAP_SHARED` and `prot = PROT_READ`. `MAP_PRIVATE` is only used with `ACCESS_COPY`, which allows read and write. Therefore, we can't quite get the same baseline flags on Windows, but this was the status quo ante and `MAP_POPULATE` is a Linux thing, so presumably it works. I realize that typically the OS differences are abstracted into the platform modules, but I'm leaving it here so that it is obvious what the differences are between the platforms. [1] https://docs.python.org/3/library/mmap.html#mmap.mmap [2] https://github.com/python/cpython/blob/5e0abb47886bc665eefdcc19fde985f803e49d4c/Modules/mmapmodule.c#L1539

typing: add type annotations to the dirstate classes The basic procedure here was to use `merge-pyi` to merge the `git/dirstate.pyi` file in (after renaming the interface class to match), cleaning up the import statement mess, and then repeating the procedure for `mercurial/dirstate.pyi`. Surprisingly, git's dirstate had more hints inferred in its *.pyi file. After that, it was a manual examination of each method in the interface, and how they were implemented in the core and git classes to verify what was inferred by pytype, and fill in the missing gaps. Since this involved jumping around between three different files, I applied the same type info to all three at the same time. Complex types I rolled up into type aliases in the interface module, and used that as needed. That way if it changes, there's one place to edit. There are some hints still missing, and some documentation that doesn't match the signatures. They should all be marked with TODOs. There are also a bunch of methods on the core class that aren't on the Protocol class that seem like maybe they should be (like `set_tracked()`). There are even more methods missing from the git class. But that's a project for another time.

interfaces: change a couple of dirstate fields to `@property` As I was adding type hints here and to the concrete classes, PyCharm flagged the property in the core class as not being compatible with the base class's version.

git: make `dirstate.parents()` return a list like the core class The core class returned a list, so that's how I type annotated it, and this got flagged. I suppose we could annotate it as a `Sequence[bytes]`, but it's a trivial difference.

typing: add type hints for the overloads of `matchmod.readpatternfile()` The return type is conditional on an argument passed, and it very much confused both pytype and PyCharm inside `dirstate._ignorefileandline()` after adding type hints for the return value there.

dirstate: subclass the new dirstate Protocol class Behold the chaos that ensues. We'll use the generated *.pyi files to apply type annotations to the interface, and see how much agrees with the documentation. Since the CamelCase name was used to try to work around pytype issues with zope interfaces and is a new innovation this cycle (see c1d7ac70980b), drop the CamelCase name. I think the Protocol classes *should* be CamelCase, but that can be done later in one pass. For now, the CamelCase alias is extra noise in the *.pyi files.

git: correct some signature mismatches between dirstate and the Protocol class These were flagged by PyCharm when subclassing the Protocol class. Note that both `is_changing_xxx` were only flagged when the Protocol class used a plain field, as mentioned in the previous commit. After converting those attrs in the Protocol class to @property to match the regular dirstate class, it stopped flagging these. But I don't think that makes sense- `@property` should look like an attribute to the outside world, not a callable.

interfaces: convert the zope `Attribute` attrs to regular fields At this point, we should have a useful protocol class. The file syntax requires the type to be supplied for any fields that are declared, but we'll leave the complex ones partially unspecified for now, for simplicity. (Also, the things documented as `Callable` are really as future type annotating worked showed- roll with it for now, but they're marked as TODO for fixing later.) All of the fields and all of the attrs will need type annotations, or the type rules say they are considered to be `Any`. That can be done in a separate pass, possibly applying the `dirstate.pyi` file generated from the concrete class. The first cut of this turned the `interfaceutil.Attribute` fields into plain fields, and thus the types on them. PyCharm flagged a few things as having incompatible signatures when the concrete dirstate class subclassed this, when the concrete class has them declared as `@property`. So they've been changed to `@property` here in those cases. The remaining fields that are decorated in the concrete class have comments noting the differences. We'll see if they need to be changed going forward, but leave them for now. We'll be in trouble if the `@util.propertycache` is needed, because we can't import that module here at runtime, due to circular imports.

interfaces: add the missing `self` arg to the dirstate Protocol class This clears all of the errors that PyCharm has been flagging in this file, since the zope interface was declared here.

interfaces: convert the dirstate zope interface to a Protocol class This is a small trial run for converting the repository interfaces enmasse, in the same series of steps. I'm not sure that this current code is valid (it has zope attribute fields, and it's missing all of the `self` args on its functions, but that was the previous state of things, and made PyCharm really unhappy). But it will be easier to review the repository interface changes if this change is separate from adding `self` and dropping the zope attributes all over. Having an empty constructor in a protocol is weird. I'm not sure if these args should be converted to fields that all subclasses would have, and comments around existing attributes say some should be going away. Comment it out for now so that it's not in the way, but also not forgotten.

tests: disable `test-check-interfaces.py` while converting to protocols The goal is to convert everything, so get it all out of the way. The interfaces don't get that much maintenance that this needs to be tested right now.

tests: always access the mercurial repo through `helpers-testrepo.sh` In some contexts the mercurial repo needs to be accessed through system hg. That's what `helpers-testrepo.sh` enforces, but some tests incorrectly use the mercurial repo without going through that script. This patch fixes those tests.

tests: in helpers-testrepo.sh switch from shell aliases to functions The reason is that I want this script to work in non-interactive shells too. (will be used in the next commit)

rust: fix the deprecation warning in NaiveDateTime::from_timestamp This warning appears between chrono 0.4.34 and 0.4.38, so isn't affecting the current lock file, but it would come when we upgraded the version.

run-tests: ensure that --no-rust do not use rust Since having a valid value in HGWITHRUSTEXT is enough to trigger the use of rust, we need to unset it before install to be sure.

revlogutils: teach issue6528 filtering about grandparents During dynamic filtering, we should assume that the current repository is correct. Therefore the parents of the delta base can tell us if that parent has metadata without having to build the whole text.

revlogutils: remember known metadata parents for issue6528 In the cases where the parent revs tell us for sure that the parent has metadata, remember this fact to avoid content recomputations later.

revlogutils: for issue6528 fix, pre-cache nullrev as metadata-free

revlogutils: for issue6528 fix, cache results for null changes

revlogutils: fix _chunk() reference _chunk is only found in the inner revlog object and not directly exposed outside.

rev-branch-cache: reenable memory mapping of the revision data Now that we are no longer truncating it, we can mmap it again. This provide a sizeable speedup on repository with a very large amount of revision for example for a mozilla-try clone with 5 793 383 revisions, this provide a speedup of 5ms - 10ms. Since they happens within the "critical" locked path during push. These miliseconds are important. In addition, the v3 branchmap format is use the rev-branch-cache more than the v2 branchmap cache so this will be important. On smaller repository we consistently see an improvement of one or two percents, but the gain in absolute time is usually < 10 ms. #### benchmark.name = hg.command.unbundle # benchmark.variants.issue6528 = disabled # benchmark.variants.reuse-external-delta-parent = yes # benchmark.variants.revs = any-1-extra-rev # benchmark.variants.source = unbundle # benchmark.variants.verbosity = quiet ### data-env-vars.name = mozilla-try-2024-03-26-zstd-sparse-revlog ## bin-env-vars.hg.flavor = default e51161b12c7e: 3.527923 ebdcfe85b070: 3.468178 (-1.69%, -0.06) ## bin-env-vars.hg.flavor = rust e51161b12c7e: 3.580158 ebdcfe85b070: 3.480564 (-2.78%, -0.10) ### data-env-vars.name = mozilla-try-2024-03-26-ds2-pnm ## bin-env-vars.hg.flavor = rust e51161b12c7e: 3.527923 ebdcfe85b070: 3.468178 (-1.69%, -0.06)

rev-branch-cache: have debugupdatecache warm rbc too Since the "v2" format can be more performant than the "v1" format (thanks to mmap), it is useful to be able to make sure it is present

rev-branch-cache: schedule a write of the "v2" format if we read from "v1" The new file can be memorymapped, while the old one cannot. So there is value in having the v2 format around as soon a possible.

rev-branch-cache: fallback on "v1" data if no v2 is found This will help smooth the transition to the v2 format for existing large repository.

rev-branch-cache: increment the version to "v2" We want to ensure no older clients will truncate the file under us. So we need to change their name. We don't change the rest of the format (unfortunaly).

rev-branch-cache: stop truncating cache file Truncating the file prevent the safe use of mmap. So instead of overwrite the existing data. If more than 20% of the file is to be overwritten, we rewrite the whole file instead. Such whole rewrite is done by replacing the old one with a new one, so mmap of the old file would be affected. This prepare a more aggressive use of mmap in later patches.

rev-branch-cache: make sure we close the name file we open We were various opening without with or try. Adding a try would not hurt.

rev-branch-cache: add a way to force rewrite of the cache This seems useful to be able to do this, for example during strip. This align with the intended expressed in the `test-branches.t` test. This will help use being more confident about future changes in the series.

rev-branch-cache: issue more truthful "truncating" message First, don't pretend it truncate to 40 when it actually truncate to 0. Second, don't pretend to truncate to 0 when the file is already empty/missing.

rev-branch-cache: move the code in a dedicated module The branchmap module is getting huge and the rev branch cache is fully independent, lets move it elsewhere.

statichttprepo: stop shadowing the `bytes` builtin PyCharm flagged it, but I also misunderstood when looking at the code, because the name implied a byte string, not a number.

statichttprepo: fix `httprangereader.read()` for py3 It looks like there were a bunch of problems, not all of them py3 related: 1) The signature of BinaryIO.read() is -1, not None 2) The `end` variable can't be bytes and interpolate into str with "%s" 3) The `end` variable can't be an int and interpolate into str with "%s" 4) The result slicing could be out of bounds if more is requested than returned I guess if somebody would have called `read(-1)` (either directly or because a wrapper defaults to that), it wouldn't have been handled correctly. The fact that it is a valid value meaning to read everything requires some additional changes later in the method around when it slices the byte string that was read, but that seems to have already been broken.

statichttprepo: use a context manager to handle a file descriptor I'm not sure if this should be reduced to `vfs.exists()`. That would seem to be equivalent code (since the result of the read is ignored, so we can't tell if the file actually has content, which has been the state of things going back to 98b6c3dde237), but this is at least safer file descriptor handling.

profiling: pass bytes to `_()` and `error.Abort()` And of course `other_tool_name` is str too, so that needs to be converted. The type hints from PyCharm say `sys.monitoring.get_tool()` can return None, so handle that case explicitly before it trips up pytype.

exchange: improve computation of relevant markers for large repos Compute the candidate nodes with relevant markers directly from keys of the predecessors/successors/children dictionaries of obsstore. This is faster than iterating over all nodes directly. This test could be further improved for repositories with relative few markers compared to the repository size, but this is no longer hot already. With the current loop structure, the obshashrange use works as well as before as it passes lists with a single node. Adjust the interface by allowing revision lists as well as node lists. This helps cases that computes ancestors as it reduces the materialisation cost. Use this in _pushdiscoveryobsmarker and _getbundleobsmarkerpart. Improve the latter further by directly using ancestors(). Performance benchmarks show notable and welcome improvement to no-op push and pull (that would also apply to other push/pull). This apply to push and pull done without evolve. ### push/pull Benchmark parameter # bin-env-vars.hg.flavor = default # benchmark.variants.explicit-rev = none # benchmark.variants.protocol = ssh # benchmark.variants.revs = none ## benchmark.name = hg.command.pull # data-env-vars.name = mercurial-devel-2024-03-22-zstd-sparse-revlog before: 5.968537 seconds after: 5.668507 seconds (-5.03%, -0.30) # data-env-vars.name = tryton-devel-2024-03-22-zstd-sparse-revlog before: 1.446232 seconds after: 0.835553 seconds (-42.23%, -0.61) # data-env-vars.name = netbsd-src-draft-2024-09-19-zstd-sparse-revlog before: 5.777412 seconds after: 2.523454 seconds (-56.32%, -3.25) ## benchmark.name = hg.command.push # data-env-vars.name = mercurial-devel-2024-03-22-zstd-sparse-revlog before: 6.155501 seconds after: 5.885072 seconds (-4.39%, -0.27) # data-env-vars.name = tryton-devel-2024-03-22-zstd-sparse-revlog before: 1.491054 seconds after: 0.934882 seconds (-37.30%, -0.56) # data-env-vars.name = netbsd-src-draft-2024-09-19-zstd-sparse-revlog before: 5.902494 seconds after: 2.957644 seconds (-49.89%, -2.94) There is not notable different in these result using the "rust" flavor instead of the "default". The performance impact on the same operation when using evolve were also tested and no impact was noted.

typing: make the localrepo classes known to pytype 9d4ad05bc91c and 1b17309cdaab both mentioned making `bundlerepository` and `unionrepository` subclass `localrepository` during the type checking phase, but that didn't apply to pytype in practice. See bcaa5d408657 and friends for how the zope interfaces confuse pytype, and end up converting the classes they decorate into `Any`. This commit is slightly more complex though, because `localrepository` has mixin classes applied to it when it is instantiated. Specifically, `RevlogFileStorage` is added, which adds `def file(f)` (which isn't defined on `localrepository`). Therefore a list of `localrepository` superclasses is provided during type checking to account for the mixins. Without this, the `bundlerepository` class gets flagged when it attempts to call its superclass implementation of `file()`. Note that pytype doesn't understand these mixin superclasses (it marks the superclass of `localrepository` as `Any`, because they are zope interfaces it doesn't understand), but that's enough to get it to not flag `bundlerepository`. PyCharm also stops flagging it as a missing function, though it seems like it is able to handle the zope interfaces.

typing: add a handful more annotations to `mercurial/vfs.py` These came out of refactoring into a protocol class, but they can stand on their own. The `audit` callback is kinda screwy because the internal lambda and the callable for `pathutil.pathauditor` have different args and a different return type. It's conditionalized where it is called, and can be cleaned up later if desired.

typing: make `vfs.isfileorlink_checkdir()` path arg required The only caller to this is `merge._checkunknownfile()`, which supplies a value. That's good, because `util.localpath()` immediately uses the value to call a method on it on Windows. The posix implementation returns the value unaltered, but then `pathutil.finddirs_rev_noroot()` would have exploded.

typing: manually add type annotations to `mercurial/vfs.py` This isn't everything, but hopefully it's close enough to hack on a protocol class.

typing: correct pytype mistakes in `mercurial/vfs.py` With the previous changes in this series (prior to merging the *.pyi file), this wasn't too bad- the only definitively wrong things were the `data` argument to `writelines()`, and the return type on `backgroundclosing()` (both of these errors were dropped in the previous commit; for some reason pytype doesn't like `contextlib._GeneratorContextManager`, even though that's what it determined it is): File "/mnt/c/Users/Matt/hg/mercurial/vfs.py", line 411, in abstractvfs: Bad return type 'contextlib._GeneratorContextManager' for generator function abstractvfs.backgroundclosing [bad-yield-annotation] Expected Generator, Iterable or Iterator PyCharm thinks this is `Generator[backgroundfilecloser], Any, None]`, which can be reduced to `Iterator[backgroundfilecloser]`, but pytype flagged the line that calls `yield` without an argument unless it's also `Optional`. PyCharm is happy either way. For some reason, `Iterable` didn't work for pytype: File "/mnt/c/Users/Matt/hg/mercurial/vfs.py", line 390, in abstractvfs: Function contextlib.contextmanager was called with the wrong arguments [wrong-arg-types] Expected: (func: Callable[[Any], Iterator]) Actually passed: (func: Callable[[Any, Any, Any], Iterable[Optional[Any]]]) Attributes of protocol Iterator[_T_co] are not implemented on Iterable[Optional[Any]]: __next__

typing: run `merge-pyi` on `mercurial/vfs.py` The *.pyi file was generated with pytype 2023.11.21. There were a few things here that were wrong (e.g. `writelines()` takes an `Iterable[bytes]`, not `bytes`, or inexplicable errors like importing several of the vfs classes from this very module), and those changes have been dropped manually here.

typing: add type annotations to `mercurial.util.makelock()` This bubbles up into the `vfs` classes, so get this out of the way.

util: avoid a leaked file descriptor in `util.makelock()` exceptional case

typing: add type annotations to the `mercurial.util.filestat` class It's referenced in the `vfs` classes, so get this out of the way to help there. The `TypeVar` definition and its usage was copied from the existing `util.pyi` file.

vfs: do minor copyediting on comments and doc strings These were flagged by PyCharm, so clear them from the gutter.

vfs: simplify the `abstractvfs.rename()` implementation PyCharm was yapping about `util.rename()` not returning anything, because it is typed to return `None`, but the value was captured and returned after calling `_avoidambig()`. Instead, drop all of that, unconditionally rename, and then call `_avoidambig()` if appropriate. While we're here, convert the ersatz ternary operator into a modern one to help pytype. When a variable is initialized the old way, pytype tends to assign the type of the LHS of the `and`. In this case, that's a bool, and it will get confused that bool doesn't have a `stat` attribute once this method gets more type annotations. (Currently it thinks the `checkambig` arg is `Any`, so it doesn't care.)

vfs: use @abstractmethod instead of homebrewing abstract methods The latter confuses PyCharm after adding more type annotations when, for example, `abstractvfs.rename()` calls `_auditpath()`- the latter unconditionally raised an error, so PyCharm thought the code that came after is unreachable. It also tricked pytype into marking the return type as `Never`, which isn't available until Python 3.11 (outside of `typing_extensions`). This also avoid PyCharm warnings that the call to the superclass constructor was missed (it couldn't be called because it raised an error to prevent instantiation). The statichttprepo module needed to be given an override for one of the abstract methods, so that it can be instantiated. In `abstractvfs`, this method is only called by `rename()`, so I think we can leave this empty. We raise an error in case somebody accidentally calls it in the future- it would have raised this same error prior to this change. I couldn't wrangle `import-checker.py` into accepting importing `ABC` and `abstractmethod`- for each subsequent import, it reports something like: stdlib import "contextlib" follows local import: abc I suspect the problem is that near the `if fullname != '__future__'` check, if the module doesn't fall into the error case, `seenlocal` gets set to the module name. That causes it to be treated like a local module on the next iteration, even though it is in `stdlib_modules`.

vfs: modernize the detection of the main thread There weren't a lot of good choices when py27 was supported, but starting with py34, `threading.main_thread()` is available. This gets us away from an undocumented, internal symbol, and drops a pytype suppression statement. It is also apparently no longer reliable after a process fork.[1][2] [1] https://stackoverflow.com/a/23207116 [2] https://github.com/python/cpython/blob/v3.6.3/Lib/threading.py#L1334

store: fix a signature mismatch for a vfs subclass This was flagged by PyCharm. I'm not sure why pytype doesn't catch this- it's not excluded from the modules that are currently checked.

lfs: fix various signature mismatches for vfs subclasses These were flagged by PyCharm. I'm not sure why pytype doesn't catch these- only `hgext/lfs/__init__.py` in the lfs extension is excluded from being checked. I'm not sure if the `*insidef` arg to `join()` was meant as an internal convencience, because I see another class that gets flagged for the same signature problem (to be fixed next). But I don't feel bold enough to make this an internal function, and provide a simplified public `join()` on the `vfs` classes. That can still be done later, if desired. For now, process the additional args and pass them along, even though there don't appear to be any current callers that provide extra args to these classes. We need all of the subclasses to agree on the signature, or they won't be considered to implement the `Vfs` protocol being developed. While we're copy/pasting from the base class, bring the type annotations along for the ride.

util: add a comment to suppress a PyCharm warning about a PEP 8 violation Slowly trying to get rid of silly warnings, so that real problems aren't hidden.

keepalive: fix a signature mismatch for a http.client.HTTPResponse subclass Also flagged by PyCharm. This is checked by pytype too, so I'm not sure why it misses this. I verified in py36 that this argument is documented for the function, so maybe this is py2 legacy.

cbor: drop a duplicate dictionary initialization entry Flagged by PyCharm.

profiling: document the py-spy value for `profiling.type` The feature was not visible otherwise.

tests: enable pytype checking on `mercurial/unionrepo.py`

unionrepo: fix mismatches with revlog classes This is a subset of cfd30df0f8e4, applied to `unionrepository`. There are none of the `write()` method overrides here, like `bundlerepository`. With these changes, pytype flags the `unionrevlog` constructor: File "/mnt/c/Users/Matt/hg/mercurial/unionrepo.py", line 55, in __init__: No attribute '_revlog' on mercurial.changelog.changelog [attribute-error] Called from (traceback): line 207, in __init__ File "/mnt/c/Users/Matt/hg/mercurial/unionrepo.py", line 55, in __init__: No attribute '_revlog' on mercurial.revlog.revlog [attribute-error] Called from (traceback): line 232, in __init__ But it turns out that both `changelog.changelog` and `revlog.revlog` do have a `target` attribute, so they wouldn't trip over this. It seems weird that the second caller to be flagged is passing the private `_revlog`, but maybe that's how it needs to be.

typing: make `unionrepository` subclass `localrepository` while type checking This is the same change as 9d4ad05bc91c made for `bundlerepository`, for the same reasons. Also, add a comment here to suppress the PyCharm warning that the superclass constructor is not called, that is new now that there's a simulated superclass. That lack of a call is by design- `makeunionrepository()` does magic that PyCharm isn't aware of. But PyCharm has been better at catching problems than pytype in a lot of cases, so I'd like to reduce the bogus things it flags, to make the real issues stand out.

tests: enable pytype checking on `mercurial/bundlerepo.py`

revlog: make `clearcaches()` signature consistent with ManifestRevlog I'm not sure if this a newly added bug, because of using a different version of pytype, or if the recent work around avoiding the zope interface types in the type checking phase (see 5eb98ea78fd7 and friends)... but pytype 2023.11.21 started flagging this series since it was last pushed ~6 weeks ago: File "/mnt/c/Users/Matt/hg/mercurial/bundlerepo.py", line 204, in <module>: Overriding method signature mismatch [signature-mismatch] Base signature: 'def mercurial.manifest.ManifestRevlog.clearcaches(self, clear_persisted_data: Any = ...) -> None'. Subclass signature: 'def mercurial.revlog.revlog.clearcaches(self) -> None'. Not enough positional parameters in overriding method. Maybe the multiple inheritance in `bundlerepo.bundlemanifest` is bad, but it seems like a `ManifestRevlog` is-a `revlog`, even though the class hierarchy isn't coded that way. Additionally, it looks like `revlog.clearcaches()` is dealing with some persistent data, so maybe this is useful to have there anyway. Also sprinkle some trivial type hints on the method, because there are other `clearcaches()` definitions in the codebase with these hints, and I don't feel like waiting for another pytype run to see if it cares that specifically about the signature matching.

bundlerepo: fix mismatches with repository and revlog classes Both pytype and PyCharm complained that `write()` and `_write()` in the bundlephasecache class aren't proper overrides- indeed they seem to be missing an argument that the base class has. PyCharm and pytype also complained that the `revlog.revlog` class doesn't have a `_chunk()` method. That looks like it was moved from revlog to `_InnerRevlog` back in e8ad6d8de8b8, and wasn't caught because this module wasn't type checked. However, I couldn't figure out a syntax with `revlog.revlog._inner._chunk(self, rev)`, as it complained about passing too many args. `bundlerevlog._rawtext()` uses this `super(...)` style to call the super class, so hopefully that works, even with the wonky dynamic subclassing. The revlog class needed the `_InnerRevlog` field typed because it isn't set in the constructor. Finally, the vfs type hints look broken. This initially failed with: File "/mnt/c/Users/Matt/hg/mercurial/bundlerepo.py", line 65, in __init__: Function readonlyvfs.__init__ was called with the wrong arguments [wrong-arg-types] Expected: (self, vfs: mercurial.vfs.vfs) Actually passed: (self, vfs: Callable) Called from (traceback): line 232, in dirlog line 214, in __init__ I don't see a raw Callable, but I tried changing some of the vfs args to be typed as `vfsmod.abstractvfs`, but that class doesn't have `options`, so it failed elsewhere. `readonlyvfs` isn't a subclass of `vfs` (it's a subclass of `abstractvfs`), so I'm not sure how to handle that. It would be a shame to have to make a union of vfs subclasses (but not all of them have `options` either).

typing: make `bundlerepository` subclass `localrepository` while type checking Currently, `mercurial/bundlerepo.py` is excluded from pytype, mostly because it complains that various `ui` and `vfs` fields in `localrepository` are missing. (`bundlerepository` dynamically subclasses `localrepository` when it is instantiated, so it works at runtime.) This makes that class hierarchy known to pytype. Having a protocol for `Repository` is probably the right thing to do, but that will be a lot of work and this still reflects the class at runtime. Subclassing also has the benefit of making sure any method overrides have a matching signature, so maybe this is a situation where we do both of these things. (I'm not sure how clear the diagnostics are if a class *almost* implements a protocol, but is missing a method argument or similar.) The subclassing is not done outside of type checking runs to avoid any side effects on already complex code.

rust: bump rust-cpython version to 0.7.2 This version supports Python 3.12 while 0.7.1 did not.

rust: add Vfs trait This will allow for the use of multiple vfs like in the Python implementation, as well as hiding the details of the upcoming Python vfs wrapper to hg-core.

rust: use new revlog configs in all revlog opening code This centralizes the more complex logic needed for the upcoming code and creates stronger APIs with fewer booleans. We also reuse `RevlogType` where needed.

rust-revlog: don't try to open the data file if the index is empty This will cover the case where the data file is not present.

rust-revlog: add revlog-specific config objects These will be used by the upcoming Rust `InnerRevlog` to better centralize config information that is relevant to revlogs.

typing: add `from __future__ import annotations` to remaining source files Most of these look newer than when the original imports referenced in the previous commit were dropped, so these weren't covered by the backout. These were found with: hg files mercurial hgext hgext3rd -I '**.py' -X '**/thirdparty' \ | xargs grep -L 'from __future__ import annotations' All of the `__init__.py` files that finds are empty, so those were ignored and the rest manually edited.

typing: add `from __future__ import annotations` to most files Now that py36 is no longer supported, we can postpone annotation evaluation. This means that the quoting is usually optional (for things imported under the guard of `if typing.TYPE_CHECKING:` to avoid circular imports), and there's less overhead on startup[1]. There may be some missing here. I backed out 6000f5b25c9b (which removed the `from __future__ import ...` that was supporting py2), reverted the changes in `contrib/`, `doc/`, and `tests/`, and then ran: $ hg status -n --change . | \ xargs sed -i -e 's/from __future__ import .*$/from __future__ import annotations/' There were some minor tweaks needed when reviewing (mostly making the spacing around the import consistent, and `mercurial/testing/__init__.py` had a multiline import that wasn't fully rewritten. [1] https://docs.python.org/3/whatsnew/3.7.html#pep-563-postponed-evaluation-of-annotations

format: add many "missing" comma Black was not adding them until the next changeset introduced a bunch of `from __future__ import annotations` to most file. Since it make the next changeset hard to read we introduce them in advance.

typing: simplify archive.gz writing and drop a few pytype suppressions I was waiting until 3.8 to use `Literal` to fix this, but there's also the ":" and "|" characters that are passed along here, meant only for the non-gz archive types. But manipulating what the local caller passes is silly- we know we're writing, so just open it for writing. As an added bonus, PyCharm stops flagging the call too (since it doesn't know about pytype suppression comments).

typing: explicitly set the return type of `_InnerRevLog.raw_text()` Somewhere between cd72a88c5599 and 2fd44b3dcc33, pytype changed the return type from `Tuple[_T1, Any, bool]` to `Any`. Both are wrong. `mdiff.patches()` is an alias for `mpatch.patches()`, which is selected via module policy (and breaks the ability to infer the types). However, `cext`, `cffi`, and `pure` implementations all agree it returns bytes.

typing: add explicit hints for recent pytype regressions Somewhere between 454feddab720 and cd72a88c5599, pytype changed how it inferred the return type in `extdiff.py` from Tuple[Any, List[Tuple[bytes, Any, os.stat_result]]] to Tuple[Any, List[nothing]] It also changed the return type in `archival.py` from `Any` to `NoReturn`. Fix those up, and also the obvious parameter types while we're here.

revlog: use the method to check if the revlog is being written to This was probably fine, but it could become not fine at some point.

revlog: add an early return for getting sidedata No point in trying to fetch sidedata if there isn't a sidedata file.

revlog: simplify rawtext return value We're always returning a tuple even though only the raw text is being used, and we're rebuilding another tuple again higher. As a bonus, this will remove one tuple creation and deletion per `raw_text` call, hence fewer gc calls, etc.

revlog: cleanup some outdated docstrings

rust-inner-revlog: always inline `get_entry` This is a very hot function.

rust-inner-revlog: derive Debug for IndexHeaderFlags

rust-inner-revlog: drop some outdated comment

rust-config: add more ways of reading the config These will be needed for future patches of this series to interpret more complex/different config values.

util: make buffer readonly There is no use of writable buffers anywhere in the code, and this helps us make sure we don't get into unsound territory when sharing memory with Rust. This `toreadonly` method was not available in Python 3.6, but we dropped the support for it earlier that week, so no need for any compatibility code.

setup: avoid the deprecated `distutils.spawn.find_executable` I noticed this was flagged with `DeprecationWarning` in py3.12 with `setuptools` 74.1.2, and it suggested `shutil.which()` instead. The signatures aren't the same, but the additional `mode` argument in the middle of the latter defaults to `os.F_OK | os.X_OK`, which maintains the same semantics.

setup: drop the hack to disable linker warning 4197 on Windows I don't see this when building on Windows with py3.8 or py3.12, so either the code was fixed, or (more likely) the compiler stopped warning about it some time after VS 2008. If we do have to put this back, it would probably be better to put a `#pragma` in a header file somewhere, and avoid `setuptools` technical debt.

ci: also offer to test 3.12 with rust The rust-cpython binding got 3.12 support very recently, it is worse keeping on a tighter watch.

ci: add the option to test more Python versions It seems like a good idea to be able to test the lowest version we support. And there have been enougth issue with 3.12 that we need to be able to make sur we do not break it. We should probably get a matrix setup for more version and flavor, but that is a simple and efficient start.

censor: document the censor.policy option (issue6909) Censor is not marked as experimental and should be documented I am not doing this on stable because the help markup change it is using seems more suitable for default.

help: add :config-doc:`section.key` shorthand to insert documentation The config items defined in the configitems.toml file can already hold their documentation. Having some way to automatically insert it was a long standing low hanging fruit. So I did a first implementation on that. It fairly simple, but it open the door to more. It will be used in the next changeset.

bzr: attempt to stabilize the test The test has flakyness where the order of a few commit swap. This is an attempt at avoiding that.

branching: merge with stable

profiling: use "stat" profiler to profile individual request The ls profiler no longer works for that. As the lsprof profiler is not default and not great is general, lets side step the issue for now.

profiling: improve 3.12 error message for calling lsprof twice Python 3.12 prevent lsprof to be enabled if it is already enabled. This break the use of lsprof in `hg serve` as both the initial `serve` command and the request serving want to profile. The "stat" profiler (the default) does not have this problem, so we focus on improving the error message for now.

test: display server error log in test-profile.t This will help us to catch error with Python 3.12

archive: defer opening the output until a file is matched Before, if no file is matched, an error is thrown, but the archive is created anyway. When using hgweb, an error 500 is returned as the response body already exists when the error is seen. Afterwards, the archive is created before the first match is emitted. If no match is found, no archive is created. This is more consistent behavior as an empty archive is not a representable in all output formats, e.g. tar archives.

run-tests: add color to the progress output More color is useful to me.

python-compat: drop support for Python3.6 and 3.7 As discussed on the mailing list¹, these are old version that seems safe to drop. Python 3.8 comes with various improvement especially regarding typing capabilities. [1] https://lists.mercurial-scm.org/pipermail/mercurial-devel/2024-July/297998.html

ci: drop path manipulation that we do not need anymore The CI image has a squarer setup now.

brancing: merge stable into default

setup: handle removal of old MSVC compiler from setuptools 65.0 (issue6910) It was removed a few years ago[1]. When trying to reproduce locally using a clean py3.12 as called out in the bug report, `setuptools` wasn't installed at all, and needed a `pip install` to fix a `ModuleNotFoundError` when building locally. Maybe that needs to be in the requirements clause now. It looks like this "private" module was added in setuptools 48.0.[2] I can't find a changelog of what version was included in which version of python, and the changelog for pip has a huge gap between when it called out 67.6.1 in `pip` 23.1 (2023-04-15), and 41.4.0 in `pip` 19.3 (2019-10-14).[3] So, we'll just add to the existing code instead of replacing it, for safety. [1] https://github.com/pypa/setuptools/commit/cc017c77948737d131f683e0c25cd37bc639b8fc [2] https://github.com/pypa/setuptools/commit/d034a5ec7f707499139f90eb846b9e720923124c [3] https://pip.pypa.io/en/stable/news/

utils: accept bytearray arguments for escapestr

http: simplify

http: use urllib's cookie handler Split the logic for loading the cookies based on the configuration in a helper function and otherwise use the library implementation directly.

http: reuse Python's implementation of read/readline/readinto Since Python 3 already provides a working implementation of readline, there is no need for our own buffering implementation. Reduce the code to transfer accounting only.

debugwireproto: redo logging to also work for https

urllib2: redo response.readlines addition via class patching

typing: lock in new pytype gains from making revlog related classes typeable These were pretty clean changes in the pyi files from earlier in this series, so add them to the code to make it more understandable. There's one more trivial hint that can be added to the return of `mercurial.revlogutils.rewrite._filelog_from_filename()`, however it needs to be imported from '..' under the conditional of `typing.TYPE_CHECKING`, and that seems to confuse the import checker- possibly because there's already an import block from that level. (I would have expected a message about multiple import statements in this case, but got one about higher level imports should come first, no matter where I put the import statement.)

typing: add types to `revlog.revlogproblem` These attrs showed as `Any` after the previous commit made the class visible to pytype.

typing: make the revlog classes known to pytype These are the same changes as c1d7ac70980b and 45270e286bdc made to dirstate, for the same reasons.

typing: make the manifest classes known to pytype These are the same changes as c1d7ac70980b and 45270e286bdc made to dirstate, for the same reasons. The migration away from decorating the classes with `@interfaceutil.implementer` was started back in 3e9a660b074a, but missed one.

typing: make the filelog class known to pytype These are the same changes as c1d7ac70980b and 45270e286bdc made to dirstate, for the same reasons.

remotefilelog: adapt the `debugindex` command to past API changes Pytype was missing these problems because it's currently inferring the classes for `filelog` and `revlog` to be `Any`. When that's fixed, these were flagged, so fix these first. The `filelog` class used to subclass `revlog`, but that was changed back in 1541e1a8e87d (with most or all of the "lost" attributes being forwarded to the embedded `revlog` attribute at that time). These forwarded references were dropped over time, and this command has been broken at least as far back as 68282a7b29a7 when the `version` field was dropped. Most of the fixes were as simple as calling the accessor for the embedded `revlog` member, but the general delta feature detection was a bit more involved- I copied the detection for it from `mercurial.revlogutils.debug.debug_revlog()`.

typing: add type hints to the `opener` attributes and arguments of revlog When making revlog and filelog classes visible to pytype, it got confused quite a bit in `mercurial/revlogutils/rewrite.py`, thinking it had a plain `Callable`, and flagging additional methods on it like `join()` and `rename()`. I couldn't figure out how it reduced to that (and PyCharm flagged `opener` references as `Any`), but this makes it happy. So make this change before making the classes visible. The vfs class hierarchy is a bit wonky (e.g. `filteredvfs` is not a `vfs`), so this may need to be revisited with a Protocol class that covers all of the `vfs` classes. But for now, everything works.

remotefilelog: honor the `--format` arg of the `debugindex` command Flagged by PyCharm while investigating pytype spew. The other `**opts` above are already accessed as str. I've never used remotefilelog, and don't have a repo to test this on, so I'm trusting the nearby code.

merge: sort filemap only if requested by the caller The name `sorted` refers to a built-in function, which is always true, so the else branch of this if statement was dead code. Because, with this fix, the function can iterate over the dict items while yielding values, the dict should not change size while the generator is running. Because of that, it is required to re-introduce code that makes a caller copy the filemap before modification, which was removed in 3c783ff08d40cbaf36eb27ffe1d296718c0f1d77 (that changeset also introduced the filemap() method including the bug that’s being fixed by this changeset).

shelve: consistently convert exception to bytes via `stringutil.forcebytestr` The other two places in this module use this, and past experience shows that this method does a nicer job. I'm not sure why we're converting to bytes here- `KeyError` is built-in and will have str attrs, and `RepoLookupError` is a subclass of the built-in `Exception` class (not `errors.Error`, which is allegedly the baseclass for all Mercurial exceptions).

typing: add type hints to `mercurial.shelve` Pytype wasn't flagging anything here yet, but PyCharm was really unhappy about the usage of `state` objects being passed to various methods that accessed attrs on it, without any obvious attrs on the class because there's no contructor. Filling that out made PyCharm happy, and a few other things needed to be filled in to make that easier, so I made a pass over the whole file and filled in the trivial hints. The other repo, ui, context, matcher, and pats items can be filled in after the context and match modules are typed.

typing: lock in correct changes from pytype 2023.04.11 -> 2023.06.16 There were a handful of other changes to the pyi files generated when updating pytype locally (and jumping from python 3.8.0 to python 3.10.11), but they were not as clear (e.g. the embedded type in a list changing from `nothing` to `Any` or similar). These looked obviously correct, and agreed with PyCharm's thoughts on the signatures. Oddly, even though pytype starting inferring `obsutil._getfilteredreason()` as returning bytes, it (correctly) complained about the None path when it was typed that way. Instead, raise a ProgrammingError if an unhandled fate is calculated. (Currently, all possibilities are handled, so this isn't reachable unless another fate is added in the future.)

monotone: replace %s interpolation with appropriate numeric specifiers The length is an int, and the version is a float. Neither work with bytes on py3. This was noticed when looking at nearby code after updating pytype changed some signatures.

shelve: raise an error when loading a corrupt state file in an impossible case The old return statement was flagged by pytype 2023.06.16 running under python 3.10.11. No idea why it isn't caught in CI running the same pytype with py3.7. This function is only called by `unshelvecmd()` (which first checks that either `--abort` or `--continue` is specified), and `hgabortunshelve()` and `hgcontinueunshelve()`, which locally apply `--abort` or `--continue` respectively. Therefore, there is no other way to call this, and this error should never be seen, but pytype can't figure that out on its own. Given that the abort case clears the state, it seems reasonable to defensively code this and not make that a blanket `else` case, on the off chance a 3rd way of calling this appears in the future.

contrib: print the version of pytype used to do the type checking This will help with CI. I don't see a way to print the version of python that's running it. When I tried `head -n 1 $(which pytype)`, the CI run printed: #!/usr/bin/env bash Locally, that gives the path to the python interpreter in the venv, so IDK what's different.

typing: create an @overload of `phasecache` ctor to handle the copy case In `phasecache.copy()`, it calls `self.__class__(None, None, _load=False)`, but the constuctor is typed to take a non-None repository. For the `_load=False` case, all args are ignored (and the copy function itself populates the attrs on the new object), so this isn't an error. For the default `_load=True` case, it needs a non-None repository. This is the simplest way to handle that duality. The reason this wasn't being detected is because pytype is confused by the interface decorators on the `localrepository` class, and is inferring the whole class as `Any`. (See 3e9a660b074a or c1d7ac70980b) Therefore, the type hint of `localrepo.localrepository` here was also effectively `Any`, which disabled the type checking entirely. This is the first foray into using `typing_extensions` to unlock future typing features. I think this is safe and reasonable because 1) it is only imported in the type checking phase (so no need to vendor our own copy), and 2) pytype has its own copy of `typing_extensions` bundled with it, so no need to alter the test environment. When run with a version of python that supports the symbol(s) natively, `typing_extensions` simply re-exports from `typing`, so there shouldn't be any future headaches with this.

typing: declare the `_phasesets` member of `phasecache` to be `Optional` Something in this area got flagged while making the repository class visible to pytype (instead of being typed as `Any`). A None assignment to something not optional is wrong, and when I tried setting it to `{}` to keep it non-Optional, some tests failed. There are checks for the attr being None elsewhere, so this seems to have just been an oversight.

typing: hide the interface version of `dirstate` during type checking As noted in the previous commit, the `dirstate` type is still inferred as `Any` by pytype, including where it is used as a base class for the largefiles dirstate. That effectively disables most type checking. The problems fixed two commits ago were flagged by this change. I'm not at all clear what the benefit of the original type is, but that was what was used at runtime, so I don't want to change the largefiles base class to the raw class. Having both a lowercase and camelcase name for the same thing isn't great, but given that this trivially finds problems without worrying about which symbol clients may be using, and the non-raw type is useless to pytype anyway, I'm not going to worry about it.

dirstate: remove the interface decorator to help pytype This is the same change that was made for some of the manifest classes in 3e9a660b074a. Note that `dirstate` is still inferred as `Any`, but at least we have `DirState` with all of the expected attributes.

largefiles: sync up `largefilesdirstate` methods with `dirstate` base class As it currently stands, pytype infers the `dirstate` class (and anything else decorated with `@interfaceutil.implementer`) as `Any`. When that is worked around, it suddenly noticed that most of these methods don't exist in the `dirstate` class anymore. Since they only called into the missing methods and there's no test failures, we can assume these are never called, and they can be dropped. In addition, PyCharm flagged `set_tracked()` and `_ignore()` as not overriding a superclass method with the same arguments. The missing default parameter for the former was the obvious issue. I'm guessing that the latter was named wrong because while there is `_ignore()` in the base class, it takes no arguments and returns a matcher. The `_ignorefiles()` superclass method also takes no args, and returns a list of bytes. The `_ignorefileandline()` superclass method DOES take a file, but returns a tuple. Therefore, the closest match is `_dirignore()`, which takes a file AND returns a bool. No idea why this needs to be overridden though.

sparse: reliably avoid writing to store without a lock With the code as written before this patch we can still end up writing to store in `debugsparse`. Obviously we'll write to it if by accident a store requirement is modified, but more importantly we write to it if another concurrent transaction modifies the requirements file on disk. We can't rule this out since we're not holding the store lock, so it's better to explicitly pass a permission to write instead of inferring it based on file contents.

debugsparse: stop taking the store lock debugsparse is a workspace-only opperation, or it better be workspace-only. Let's make it to stop taking the store lock.

scmutils: read the requires file before writing to avoid unnecessary rewrite This lets us get away without the repo lock in situations where we need to write requirements, but we know we're not changing the store requirements.

localrepo: remove _readrequires function in favor of scmutil.readrequires

scmutil: add `readrequires` next to `writerequires` The code is copied from localrepo.py.

typing: correct a type hint in `mercurial.manifest` Obvious typo that was flagged by PyCharm.

typing: add hints to `mercurial.util.mktempcopy()` Might as well, now that the previous commit indicated what types are required.

typing: fix the hint for the `mode` argument of `platform.copymode()` The posix module is doing a bitwise AND with this and an integer, so it can't be bytes. The only caller that provides the argument is `util.mktempcopy()`, and pytype infers the type as Any, which explains why this wasn't caught.

largefiles: fix check that ensures that --all-largefiles is only used locally Previously, the command added in the test failed with “abort: --all-largefiles is incompatible with non-local destination existing_destination”. The reason for the buggy behavior was the use of hg.islocal(), which does “return true if repo (or path pointing to repo) is local” and, for local paths, assumes that the path is actually pointing to an existing repository and returns whether the path is not a regular file (in which case it assumes that it is a bundlerepo, which are considered non-local).

exchange: trivial simplification Both sides of the condition do essentially the same thing, except one with fastpath=True. No functional changes.

import: fix erroneous comparison of str with bytes

histedit: create state and acquire locks earlier This makes chistedit (histedit with curses UI) not write any files inside repo without wlock. It also makes sense to wrap the entire process of preparing commands inside the curses UI inside locks because we don't want anything else to touch wdir or repo during this time.

py3: use str literal instead of bytes literal

typing: fix type annotation

cffi: pass bytes instead of str to ffi.new("char[]", …) The type annotations seem to imply that the passed values are always already bytes, but they aren’t necessarily. Before Python 3.11, the documentation stated that bytes can be used to annotate arguments whose type is actually any of bytes, bytearray, or memoryview.

cffi: call bytes() instead of str() on CFFI buffer instances

cffi: pass C type and attribute names as str instead of bytes

py3: fix type of some elements of __all__ lists

manifest: deprecated readdelta and readfast These method should not have any user left.

manifest: use read_delta_new_entries in verify too This seems like the proper semantic for the usage.

manifest: use read_delta_new_entries in changegroup validate This new method have a well defined semantic and can be adjusted by narrow as it needs. This should prevent some unwanted filelog access when running validate on a server using narrow profile to restrict access.

manifest: add a read_delta_new_entries method This new method have a well defined semantic and can be adjusted by narrow as it needs.

manifest: use `read_delta_parents` when adjusting linkrev Let's use the more accurate API.

manifest: use the `read_delta_parents` method Let's use the more accurate API.

manifest: use `read_delta_parents` when adjusting linkrev in remotefile Let's use the more accurate API.

manifest: introduce a `read_delta_parents` method This new method have a clearer semantic and can be used by code that need this semantic. This should avoid bugs, allow for more targeted optimisation, and provide a clearer API. Users will be updated in subsequent changesets. This is also part of the wider effort to clarify and fix this API. one more method coming.

manifest: use `read_any_fast_delta` for tag rev cache computation This will have the benefit of using the fast path more often, and being (a bit) less buggy. See inline comment for details.

manifest: use `read_any_fast_delta` during shallow prefetch's We now have a better function with a clear semantic. This simplify the usage in the remotefilelog code.

manifest: use `read_any_fast_delta` during remotefilelog's repack We now have a better function with a clear semantic. This simplify the usage in the remotefilelog code.

manifest: use read_any_fast_delta in changectx The new API is clearer but also more expressive. It allow to detect case where we did return a full read and populated the associated cache. Saving time!

manifest: allow skipping valid_bases argument to `read_any_fast_delta` In some case it make sens to just want a delta. So we update the API to support this.

manifest: introduce a `read_any_fast_delta` method This method is a clearer semantic than `readbase` and `readfast` and will allow for more accurate optimization and usage. This is part of a wider series introducing such clearer method.

manifest: add many type annotations to the manifest module This help to clarify the API a bit, this caught various bug in the process and will help to catch more in the future. This also make large refactoring significantly simpler.

manifest: help pytype to understant `writesubtrees`'s `getnode` type Since we provide a default, the return of `_lazydirs.get` is cannot be None. We help pytype to understand that.

manifest: use explicit None checking in `_loaddifflazy` This helps pytype to understand what is going here with `v2` type.

manifest: use explicit None checking in `_loadlazy` This help pytype to understand what is going on with `v` type.

manifest: clear `_lazydirs` in place in `_loadalllazy` This avoid resetting the type of the dictionary in pytype eyes. This is consistent with the way the dictionary is cleared bits by bits in `_loadalllazy` Having more accurate code will help pytype. We do it in advance to help bisecting and avoid drowning them in the future type annotation noise.

manifest: use tuple for `delta` in `fastdelta` This make the list content consistent and will help type annotation.

manifest: expose a version of the Class without interface decorator The decorator confuse Pytype. Having the "raw" python class exposed will also helps pytype when it get replaced by a native implementation. At least until we start using `typing.Protocol` in the future.

pytype: stop ignoring manifest.py pytype no longer complains about the file contents.

manifest: align some vfs option access on the fact we might not have options This make the usage consistent with the other option. Caught by pytype.

manifest: adds some type things for manifestdict.added This appeases pytype.

manifest: type and fix unhexlify Some part of that function seems to date back from Python 2. It raise question about whether this function is useful or not, but let us just fix it for now. This was caught by pytype.

docker-pytype: use version v2.1 of the CI image It use a more recent pytype as far as I understand.

context: some gratuitous documentation improvement I wrote it as I was reading the code.

profiling: add a py-spy profiling backend The recommended way to use this backend is by setting the config `profiling.output` to point to a file because py-spy output is not human-readable.

copytracing: fix a bug in an edge case in metadata.compute_all_files_changes

rhg: ignore readonly FS error when saving dirstate The error is already ignored when the .hg directory is read-only, so this is only fair. (the python hg is silent on readonly fs, too)

commit: set whole manifest entries at once (node with its associated flags) Add a new function manifest.set that sets whole manifest entries at once, so the caller doesn't have to do two separate operations: m[p] = n m.set_flags(f) becomes: m.set(p, n, f) This obviously saves an extra lookup by path, and it also lets the underlying manifest implementation to be more efficient as it doesn't have to deal with partially-specified entries. It makes the interaction conceptually simpler, as well, since we don't have to go through an intermediate state of incorrect partially-written entry. (the real motivation for this change is an alternative manifest implementation where we batch pending writes, and dealing with fully defined entries makes the batching logic muchsimpler while avoiding slowdown due to alternating writes and reads)

typing: add type hints around the matcher for subrepo archiving Mostly this is meant to try to smoke out any other potential issues around the matcher, since these args were mostly previously treated as `Any`, and therefore checking wasn't done.

subrepo: drop the default value of None for the archive matcher This was flagged by pytype after adding hints to `match.subdirmatcher` that it takes a non-optional matcher. That matcher argument is used without a guard in the subdirmatcher constructor, so that's the correct restriction. I don't think this fixes a bug in practice because the only way these are invoked is either by a parent `hgsubrepo.archive()`, `archival.archive()`, or the largefiles override of these. The `hgsubrepo.archive()` case (and the largefiles override) uses what the caller provided, so the caller will eventually be `archival.archive()` (or the largfiles override) up the call chain. The `archival.archive()` method also has None for its matcher's default arg. However, the three callers of that (`commands.archive()`, `webcommands.archive()`, and `extdiff.snapshot()`) all provide a matcher argument, so the None case can never occur unless a 3rd party extension swaps it for None. Sadly, we can't make the argument on the `archival.archive()` non-optional because there is a kwarg prior to it. Even though the largefiles override of `archival.archive()` is provided a valid matcher, we duplicate the internal creation of the matcher that the original `archival.archive()` does for consistency. By eliminating an impossible to hit case, we can simplify some of the subrepo code too, by dropping unreachable code.

branching: merge stable into default Post 6.8.1 release.

Added signature for changeset 11a9e2fc0caf

Added tag 6.8.1 for changeset 11a9e2fc0caf

relnotes: add 6.8.1

rhg: expand user and environment variables in ignore includes This was reported by a user, and was a TODO long overdue.

utils: avoid using internal _imp.is_frozen() imp has been deprecated for a long time, and were removed in Python 3.12 . As a workaround, we started using the internal _imp. That is ugly and risky. It seems less risky to get the functionality in some other way. Here, we just inspect if 'origin' of the '__main__' module is set and 'frozen'. That seems to work and do the same, and might be better than using the internal _imp directly. This way of inspecting module attributes seems to work in some test cases, but it is a risky change. This level of importlib doesn't have much documentation, a complicated implementation, and we are dealing with some odd use cases.

utils: fix resourceutil use of deprecated importlib.resources Some importlib functionality was deprecated in 3.11 . The documentation on https://docs.python.org/3.12/library/importlib.resources.html recommends using the new .files() API that was introduced in 3.9.

cext: use sys.executable instead of deprecated Py_GetProgramFullPath Fix warning with Python 3.13: mercurial/cext/parsers.c: In function 'check_python_version': mercurial/cext/parsers.c:1243:30: warning: 'Py_GetProgramFullPath' is deprecated [-Wdeprecated-declarations] 1243 | Py_GetProgramFullPath()); | ^~~~~~~~~~~~~~~~~~~~~ In file included from /usr/include/python3.13/Python.h:119, from mercurial/cext/parsers.c:11: /usr/include/python3.13/pylifecycle.h:43:43: note: declared here 43 | Py_DEPRECATED(3.13) PyAPI_FUNC(wchar_t *) Py_GetProgramFullPath(void); | ^~~~~~~~~~~~~~~~~~~~~ At this point in time, the PyConfig struct memory has been released and the PyConfig API can't be used. https://docs.python.org/3.13/c-api/init.html#c.Py_GetProgramFullPath recommands using sys.executable instead. Let's assume that will work in all versions. It would perhaps be better to use PySys_GetObject, but I prefer to stay consistent with how the same function is retrieving sys.hexversion.

subrepoutil: pass re.sub 'count' argument by name Python 3.13 started warning: DeprecationWarning: 'count' is passed as positional argument

tests: pass re.MULTILINE to re.sub as 'flags' - not in 'count' position This bug was caught by the new Python 3.13 warning: DeprecationWarning: 'count' is passed as positional argument

tests: use packaging from setuptools instead of deprecated distutils When invoking StrictVersion in 3.12 we got: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. distutils is dead in the standard library, and we have to move towards using `setuptools` as general extern dependency. Instead of also requiring the extern `packaging`, we will just use the packaging that is vendored in setuptools.

tests: drop test-demandimport.py distutils test that failed with warnings The test would fail because warnings: /usr/lib/python3.11/site-packages/_distutils_hack/__init__.py:18: UserWarning: Distutils was imported before Setuptools, but importing Setuptools also replaces the `distutils` module in `sys.modules`. This may lead to undesirable behaviors or errors. To avoid these issues, avoid using distutils directly, ensure that setuptools is installed in the traditional way (e.g. not an editable install), and/or make sure that setuptools is always imported before distutils. warnings.warn( /usr/lib/python3.11/site-packages/_distutils_hack/__init__.py:33: UserWarning: Setuptools is replacing distutils. warnings.warn("Setuptools is replacing distutils.") The test for distutils.msvc9compiler comes from 2205d00b6d2b. But since then, distutils is going away, and this test must change somehow. It is unclear exactly how setuptools depended on msvc9compiler, but setuptools also moved forward, and this exact test no longer seems relevant. It thus seems like a fair solution to remove the test while keeping the demandimport blacklist of distutils.msvc9compiler.

utils: test coverage of makedate Explore the scenario from ae04af1ce78d to avoid future regressions. This was intended to give some coverage of the change in faccec1edc2c.

mmap: populate mapping in a background thread When possible, we populate the memory mapping in a second thread. The mmap population does not only read the data from disk to memory. It also actually fill the memory mapping between process memory address and the physical memory used by the file system cache containing the mmap'ed data. Doing so buy back the slowdown from pre-population when it matters. When most data is accessed, only a few page fault will occurs, while the background thread fill the memory controller. When few data is accessed, the non-blocking mmap won't have to wait for all data to be populated. Here is a few example of improvement seen in benchmark around unbundle and push: ### data-env-vars.name = netbeans-2018-08-01-zstd-sparse-revlog # benchmark.name = hg.command.unbundle # benchmark.variants.issue6528 = disabled # benchmark.variants.reuse-external-delta-parent = yes # benchmark.variants.revs = any-100-extra-rev before: 0.758101 after: 0.732129 (-3.43%, -0.03) ## data-env-vars.name = mozilla-try-2019-02-18-zstd-sparse-revlog before: 1.519941 after: 1.503473 (-1.08%, -0.02) ### data-env-vars.name = mozilla-try-2019-02-18-zstd-sparse-revlog # benchmark.name = hg.command.push # bin-env-vars.hg.flavor = default # benchmark.variants.issue6528 = disabled # benchmark.variants.protocol = ssh # benchmark.variants.reuse-external-delta-parent = yes # benchmark.variants.revs = any-1-extra-rev before: 4.801442 after: 4.695810 (-1.46%, -0.07) # benchmark.variants.revs = any-100-extra-rev before: 4.848596 after: 4.794075 (-1.12%, -0.05) # bin-env-vars.hg.flavor = rust # benchmark.variants.revs = any-1-extra-rev before: 4.818410 after: 4.700053 (-2.46%, -0.12)

pure: stringify builtin exception messages Builtin exceptions usually want strings, and display with a wierd b'' prefix if given bytes.

httppeer: avoid another bad reference before assignment warning This wasn't a problem, because `b''` from the `AttributeError` handler is in `bundle2.bundletypes`, so the following loop and conditional always run at least once. But PyCharm can't figure that out on its own, and it took a little exploring to figure out it wasn't a problem. The usage in `bundle2.writebundle` is to look it up in the map of bundle types, so it will break in a more obvious way in the unlikely event that the empty string is removed from the map in the future.

httppeer: move a variable to avoid a bad reference before assignment warning No actual bug here, because the conditional used to assign is the same as the conditional in the `finally` block that guards the reference.

httppeer: simplify two-way stream cleanup No need to conditionalize the cleanup if the filename is assigned outside the exception handler. I suppose `fd` leaks if `os.fdopen()` fails, but that was the case before too (and may trigger another exception in the `finally` block on Windows, when the file is still open).

rustfmt: update expected Rust edition In this case it doesn't change anything, but we've been using 2021 for a while now.

hghave: update expected rustfmt version We still use nightly, but have moved to a newer nightly after the last CI image upgrade in 74f1bf147a6d and 3876d4c6c79e.

rustfmt: apply formatting expected by newer nightly version This was missed in 3876d4c6c79ec5c71e8c51b876cc157e93a5eaac somehow.

tests: stop skipping `mercurial/pure/osutil.py` during pytype runs Not sure when the original issue(s) were fixed, but it works for me now.

largefiles: avoid a potentially undefined variable in exception case The `wlock` variable is used to release the lock in the `finally` block, so it would be undefined if `repo.wlock()` itself failed. Caught by pytype 2024.04.11 with py3.10.11.

typing: add trivial type hints to `mercurial.scmutil` There's still a lot to go, but there's a lot here already, so I tried to keep it to obvious/trivial things. I didn't bother with contexts, matchers, and revisions that can be `bytes | int | None`.

typing: narrow the scope of some recent disabled import warnings These comments were added in 39e2b2d062c1, but had the effect of changing the known type to `Any`, which cascaded through a few function signatures. Just ignore the import error instead.

demandimport: don't delay threading import A recent cpython change breaks demandimport by importing threading locally in importlib.util.LazyLoader.exec_module; add it (plus warnings and _weakrefset, which are imported by threading) to demandimport's ignore list. ``` Traceback (most recent call last): File "/usr/bin/hg", line 57, in <module> from mercurial import dispatch File "<frozen importlib._bootstrap>", line 1360, in _find_and_load File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 935, in _load_unlocked File "/usr/lib/python3/dist-packages/hgdemandimport/demandimportpy3.py", line 52, in exec_module super().exec_module(module) File "<frozen importlib.util>", line 257, in exec_module File "<frozen importlib._bootstrap>", line 1360, in _find_and_load File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 935, in _load_unlocked File "/usr/lib/python3/dist-packages/hgdemandimport/demandimportpy3.py", line 52, in exec_module super().exec_module(module) File "<frozen importlib.util>", line 267, in exec_module AttributeError: partially initialized module 'threading' has no attribute 'RLock' (most likely due to a circular import) ``` Ref: https://github.com/python/cpython/issues/117983 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1076449 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1076747

typing: induce pytype to use the standard `attr` instead of the vendored copy What was previously happening with the vendored copy was that pytype would stub out all(?) classes that were decorated with `@attr.s` as `Any`. After this, we get a ton of classes defined, and numerous fields and methods now have proper types.

typing: disable some pytype errors in `mercurial.store` These seem to be legitimate errors, since one of the two callers (`encodedstore.data_entries()`) was previously typed to generate `BaseStoreEntry`. However, that and the other caller only pass `RevlogStoreEntry` objects. I can't tell if this was a WIP or what, but don't want to get side tracked on this. So flag as a TODO for later.

linelog: correct the default value of `annotateresult.lines` This was flagged by pytype once it was tricked into using the standard `attr` package instead of the vendored copy.

phabricator: correct the default value of `phabhunk.corpus` There's only one caller to this constructor (which does provide this argument), and no direct assignments, so there's no runtime bug here. However, when pytype is tricked into using the standard `attr` package instead of the vendored version, it flags this because bytes is passed to the one constructor invocation. Tricking pytype into using the standard package will generate many more type hints, noteably around `@attr.s` decorated things.

rust-changelog: accessing the index The `Index` object is currently the one providing all DAG related algorithms, starting with simple ancestors iteration up to more advanced ones (ranges, common ancestors…). From pure Rust code, there was no way to access the changelog index for a given `Repository`, probably because `rhg` does not use any such algorithm yet.

typing: add type hints to `mercurial.policy` Mostly trivial, but this seems like the logical module to use to inject the hints from `cext`, `pure`, etc, given that this file has the fallback policy. This is a first step. There doesn't appear to be a predefined type for a module in py3.7, so those are omitted for now.

cext: correct the argument handling of `b85encode()` The type stub indicated that this argument is `Optional`, which implies None is allowed. I don't see in the documentation where that's the case for `i`[1], and trying it in `hg debugshell` resulted in the method failing with a TypeError. I guess it was typed as an `int` argument because the `p` format unit wasn't added until Python 3.3[2]. In any event, 2 clients in core (`pvec` and `obsolete`) call this with no argument supplied, and `mdiff` calls it with True. So I guess we've avoided the None arg case, and when no arg is supplied, it defaults to the 0 initialization of the `pad` variable in C. Since the `p` format unit accepts both `int` and None, as well as `bool`, I'm not bothering to bump the module version- this code is more permissive than it was, in addition to being more correct. Interestingly, when I first imported the `cext` and `pure` methods in the same manner as the previous commit, it dropped the `Optional` part of the argument type when generating `util.pyi`. No idea why. [1] https://docs.python.org/3/c-api/arg.html#numbers [2] https://docs.python.org/3/c-api/arg.html#other-objects

typing: add type hints to the `charencode` module Since this module is dynamically imported from either `mercurial.pure` or `mercurial.cext`, these hints aren't detected in `mercurial.encoding`, and need to be imported directly there during the type-checking phase. This keeps the runtime selection via the policy config in place, but allows pytype to see these as functions with proper signatures instead of just `Any`. We don't attempt to import the `mercurial.cext` version yet because there's no types stubs for that module, but this will get the ball rolling. I thought this would spill over into other modules from there, but the only two *.pyi files that changed were for `encoding` and `charencode`. Applying this to other dynamically selected modules will clean some things up in other files, so this is a start. I had originally redefined the functions in the type-checking block (like some of the `os.path` aliasing in `mercurial.util`), but this is better because we won't have another duplication of the definitions that may get out of date.

typing: explicitly type some `mercurial.util` eol code to avoid @overload Unlike the previous commit, this makes a material difference in the generated stub file- the `pycompat.identity()` aliases generated an @overload like this: @overload def fromnativeeol(a: _T0) -> _T0: ... ... which might fail to detect a bad argument, like str. This drops the @overload for the 3 related methods, so there's a single definition for each. The `typelib.BinaryIO_Proxy` is used for subclassing (the same as was done in 8147abc05794), so that it is a `BinaryIO` type during type checking, but still inherits `object` at runtime. That way, we don't need to implement unused abstract methods.

typing: avoid some useless @overload definitions in `mercurial.util` Apparently pytype considered the name as well as the type of each argument, and generates @overload definitions if they don't match. At best this is clutter, and can easily be removed.

dirstate: stringify a few exception messages Built in exceptions want str, and ProgrammingError converts bytes to str internally (because it subclasses RuntimeError).

typing: add type hints to `mercurial.verify._normpath()` Since 10db46e128d4, pytype almost figured this out, going from `Any` -> `_T0`, but the intent is obvious.

typing: add type hints to `i18n._msgcache` Since 10db46e128d4, pytype stopped inferring that the key is bytes.

typing: add type hints to `mercurial.dirstatemap` Somewhere since 10db46e128d4, pytype stopped being able to infer the type of the `identity` field. Fill in some obvious other hints along the way. These hints caused pytype to flag a missing attribute: File "/mnt/c/Users/Matt/hg/mercurial/dirstatemap.py", line 714, in _v1_map: No attribute 'stat' on mercurial.windows.cachestat [attribute-error] In Union[Any, mercurial.posix.cachestat, mercurial.windows.cachestat] File "/mnt/c/Users/Matt/hg/mercurial/dirstatemap.py", line 715, in _v1_map: No attribute 'stat' on mercurial.windows.cachestat [attribute-error] In Union[Any, mercurial.posix.cachestat, mercurial.windows.cachestat] In practice, the `identity` field is NOT replaced with None if it isn't cacheable, so it's probably safer to just add the field and set it to None, since that check is already in place on line 715.

typing: add type hints to `cmdutil.findrepo()` Since 10db46e128d4, pytype almost figured this out, going from `Any` -> `_T0`, but the intent is obvious.

typing: add some type hints to fastannotate that have decayed in the last year Somewhere since 10db46e128d4, `_knownopts` decayed to `set` for unknown reasons. Also, `annotateopts.default` changed from bytes to str. While that is correct, I noticed that PyCharm was flagging the member fields as undefined in `shortstr()`, so add those to keep it happy. (There are no complaints from pytype because that module is excluded, due to the missing reference to `linelog.copyfrom()` that I'm not sure how to fix.)

heptapod-ci: use new v2.1 image This is finally catching up to ~3 years of tech debt.

heptapod-ci: move version prints closer to the start This makes debugging a lot easier if anything is to go wrong, and shows output earlier.

pytype: only try the hacky way of finding PYTHON if not provided This allows us to work in more environments, like when using pyenv. This syntax is compatible with all POSIX shells.

dummysmtpd: fix EOF handling on newer versions of OpenSSL Explanations inline.

test-install: add new glob for the upgrade notice in newer versions of pip We only globbed for the old warning, newer versions of pip use a cleaner one.

rust: use `.cargo/config.toml` instead of `.cargo/config` This has been deprecated for a while now and we don't support Rust versions that only understand the old path.

rust: apply clippy lints They are at most harmless and at best make the codebase more readable and simpler.

rust: change minimum supported version everywhere applicable This will help users and downstream packaging.

rustfmt: format the codebase with nightly-2024-07-16 The CI has moved to a newer nightly, which slightly changes how it wraps comments (which is the very option we use nightly for).

hghave: update detection of black version to a newer minimum The CI has moved to version 23.3.0, which is the last one to support 3.7 at runtime.

black: format the codebase with 23.3.0 The CI has moved to 23.3.0, which is the last version that supports 3.7 at runtime, so we should honor this change. # skip-blame mass-reformating only

pytype: work around wrong ImportError flagging As documented in https://github.com/google/pytype/issues/163, newer versions of Pytype do not understand caught `ImportError`, so we temporarily ignore them where applicable.

zeroconf: fix boolean return value This was (wrongly) flagged by Pytype as being undefined since it doesn't seem to understand `try` blocks. However, the caller is expecting a boolean and the fix to appease Pytype is simple, so we do both.

Backout accidental publication of a large range of revisions I accidentally published 25e7f9dcad0f::bd1483fd7088, this is the inverse.

Latest image and pytype fix

dummysmtpd: fix EOF handling on newer versions of OpenSSL Explanations inline.

test-install: add new glob for the upgrade notice in newer versions of pip We only globbed for the old warning, newer versions of pip use a cleaner one.

Try the full CI run

WIP test new CI image

rust: use `.cargo/config.toml` instead of `.cargo/config` This has been deprecated for a while now and we don't support Rust versions that only understand the old path.

rust: apply clippy lints They are at most harmless and at best make the codebase more readable and simpler.

rustfmt: format the codebase with nightly-2024-07-16 The CI has moved to a newer nightly, which slightly changes how it wraps comments (which is the very option we use nightly for).

hghave: update detection of black version to a newer minimum The CI has moved to version 23.3.0, which is the last one to support 3.7 at runtime.

black: format the codebase with 23.3.0 The CI has moved to 23.3.0, which is the last version that supports 3.7 at runtime, so we should honor this change. # skip-blame mass-reformating only

pytype: work around wrong ImportError flagging As documented in https://github.com/google/pytype/issues/163, newer versions of Pytype do not understand caught `ImportError`, so we temporarily ignore them where applicable.

zeroconf: fix boolean return value This was (wrongly) flagged by Pytype as being undefined since it doesn't seem to understand `try` blocks. However, the caller is expecting a boolean and the fix to appease Pytype is simple, so we do both.

convert: fix various leaked file descriptors Some of these only leaked if an exception occurred between the open and close, but a lot of these leaked unconditionally. A type hint is added to `parsesplicemap` because otherwise this change caused pytype to change the return type from this to `Dict[nothing, nothing]`.

convert: stringify `shlex` class argument The documentation is handwavy, but typeshed says this should be `str`[1]. I'm not sure if this is the correct encoding (vs `fsencode` or "latin1" like the tokens returned by the proxy class). While we're here, we can add a few more type hints that would have caused pytype to flag the problem. [1] https://github.com/python/typeshed/blob/6a9b53e719a139c2d6b41cf265ed0990cf438192/stdlib/shlex.pyi#L51

typing: add trivial type hints to the convert extension's common modules This started as ensuring that the `encoding` and `orig_encoding` attributes has a type other than `Any`, so pytype can catch problems where it needs to be str for stdlib encoding and decoding. It turns out that adding the hint in `mercurial.encoding` is what was needed, but I picked a bunch of low hanging fruit while here. There's definitely more to do, and I see a problem where `shlex.shlex` is being fed bytes instead of str, but there are not enough type hints yet to make pytype notice.

convert: drop a duplicate implementation of `dateutil.makedate()` I noticed this because the signature generated by pytype recently changed to be less specific. When the method was introduced back in 337d728e644f, `util.makedate()` didn't take an optional timestamp arg. But now it does, and the methods are the same (except the `dateutil` version validates that the timestamp isn't a negative value). I left the old method in place in case anyone has custom convert code that monkey patches it.

revlog: use mmap by default is pre-population is available Using mmap has a great impact of memory usage on server, and a good impact on performance in multiple case. Now that we pre-populate memory mapping by default, there is case where it using mmap is slower. So we use it by default (if pre-population is available). Further work to reduce the performance impact of the pre-population will be done later. Some benchmark below (using the same setup as 522b4d729e89): As for 522b4d729e89 the impact on small repository like Mercurial or Pypy is tiny, ~1% best. However for large repositories we see some performance improvement without seeing the performance regression that we could have without pre-populate. ##### For netbeans ### data-env-vars.name = netbeans-2018-08-01-zstd-sparse-revlog ## benchmark.name = hg.command.log # bin-env-vars.hg.flavor = rust # benchmark.variants.limit-rev = 1 # benchmark.variants.patch = yes no-mmap: 0.171579 mmap: 0.166311 (-3.07%, -0.01) # bin-env-vars.hg.flavor = default no-mmap: 0.170716 mmap: 0.165218 (-3.22%, -0.01) # benchmark.variants.patch = no # benchmark.variants.rev = tip no-mmap: 0.140862 mmap: 0.137566 (-2.34%, -0.00) ## benchmark.name = hg.command.unbundle # bin-env-vars.hg.flavor = rust # benchmark.variants.issue6528 = disabled # benchmark.variants.reuse-external-delta-parent = yes # benchmark.variants.revs = any-1-extra-rev # benchmark.variants.source = unbundle no-mmap: 0.238038 mmap: 0.239912 no-populate: 0.cbd4c9 (+11.71%, +0.03) #### For Mozilla ### data-env-vars.name = mozilla-try-2019-02-18-ds2-pnm # benchmark.name = hg.command.log # bin-env-vars.hg.flavor = rust # bin-env-vars.hg.py-re2-module = default # benchmark.variants.limit-rev = 1 # benchmark.variants.patch = yes no-mmap: 0.258440 mmap: 0.237813 (-7.98%, -0.02) # benchmark.variants.limit-rev = 10 no-mmap: 1.235323 mmap: 1.213578 (-1.76%, -0.02) ## benchmark.name = hg.command.push # bin-env-vars.hg.flavor = rust # bin-env-vars.hg.py-re2-module = default # benchmark.variants.explicit-rev = none # benchmark.variants.issue6528 = disabled # benchmark.variants.protocol = ssh # benchmark.variants.reuse-external-delta-parent = yes # benchmark.variants.revs = any-1-extra-rev no-mmap: 4.790135 mmap: 4.668971 (-2.53%, -0.12) no-populate: 4.841141 (+1.06%, +0.05) ### data-env-vars.name = mozilla-try-2019-02-18-zstd-sparse-revlog ## benchmark.name = hg.command.log # bin-env-vars.hg.flavor = default # benchmark.variants.limit-rev = 1000 # benchmark.variants.rev = tip no-mmap: 0.206187 mmap: 0.197348 (-4.29%, -0.01) ## benchmark.name = hg.command.push # bin-env-vars.hg.flavor = default # benchmark.variants.explicit-rev = none # benchmark.variants.issue6528 = disabled # benchmark.variants.protocol = ssh # benchmark.variants.reuse-external-delta-parent = yes # benchmark.variants.revs = any-1-extra-rev no-mmap: 4.768259 mmap: 4.798632 no-populate: 4.953295 (+3.88%, +0.19) # benchmark.variants.revs = any-100-extra-rev no-mmap: 4.785946 mmap: 4.903618 no-populate: 5.014963 (+4.79%, +0.23) ## benchmark.name = hg.command.unbundle # bin-env-vars.hg.flavor = default # benchmark.variants.issue6528 = disabled # benchmark.variants.reuse-external-delta-parent = yes # benchmark.variants.revs = any-1-extra-rev # benchmark.variants.source = unbundle no-mmap: 1.400121 mmap: 1.423411 no-populate: 1.585365 (+13.23%, +0.19)

revlog: use an explicit config option to enable mmap usage for index We replace the `experimental.mmapindexthreshold` with two options: The `storage.revlog.mmap.index` is a boolean option to enable or disable the feature. The `storage.revlog.mmap.index:size-threshold` is a bytes option that control when we will be using mmap instead of plain reading.

mmap: populate the mapping by default Without pre-population, accessing all data through a mmap can result in many pagefault, reducing performance significantly. If the mmap is prepopulated, the performance can no longer get slower than a full read. (See benchmark number below) In some cases were very few data is read, prepopulating can be overkill and slower than populating on access (through page fault). So that behavior can be controlled when the caller can pre-determine the best behavior. (See benchmark number below) In addition, testing with populating in a secondary thread yield great result combining the best of each approach. This might be implemented in later changesets. In all cases, using mmap has a great effect on memory usage when many processes run in parallel on the same machine. ### Benchmarks # What did I run A couple of month back I ran a large benchmark campaign to assess the impact of various approach for using mmap with the revlog (and other files), it highlighted a few benchmarks that capture the impact of the changes well. So to validate this change I checked the following: - log command displaying various revisions (read the changelog index) - log command displaying the patch of listed revisions (read the changelog index, the manifest index and a few files indexes) - unbundling a few revisions (read and write changelog, manifest and few files indexes, and walk the graph to update some cache) - pushing a few revisions (read and write changelog, manifest and few files indexes, walk the graph to update some cache, performs various accesses locally and remotely during discovery) Benchmarks were run using the default module policy (c+py) and the rust one. No significant difference were found between the two implementation, so we will present result using the default policy (unless otherwise specified). I ran them on a few repositories : - mercurial: a "public changeset only" copy of mercurial from 2018-08-01 using zstd compression and sparse-revlog - pypy: a copy of pypy from 2018-08-01 using zstd compression and sparse-revlog - netbeans: a copy of netbeans from 2018-08-01 using zstd compression and sparse-revlog - mozilla-try: a copy of mozilla-try from 2019-02-18 using zstd compression and sparse-revlog - mozilla-try persistent-nodemap: Same as the above but with a persistent nodemap. Used for the log --patch benchmark only # Results For the smaller repositories (mercurial, pypy), the impact of mmap is almost imperceptible, other cost dominating the operation. The impact of prepopulating is undiscernible in the benchmark we ran. For larger repositories the benchmark support explanation given above: On netbeans, the log can be about 1% faster without repopulation (for a difference < 100ms) but unbundle becomes a bit slower, even when small. ### data-env-vars.name = netbeans-2018-08-01-zstd-sparse-revlog # benchmark.name = hg.command.unbundle # benchmark.variants.issue6528 = disabled # benchmark.variants.reuse-external-delta-parent = yes # benchmark.variants.revs = any-1-extra-rev # benchmark.variants.source = unbundle # benchmark.variants.verbosity = quiet with-populate: 0.240157 no-populate: 0.265087 (+10.38%, +0.02) # benchmark.variants.revs = any-100-extra-rev with-populate: 1.459518 no-populate: 1.481290 (+1.49%, +0.02) ## benchmark.name = hg.command.push # benchmark.variants.explicit-rev = none # benchmark.variants.issue6528 = disabled # benchmark.variants.protocol = ssh # benchmark.variants.reuse-external-delta-parent = yes # benchmark.variants.revs = any-1-extra-rev with-populate: 0.771919 no-populate: 0.792025 (+2.60%, +0.02) # benchmark.variants.revs = any-100-extra-rev with-populate: 1.459518 no-populate: 1.481290 (+1.49%, +0.02) For mozilla-try, the "slow down" from pre-populate for small `hg log` is more visible, but still small in absolute time. (using rust value for the persistent nodemap value to be relevant). ### data-env-vars.name = mozilla-try-2019-02-18-ds2-pnm # benchmark.name = hg.command.log # bin-env-vars.hg.flavor = rust # benchmark.variants.patch = yes # benchmark.variants.limit-rev = 1 with-populate: 0.237813 no-populate: 0.229452 (-3.52%, -0.01) # benchmark.variants.limit-rev = 10 # benchmark.variants.patch = yes with-populate: 1.213578 no-populate: 1.205189 ### data-env-vars.name = mozilla-try-2019-02-18-zstd-sparse-revlog # benchmark.variants.limit-rev = 1000 # benchmark.variants.patch = no # benchmark.variants.rev = tip with-populate: 0.198607 no-populate: 0.195038 (-1.80%, -0.00) However pre-populating provide a significant boost on more complex operations like unbundle or push: ### data-env-vars.name = mozilla-try-2019-02-18-zstd-sparse-revlog # benchmark.name = hg.command.push # benchmark.variants.explicit-rev = none # benchmark.variants.issue6528 = disabled # benchmark.variants.protocol = ssh # benchmark.variants.reuse-external-delta-parent = yes # benchmark.variants.revs = any-1-extra-rev with-populate: 4.798632 no-populate: 4.953295 (+3.22%, +0.15) # benchmark.variants.revs = any-100-extra-rev with-populate: 4.903618 no-populate: 5.014963 (+2.27%, +0.11) ## benchmark.name = hg.command.unbundle # benchmark.variants.revs = any-1-extra-rev with-populate: 1.423411 no-populate: 1.585365 (+11.38%, +0.16) # benchmark.variants.revs = any-100-extra-rev with-populate: 1.537909 no-populate: 1.688489 (+9.79%, +0.15)

win32mbcs: use str for encoding value This was reported to the TortoiseHg tracker as: https://foss.heptapod.net/mercurial/tortoisehg/thg/-/issues/5980 It doesn't look like we have any tests for this extension, but the explicit type hints are enough to convince pytype that the module level `_encoding` attr is str. The `encode()` and `decode()` methods are too complex to add type hints for them.

typing: add a trivial type hint to `mercurial/vfs.py` Since hg 3dbc7b1ecaba, pytype stopped seeing the return value of `rmtree` as `None`, and substituted `Any`.

typing: add a few trivial type hints to `mercurial/templater.py` Since hg 3dbc7b1ecaba, pytype started inferring that the second value in the tuple is `BinaryIO`, but still hasn't been able to figure out the rest of `open_template()`. We can be more precise.

typing: add a few type hints to `mercurial/revlog.py` Somewhere between hg 3dbc7b1ecaba and hg 8e3f6b5bf720, pytype stopped being able to infer the type for `_docket_file` and `compress()`. Lock those types in before they get lost.

typing: add a trivial type hint to `mercurial/posix.py` to avoid an @overload Since hg 3dbc7b1ecaba, pytype added an `@overload` for this function, without a type on the parameter. That's wrong, and undermines the hints on the non-trivial functions.

typing: add some trivial type hints to `mercurial/match.py` These were new methods since hg 3dbc7b1ecaba, but surprisingly pytype couldn't figure them out.

typing: add a type hint to `mercurial/hg.py` Somewhere between hg 3dbc7b1ecaba and hg 8e3f6b5bf720, the first value of the tuple changed from bytes to str. Let's lock this in, so that pytype flags it if someone mistakenly adds a tuple with bytes somewhere.

typing: restore `encoding.encoding` and `encoding.encodingmode` to bytes Somewhere between hg 3dbc7b1ecaba and hg 8e3f6b5bf720, pytype determined the signature of these fields changed from `bytes` to `Any`. Not sure why- the type of `environ` then and now is: `Union[WindowsEnviron, Dict[bytes, bytes], os._Environ[bytes]]` That said, PyCharm wasn't able to figure out the type of `environ`, and the `WindowsEnviron` class extends `MutableMapping` without specifying bytes for the key and value types in py3.9. But that's not changed in my setup, so I can't explain it.

typing: add some trivial type hints to `mercurial/bundlecaches.py` The function is meant for extensions, but it wasn't obvious what was expected without looking through the code. Also, pytype couldn't figure it out either.

typing: add some type hints for bundle2 capabilities Somewhere between hg 3dbc7b1ecaba and hg 8e3f6b5bf720, pytype determined the signature of `bundle20.capabilities` changed from `Dict[bytes, Tuple[bytes]]` to `Dict[bytes, Union[List[bytes], Tuple[bytes]]]`. First, I did try to simply be explicit about the previously inferred type, but it does seem to mix and match list/tuple now (e.g. in `writenewbundle()`). I tried changing the new list usage to tuple, but a couple of things complained, (and I think lists of one item are a little more clear to read anyway). So then I typed the dict value as `Sequence[bytes]`, which worked fine. But there's also a module level `capabilities` field, and when that's typed, pytype complains about `Sequence[bytes]` lacking `__add__`[1]. So I gave up, and just assigned it the type it wanted, with an alias. If somebody feels motivated to make the type consistent, it's simple enough to change the alias. The mutable default value to the constructor was removed to appease PyCharm's type checking on the field. (I didn't bother running the code through pytype prior to changing it, because we've previously made an effort to remove this pattern anyway.) I'm not sure why `getrepocaps()` has a default value for `role` that apparently raises an exception. It's just flagged for now so this series can land without risking additional problems. [1] https://foss.heptapod.net/mercurial/mercurial-devel/-/jobs/2466903

typing: add a few type hints to `mercurial/utils/urlutil.py` Somewhere between hg 3dbc7b1ecaba and hg 8e3f6b5bf720, `_pathsuboptions` changed from `Dict[bytes, Tuple[bytes, Any]]` to `Dict[bytes, Tuple[str, Any]]`, and it caught my attention from diffing the local *.pyi files. The change is correct based on the assertion, so let's get pytype to check for this instead of relying on the assertion alone.

typing: add type hints to `mercurial/utils/resourceutil.py` The `except` path requires byte args (because of the byte based manipulation in `_package_path()`), while the `else` case tolerates `AnyStr`. Pytype was unable to figure this out, and we should make sure the interface is the same for all environments.

copyright: update to 2024

branching: merge stable into default for 6.9 cycle

Added signature for changeset 11f41248595b

Added tag 6.8 for changeset 11f41248595b

relnotes: add 6.8

test-check: don't report distutils as a local import On python 3.12 this is wrongly reported as a local import. So we adjust the checker to avoid it.

Backed out changeset f28c52a9f7b4 This backout and the previous are due to a large performance regression detected in repositories with a lot of obsmarkers when performing a clone. A better fix will come along at the start of the next cycle.

Backed out changeset ff523675cd69

rust: use `cpython` 0.7.2 crate to add support for Python 3.12 This will give us more headroom until we can migrate to PyO3 some day.

revbranchcache: disable mmap access by default The revbranchcache can be truncated (if some part of it is detected as invalid). Using mmap on file we truncate is not an option at access to truncated part would result in a SIGBUS signal. So we disable the mmap by default until we fix this issue.

portability: fix build on Solaris-derived systemd Current Illumos and older Solaris require _XOPEN_SOURCE for msg_control. O_DIRECTORY doesn't exist on older systems either, so fallback to O_RDONLY. It's good enough as a repository will require both R and X permission anyway.

mmap: only use mmap to read revlog persistent nodemap if it is safe Cf `is_mmap_safe` docstring.

mmap: fix another instance of reverse mmap logic in persistent nodemap This fix the same kind of issue as 85d96517e650

mmap: only use mmap to read rev-branch-cache data if it is safe Cf `is_mmap_safe` docstring.

mmap: only use mmap to read revlog index if it is safe Cf `is_mmap_safe` docstring.

mmap: add a `is_mmap_safe` method to vfs This will be useful to safeguard mmap usage to void SIGBUS when repositories lives on a NFS drive.

branching: merge stable into default

Added signature for changeset 6454c117c6a4

Added tag 6.8rc0 for changeset 6454c117c6a4

branching: merge default into stable for 6.8rc0

relnotes: add 6.8rc0

branch3: use an explicitely experimental name for files Since this format is still experimental, we don't want to have to side-step the `branch3` name in case people do start using it before it's stable.

obsolete: simplify relevantmarker Drop duplicate assignment from a merge failure. Save one loop iteration by exploiting that pendingnodes will be seennodes after the first round anyway, so just pre-initialize the set accordingly. From Anton Shestakov's review on !867. Performance difference for my test case is in the noise.

exchange: improve computation of relevant markers for large repos Find the candidates for nodes with relevant markers by going over all markers instead of iterating over all nodes. Most nodes will not have markers anyway. Further optimize the code by allowing revsets as well, which reduces the materialization cost.

test: better glob some timing related line to avoid flakiness If we go over 10 seconds, the number of white space changes.

branching: merge stable into default

Added signature for changeset a1a011d4b148

Added tag 6.7.4 for changeset a1a011d4b148

relnotes: add 6.7.4 and warn about 6.7.{1,2,3}

inline-changelog: fix pending transaction visibility when splitting We move the name back to the expected name of `changelog.i.a`.

inline-changelog: fix a critical bug in write_pending that delete data Since a93e52f0b6ff we no longer use inline-revlog for the changelog. The goal there was to solve the lack of testing for the two variants (inline vs split) and reduce the complexity of the interaction with "diverted-write" on the changelog level. However many existing repository still have inline-changelog and we automatically move them to normal revlog as soon as we have the chances. Unfortunately This conversion is buggy and can result in the destruction of the changelog.i if hook triggers the "write pending" mechanism. The bugs comes from the "revlog splitting" logic and the "write_pending" logic stepping over each other. Ironically the change in a93e52f0b6ff aims at no longer having this kind of problem. This changesets fix this issue and add associated tests. Fixing this reveal that the transaction hooks end up not seeing the pending transaction content, because the name is not right ("changelog.i.s.a" instead of "changelog.i.s") we fix this in the next changeset.

bookmark: fix remote bookmark deletion when the push is raced Before this patch, running `hg push -B book` to push the `book` bookmark sideway at the same time as a commit making it moving forward might result in the removal of the bookmark remotely. After this changeset, the push can still be raced, but to remove deletion happens. This is progress.

hooks: add a prewlock and a prelock hooks This is useful for testing.

exchange: fix locking to actually be scoped The previous code was taking locks before entering with statements, so exception before the with statement would not release the lock (except for garbage collection). We need to move to a try except here because the logic is more complicated.

exchange: fix locking to actually be scoped The previous code was taking locks before entering with statements, so exception before the with statement would not release the lock (except for garbage collection).

narrow: add a test for linkrev computation done during widen This new tests show that the linkrev computed and sent by the server might end up being wrong during a widen operation.

obsolete: quote the feature name This makes it at least somewhat clearer that hg is talking about some specific feature and not just outdated code.

branching: merge stable into default

rust-status: sort the failed matches when printing them This was making the tests flaky after the recent patch¹ that opened up more of the code to the Rust-augmented status. [1] 865efc020c3355dca1cbaa35db80600009c01dd5

clonebundles: add missing newline to legacy response This seems to have been removed in 6.5 (likely by 60f9602b413e).

chistedit: change action for the correct item We have an experimental config histedit.later-commits-first from c820866c52f9, and when it's true, the order of commits in histedit UI is reversed, both in text mode and in curses mode. But before this patch key presses in curses mode would change histedit actions in the same old order, i.e. trying to edit the latest commit (which would be first now) would put "edit" action on the last commit in the list. This wasn't a cosmetic issue, histedit would actually proceed to edit the first commit in the list. Let's map rules to display items (hopefully now correctly).

dirstate: remove the python-side whitelist of allowed matchers This whitelist is too permissive because it allows matchers that contain disallowed ones deep inside, for example through `intersectionmatcher`. It is also too restrictive because it doesn't pass through some of the matchers we support, such as `patternmatcher`. It's also unnecessary because unsupported matchers raise `FallbackError` and we fall back anyway. Making this change makes more of the tests use rust code path, and therefore subtly change behavior. For example, rust status in largefiles repos seems to have strange behavior.

match: make `was_tampered_with` work recursively This is useful if we are to use it outside of Rust, when deciding whether or not to do some fast-path operation with a given matcher.

largefiles: mark more matchers as having been tampered with These happened to slip through the cracks earlier because they weren't caught by tests. Now that we're enabling rust fast path more widely these start breaking.

branching: merge stable into default

Added signature for changeset 028dc3f92dbd

Added tag 6.7.3 for changeset 028dc3f92dbd

relnotes: add 6.7.3

rust: blanket implementation of Graph for Graph references The need comes from the fact that `AncestorsIterator` and many Graph-related algorithms take ownership of the `Graph` they work with. This, in turn is due to them needing to accept the `Index` instances that are provided by the Python layers (that neither rhg nor `RHGitaly` use, of course): the fact that nowadays the Python layer holds an object that is itself implemented in Rust does not change the core problem that they cannot be tracked by the borrow checker. Even though it looks like cloning `Changelog` would be cheap, it seems hard to guarantee that on the long run. The object is already too rich for us to be comfortable with it, when using references is the most natural and guaranteed way of proceeding. The added test seems a bit superfleous, but it will act as a reminder that this feature is really useful until something in the Mercurial code base actually uses it.

rust-cpython: don't swallow the dirstate error message In case we do get a dirstate error, we want to get the full error message and not just an opaque `Dirstate error`.

dirstate-v2: check that root nodes are at the root before writing More explanations in the previous changeset.

dirstate-v2: add check of parent/child nodes being related when writing This stems from a corruption seen in a private repository. We're not sure of the source of the corruption, and it's very possible that we're seeing compounded effects of multiple writes on a corrupted dirstate. Adding this check is not expensive in itself and large writes of the dirstate are not common. This change does not catch this problem at the root node, the next one will.

admin-verify: expect a number of errors to be returned It's the responsibility of the check to handle errors, we only care about the total count to sum up the check's work. We use `admin::verify -c dirstate` to test this path at least somewhat.

admin-verify: fix error message handling `dirstate.verify` used to return tuples but does not anymore, it returns the pre-formatted error message, which is a nicer interface anyway.

admin-verify: pass p1 down to the dirstate function This was forgotten and can break with certain kinds of corruption.

Backed out changeset 3e0f86f09f26

Backed out changeset fc317bd5b637

re2: make errors quiet By default, the re2 library will output error on its own instead of keeping the error in an exception. This make re2 printing spurious error before fallback to the stdlib remodule that may accept the pattern or also fails to parse it and raise a proper error that will be handled by Mercurial. So we also pass an Option object that changes this default.

fold-or-prune-me: update proposal This does the same things but with a narrower wrapping.

subrepo: propagate non-default path on outgoing There was already a fix made in 5dbff89cf107 for pull and push commands. I did the same for the outgoing command. The problem I identified is that when the parent repository has multiple paths, the outgoing command was not respecting the parent path used and was always using the default path for subrepositories.

hgrc: search XDG_CONFIG_HOME on mac Searching for hgrc was special cased not to look through ~/.config/hg on Mac, but that’s unnecessary: Macs support it as do other unix based systems. There are plenty tools that use it there, e.g. git, and people expect it to work, e.g. "https://stackoverflow.com/questions/72499837/mercurial-on-macos-doesnt-read-config-hg-hgrc". Initial code introduced in 354020079723.

base-revsets: use an author that actually exercises a lot of changesets This was caught in my big find-and-replace: d4ba4d51f85f. The point of `base-revsets` is to give revsets that will give a good coverage of the repository. Using Pierre-Yves as the second largest committer (in terms of number of changesets) seems like a good idea.

match: simplify the rust-side file pattern kind parsing There's no need to add the ':' characters if we're simply pattern matching against constants next.

match: share code between includematcher and patternmatcher No need to have this duplication.

matchers: support patternmatcher in rust

match: avoid rust fast path if the matcher was tampered with Otherwise the fast path does not respect the modifications made by the extension (concretely largefiles, but other extensions can start using that too)

largefiles: track if a matcher was tampered with This is used to make sure rust fast path is not taken for the modified matchers.

branching: merge stable into default

tags-cache: directly perform a monimal walk for hgtagsfnodescache warming We do something narrower than the path retrieving data. So lets use dedicated code instead. This provides further useful speedup: ### data-env-vars.name = mozilla-try-2023-03-22-zstd-sparse-revlog # benchmark.name = hg.debug.debug-update-cache # bin-env-vars.hg.flavor = default # bin-env-vars.hg.py-re2-module = default # benchmark.variants.pre-state = warm before-this-series: 19.947581 skip-fnode-filter: 18.916804 (-5.17%, -1.03) use-rev-num: 17.493725 (-12.30%, -2.45) this-changesets: 15.919466 (-20.19%, -4.03)

tags-cache: directly operate on rev-num warming hgtagsfnodescache Not having to goes through nodeid speed up things notably. ### data-env-vars.name = mozilla-try-2023-03-22-zstd-sparse-revlog # benchmark.name = hg.debug.debug-update-cache # bin-env-vars.hg.flavor = default # bin-env-vars.hg.py-re2-module = default # benchmark.variants.pre-state = warm before-this-series: 19.947581 before-this-changes: 18.916804 (-5.17%, -1.03) this-changesets: 17.493725 (-12.30%, -2.45)

tags-cache: skip the filternode step if we are not going to use it When warming the hgtagsfnodescache, we don't need the actual result, so we can simply skip the part that "filter" fnode we read from the cache. So provide a quite visible speed up to the top level `hg debugupdatecache` function. ### data-env-vars.name = mozilla-try-2023-03-22-zstd-sparse-revlog # benchmark.name = hg.debug.debug-update-cache # bin-env-vars.hg.flavor = default # bin-env-vars.hg.py-re2-module = default # benchmark.variants.pre-state = warm before: 19.947581 after: 18.916804 (-5.17%, -1.03)

tags-cache: add a dedicated warm cache function to hgtagsfnodescache Having a dedicated API point will help to optimize that specific usage. Right doing a full phases weam takes a long time, even when the cache is already filled.

outgoing: add a simple fastpath when there is no common This further speed up case like `hg bundle --all` for larger repository. ### data-env-vars.name = mozilla-try-2023-03-22-zstd-sparse-revlog # benchmark.name = hg.command.bundle # bin-env-vars.hg.flavor = default # bin-env-vars.hg.py-re2-module = default # benchmark.variants.revs = all # benchmark.variants.type = none-streamv2 before: 316.749699 after: 311.165461 (-1.76%, -5.58) There is further work to be done in this area like not doing any outgoing computation in the stream case for example. however the recent changes already gives use a large win for a small amount of local work. ### benchmark.name = hg.command.bundle # bin-env-vars.hg.flavor = default # bin-env-vars.hg.py-re2-module = default # benchmark.variants.revs = all # benchmark.variants.type = none-streamv2 ## data-env-vars.name = mercurial-public-2024-03-22-zstd-sparse-revlog pre-%ln-change: 1.263859 the-%ln-change: 0.700229 (-44.60%, -0.56) prev-changeset: 0.496050 (-60.75%, -0.77) this-changeset: 0.495243 (-60.81%, -0.77) ## data-env-vars.name = tryton-public-2024-03-22-zstd-sparse-revlog pre-%ln-change: 2.975765 the-%ln-change: 1.870798 (-37.13%, -1.10) prev-changeset: 1.461583 (-50.88%, -1.51) this-changeset: 1.469185 (-50.63%, -1.51) ## data-env-vars.name = pypy-2024-03-22-zstd-sparse-revlog pre-%ln-change: 4.540080 the-%ln-change: 3.401700 (-25.07%, -1.14) prev-changeset: 2.915810 (-35.78%, -1.62) this-changeset: 2.911643 (-35.87%, -1.63) ## data-env-vars.name = heptapod-public-2024-03-25-zstd-sparse-revlog pre-%ln-change: 10.138396 the-%ln-change: 7.750458 (-23.55%, -2.39) prev-changeset: 6.665565 (-34.25%, -3.47) this-changeset: 6.672078 (-34.19%, -3.47) ## data-env-vars.name = mozilla-try-2023-03-22-zstd-sparse-revlog pre-%ln-change: 399.484481 the-%ln-change: 346.508952 (-13.26%, -52.98) prev-changeset: 316.749699 (-20.71%, -82.73) this-changeset: 311.165461 (-22.11%, -88.32)

outgoing: rework the handling of the `missingroots` case to be faster The previous implementation was slow, to the point it was taking a significant amount of `hg bundle --type none-streamv2` call. We rework the code to compute the same value much faster, making the operation disappear from the `hg bundle --type none-streamv2` profile. Someone would remark that producing a streamclone does not requires an `outgoing` object. However that is a matter for another day. There is other user of `missingroots` (non stream `hg bundle` call for example), and they will also benefit from this rework. We implement an old TODO in the process, directly computing the missing and common attribute as we have most element at hand already. ### benchmark.name = hg.command.bundle # bin-env-vars.hg.flavor = default # bin-env-vars.hg.py-re2-module = default # benchmark.variants.revs = all # benchmark.variants.type = none-streamv2 ## data-env-vars.name = heptapod-public-2024-03-25-zstd-sparse-revlog before: 7.750458 after: 6.665565 (-14.00%, -1.08) ## data-env-vars.name = mercurial-public-2024-03-22-zstd-sparse-revlog before: 0.700229 after: 0.496050 (-29.16%, -0.20) ## data-env-vars.name = mozilla-try-2023-03-22-zstd-sparse-revlog before: 346.508952 after: 316.749699 (-8.59%, -29.76) ## data-env-vars.name = pypy-2024-03-22-zstd-sparse-revlog before: 3.401700 after: 2.915810 (-14.28%, -0.49) ## data-env-vars.name = tryton-public-2024-03-22-zstd-sparse-revlog before: 1.870798 after: 1.461583 (-21.87%, -0.41) note: this whole `missingroots` of outgoing has a limited number of callers and could likely be replace by something simpler (like taking an explicit "missing_revs" set for example). However this is a wider change and we focus on a small impact, quick rework that does not change the API for now.

proxy-vfs: also proxy the `audit` attribute In the previous changeset, we had to do a little dance to access the useful `audit` attribute. We now provide a proper accessors to it. We don't update the code in `perf.py` because it has to remain compatible with older version of Mercurial. This will just be nicer in the future.

perf: clear vfs audit_cache before each run When generating a stream clone, we spend a large amount of time auditing path. Before this changes, the first run was warming the vfs cache for the other runs, leading to a large runtime difference and a "faulty" reported timing for the operation. We now clear this important cache between run to get a more realistic timing. Below are some example of median time change when clearing these cases. The maximum time for a run did not changed significantly. ### data-env-vars.name = mozilla-central-2018-08-01-zstd-sparse-revlog # benchmark.name = hg.perf.exchange.stream.generate # bin-env-vars.hg.flavor = default # bin-env-vars.hg.py-re2-module = default # benchmark.variants.version = latest no-clearing: 17.289905 cache-clearing: 21.587965 (+24.86%, +4.30) ## data-env-vars.name = mozilla-central-2024-03-22-zstd-sparse-revlog no-clearing: 32.670748 cache-clearing: 40.467095 (+23.86%, +7.80) ## data-env-vars.name = mozilla-try-2019-02-18-zstd-sparse-revlog no-clearing: 37.838858 cache-clearing: 46.072749 (+21.76%, +8.23) ## data-env-vars.name = mozilla-unified-2024-03-22-zstd-sparse-revlog no-clearing: 32.969395 cache-clearing: 39.646209 (+20.25%, +6.68) In addition, this significantly reduce the timing difference between the performance command, from the perf extensions and a `real `hg bundle` call producing a stream bundle. Some significant differences remain especially on the "mozilla-try" repositories, but they are now smaller. Note that some of that difference will actually not be attributable to the stream generation (like maybe phases or branch map computation). Below are some benchmarks done on a currently draft changeset fixing some unrelated slowness in `hg bundle` (34a78972af409d1ff37c29e60f6ca811ad1a457d) ### data-env-vars.name = mozilla-central-2018-08-01-zstd-sparse-revlog # bin-env-vars.hg.flavor = default # bin-env-vars.hg.py-re2-module = default hg.perf.exchange.stream.generate: 21.587965 hg.command.bundle: 24.301799 (+12.57%, +2.71) ## data-env-vars.name = mozilla-central-2024-03-22-zstd-sparse-revlog hg.perf.exchange.stream.generate: 40.467095 hg.command.bundle: 44.831317 (+10.78%, +4.36) ## data-env-vars.name = mozilla-unified-2024-03-22-zstd-sparse-revlog hg.perf.exchange.stream.generate: 39.646209 hg.command.bundle: 45.395258 (+14.50%, +5.75) ## data-env-vars.name = mozilla-try-2019-02-18-zstd-sparse-revlog hg.perf.exchange.stream.generate: 46.072749 hg.command.bundle: 55.882608 (+21.29%, +9.81) ## data-env-vars.name = mozilla-try-2023-03-22-zlib-general-delta hg.perf.exchange.stream.generate: 334.716708 hg.command.bundle: 377.856767 (+12.89%, +43.14) ## data-env-vars.name = mozilla-try-2023-03-22-zstd-sparse-revlog hg.perf.exchange.stream.generate: 302.972301 hg.command.bundle: 326.098755 (+7.63%, +23.13)

perf: start recording total time after warming The warming might be costly and this should not affect the "time profile" of the actual collection.

perf: run the gc before each run The python garbage collector is a large source of performance troubles, we run it right before the timed section to reduce the change for the gc to add noise to the benchmark.

perf: allow profiling of more than one run By default, we still profile the first run only. However profiling more run help to understand side effect from one run to the other. So we add an option to be able to do so.

profiler: flush after writing the profiler output Otherwise, the profiler output might only partially appears until the next flush of the buffer. Since profiling often happens for long operation, the next flush can be a long time away.

stream-clone: disable gc for the entry listing section for the v2 format This is similar to the change we did for the v3 format in 6e4c8366c5ce. The benchmark bellow show this gives us a notable gains, especially on larger repositories. ### benchmark.name = hg.perf.stream-locked-section # benchmark.name = hg.perf.stream-locked-section # bin-env-vars.hg.flavor = default # bin-env-vars.hg.py-re2-module = default # benchmark.variants.version = v2 ## data-env-vars.name = pypy-2018-08-01-zstd-sparse-revlog 5e931bf8707c: 0.503820 ~~~~~ 1106d1bf695e: 0.470078 (-6.70%, -0.03) ## data-env-vars.name = pypy-2024-03-22-zstd-sparse-revlog 5e931bf8707c: 0.535756 ~~~~~ 1106d1bf695e: 0.490249 (-8.49%, -0.05) ## data-env-vars.name = heptapod-public-2024-03-25-zstd-sparse-revlog 5e931bf8707c: 1.327041 ~~~~~ 1106d1bf695e: 1.174636 (-11.48%, -0.15) ## data-env-vars.name = netbeans-2018-08-01-zstd-sparse-revlog 5e931bf8707c: 2.439158 ~~~~~ 1106d1bf695e: 2.220515 (-8.96%, -0.22) ## data-env-vars.name = netbeans-2019-11-07-zstd-sparse-revlog 5e931bf8707c: 2.630794 ~~~~~ 1106d1bf695e: 2.261473 (-14.04%, -0.37) ## data-env-vars.name = mozilla-central-2018-08-01-zstd-sparse-revlog 5e931bf8707c: 5.769002 ~~~~~ 1106d1bf695e: 5.062000 (-12.26%, -0.71) ## data-env-vars.name = mozilla-try-2019-02-18-zstd-sparse-revlog 5e931bf8707c: 13.351750 ~~~~~ 1106d1bf695e: 12.346655 (-7.53%, -1.01) ## data-env-vars.name = mozilla-central-2024-03-22-zstd-sparse-revlog 5e931bf8707c: 10.772939 ~~~~~ 1106d1bf695e: 9.495407 (-11.86%, -1.28) ## data-env-vars.name = mozilla-unified-2024-03-22-zstd-sparse-revlog 5e931bf8707c: 10.864297 ~~~~~ 1106d1bf695e: 9.475597 (-12.78%, -1.39) ## data-env-vars.name = mozilla-try-2023-03-22-zstd-sparse-revlog 5e931bf8707c: 17.448335 ~~~~~ 1106d1bf695e: 16.027474 (-8.14%, -1.42)

phases: rework the logic of _pushdiscoveryphase to bound complexity This rework the various graph traversal in _pushdiscoveryphase to keep the complexity in check. This is done though a couple of things: - first, limiting the space we have to explore, for example, if we are not in publishing push, we don't need to consider remote draft roots that are also draft locally, as there is nothing to be moved there. - avoid unbounded descendant computation, and use the faster "rev between" computation. This provide a massive boost to performance when exchanging with repository with a massive amount of draft, like mozilla-try: ### data-env-vars.name = mozilla-try-2023-03-22-zstd-sparse-revlog # benchmark.name = hg.command.push # bin-env-vars.hg.flavor = default # bin-env-vars.hg.py-re2-module = default # benchmark.variants.explicit-rev = all-out-heads # benchmark.variants.issue6528 = disabled # benchmark.variants.protocol = ssh # benchmark.variants.reuse-external-delta-parent = default ## benchmark.variants.revs = any-1-extra-rev before: 20.346590 seconds after: 11.232059 seconds (-38.15%, -7.48 seconds) ## benchmark.variants.revs = any-100-extra-rev before: 24.752051 seconds after: 15.367412 seconds (-37.91%, -9.38 seconds) After this changes, the push operation is still quite too slow. Some of this can be attributed to general phases slowness (reading all the roots from disk for example) and other know slowness (not using persistent-nodemap, branchmap, tags, etc. We are also working on them, but with this series, phase discovery during push no longer showing up in profile and this is a pretty nice and bit low-hanging fruit out of the way. ### (same case as the above) # benchmark.variants.revs = any-1-extra-rev pre-%ln-change: 44.235070 this-changeset: 11.232059 seconds (-74.61%, -33.00 seconds) # benchmark.variants.revs = any-100-extra-rev pre-%ln-change: 49.234697 this-changeset: 15.367412 seconds (-68.79%, -33.87 seconds) Note that with this change, the `hg push` performance is now much closer to the `hg pull` performance, even it still lagging behind a bit. (and the overall performance are still too slow). ### data-env-vars.name = mozilla-try-2023-03-22-ds2-pnm # benchmark.variants.explicit-rev = all-out-heads # benchmark.variants.issue6528 = disabled # benchmark.variants.protocol = ssh # benchmark.variants.pulled-delta-reuse-policy = default # bin-env-vars.hg.flavor = rust ## benchmark.variants.revs = any-1-extra-rev hg.command.pull: 6.517450 hg.command.push: 11.219888 ## benchmark.variants.revs = any-100-extra-rev hg.command.pull: 10.160991 hg.command.push: 14.251107 ### data-env-vars.name = mozilla-try-2023-03-22-zstd-sparse-revlog # bin-env-vars.hg.py-re2-module = default # benchmark.variants.explicit-rev = all-out-heads # benchmark.variants.issue6528 = disabled # benchmark.variants.protocol = ssh # benchmark.variants.pulled-delta-reuse-policy = default ## bin-env-vars.hg.flavor = default ## benchmark.variants.revs = any-1-extra-rev hg.command.pull: 8.577772 hg.command.push: 11.232059 ## bin-env-vars.hg.flavor = default ## benchmark.variants.revs = any-100-extra-rev hg.command.pull: 13.152976 hg.command.push: 15.367412 ## bin-env-vars.hg.flavor = rust ## benchmark.variants.revs = any-1-extra-rev hg.command.pull: 8.731982 hg.command.push: 11.178751 ## bin-env-vars.hg.flavor = rust ## benchmark.variants.revs = any-100-extra-rev hg.command.pull: 13.184236 hg.command.push: 15.620843

phases: introduce a performant efficient way to access revision in a set This will be useful in the next changesets.

phases: use revision number in `_pushdiscoveryphase` We now reach our target checkpoint in terms of rev-num conversion. The `_pushdiscoveryphase` function is now performing graph computation based on revision number only. Avoiding repeated conversion from node-id to rev-num. See previous changeset updated `new_heads` for rationnal. Again, time saved in the 100 milliseconds order of magnitude for the mozilla-try benchmark I have been using. However, wow that the logic is done using revision number, we can look into having better logic in the next changesets, which will provide a much bigger speedup.

phases: move RemotePhasesSummary to revision number This continue our quest to align more logic on revision number instead of node-ids. The motivation is similar to the change to `new_heads` and `analyze_remote_phases` a few changeset earlier. Again, we take this as an opportunity to rename the class, and the attribute to the new naming scheme. This will highlight the need for code update for any code using it an expecting node-ids. Many of the rev-num → node-id conversion we had to introduce in the previous changesets can now be removed. More will be removed in the future as we continue to align code toward rev-num usage. time saved in the 100 milliseconds order of magnitude for the mozilla-try benchmark I have been using.

phases: stop using `repo.set` in `remotephasessummary` The `repository.set` create changectx on the fly, an expensive operation. Using `repo.revs` and a direct rev-num → node-id translation will be significantly faster. This is especially true as we prepare ourself to no longer do the rev-num → node-id transalation there. The speedup is a bit lost in the overall noisyness of the slow phase discovery algorithm, but it save a small amount of time in my benchmark.

phases: use revision number in analyze_remote_phases Same logic as the previous change to `new_heads`, see rationnal there. This avoids a small number of `nodes -> revs` conversion speeding thing up in the 100 milliseconds order of magnitude for the worses cases. However, the rest of the logic is noisy enough that it hardly matters for now.

phases: use revision number in new_heads All graph operations will be done using revision numbers, so passing nodes only means they will eventually get converted to revision numbers internally. As part of an effort to align the code on using revision number we make the `phases.newheads` function operated on revision number, taking them as input and using them in returns, instead of the node-id it used to consume and produce. This is part of multiple changesets effort to translate more part of the logic, but is done step by step to facilitate the identification of issue that might arise in mercurial core and extensions. To make the change simpler to handle for third party extensions, we also rename the function, using a more modern form. This will help detecting the different between the node-id version and the rev-num version. I also take this as an opportunity to add some comment about possible performance improvement for the future. They don't matter too much now, but they are worse exploring in a while.

phases: convert remote phase root to node while reading them This is currently a bit silly as we will convert them back to node right after, but that is an intermediate step before doing more disruptive changes.

phases: more compact error handling in analyzeremotephases using an intermediate variable result in more readable code, so let us use it.

push: rework the computation of fallbackheads to be correct The previous computation tried to be smart but ended up being wrong. This was caught by phase movement test while reworking the phase discovery logic to be faster. The previous logic was failing to catch case where the pushed set was not based on a common heads (i.e. when the discovery seemed to have "over discovered" content, outside the pushed set) In the following graph, `e` is a common head and we `hg push -r f`. We need to detect `c` as a fallback heads and we previous failed to do so:: e | d f |/ c | b | a The performance impact of the change seems minimal. On the most impacted repository at hand (mozilla-try), the slowdown seems mostly mixed in the overall noise `hg push` but seems to be in the hundred of milliseconds order of magnitude. When using rust, we seems to be a bit faster, probably because we leverage more accelaratd internals. I added a couple of performance related common for further investigation later on.

revset: stop serializing node when using "%ln" Turning hundred of thousand of node from node to hex and back can be slow… what about we stop doing it? In many case were we are using node id we should be using revision id. However this is not a good reason to have a stupidly slow implementation of "%ln". This caught my attention again because the phase discovery during push make an extensive use of "%ln" or huge set. In absolute, that phase discovery probably should use "%ld" and need to improves its algorithmic complexity, but improving "%ln" seems simple and long overdue. This greatly speeds up `hg push` on repository with many drafts. Here are some relevant poulpe benchmarks: ### data-env-vars.name = mozilla-try-2023-03-22-zstd-sparse-revlog # benchmark.name = hg.command.push # bin-env-vars.hg.flavor = default # bin-env-vars.hg.py-re2-module = default # benchmark.variants.explicit-rev = all-out-heads # benchmark.variants.issue6528 = disabled # benchmark.variants.protocol = ssh # benchmark.variants.reuse-external-delta-parent = default ## benchmark.variants.revs = any-1-extra-rev before: 44.235070 after: 20.416329 (-53.85%, -23.82) ## benchmark.variants.revs = any-100-extra-rev before: 49.234697 after: 26.519829 (-46.14%, -22.71) ### benchmark.name = hg.command.bundle # bin-env-vars.hg.flavor = default # bin-env-vars.hg.py-re2-module = default # benchmark.variants.revs = all # benchmark.variants.type = none-streamv2 ## data-env-vars.name = heptapod-public-2024-03-25-zstd-sparse-revlog before: 10.138396 after: 7.750458 (-23.55%, -2.39) ## data-env-vars.name = mercurial-public-2024-03-22-zstd-sparse-revlog before: 1.263859 after: 0.700229 (-44.60%, -0.56) ## data-env-vars.name = mozilla-try-2023-03-22-zstd-sparse-revlog before: 399.484481 after: 346.5089 (-13.26%, -52.98) ## data-env-vars.name = pypy-2024-03-22-zstd-sparse-revlog before: 4.540080 after: 3.401700 (-25.07%, -1.14) ## data-env-vars.name = tryton-public-2024-03-22-zstd-sparse-revlog before: 2.975765 after: 1.870798 (-37.13%, -1.10)

bundlespec: drop unused _bundlespecvariants dictionary Why do we have a `_bundlespecvariants`?

bundlespec: type the _bundlespeccontentopts dictionary If only we had a tool to detect the kind of stupid error we just fixed… ho wait.

bundlespec: fix the "streamv2" and "streamv3-exp" variant In c4aab3661f25, we broken this feature by adding unicode instead of bytes to the dictionary. On the other hand, this feature was never tested, so augment the tests to tests this.

wireprotoserver: ensure that output stream gets flushed on exception Previously flush was happening due to Python finalizer being run on `BufferedWriter`. With upgrade to Python 3.11 this started randomly failing. My guess is that the finalizer on the raw `FileIO` object may be running before the finalizer of `BufferedWriter` has a chance to run. At any rate, since we're not relying on finalizers in the happy case we should also not rely on them in case of exception.

match: strengthen visit_children_set invariant, Recursive means "all files" My previous interpretation of "Recursive" was too relaxed: I thought it instructed the caller to do something like this: > you can stop calling `visit_children_set` because you'll need to descend into > every directory recursively, but you should still check every file if it > matches or not Whereas the real instruction seems to be: > I guarantee that everything in this subtree matches, you can stop > querying the matcher for all files and dirs altogether. The evidence to support this: - the test actually passes with the stronger invariant, revealing no exceptions from this rule - the implementation of `visit_children_set` for `DifferenceMatcher` clearly relies on this requirement, so it must hold for that not to lead to bugs.

match: fix the rust-side bug in visit_children_set for rootfilesin matchers The fix is checked by `test_pattern_matcher_visit_children_set` test, which is what caught the bug in the first place, but also by an end-to-end test that I made for this purpose. Accept the new results of Cargo tests Many of these were already annotated with "FIXME", which is a good sign.

match: fix the "visitdir" method on "rootfilesin" matchers This fixes just the Python side, the fix for the rust side will follow shortly.

match: rename RootFiles to RootFilesIn for more consistency

match: small tweak to PatternMatcher.visit_children_set This makes it a bit more efficient (avoid a computation in case of early return), and in my opinion clearer.

matchers: fix the bug in rust PatternMatcher that made it cut off early This brings the rust output in line with the Python output.

tests: add an end-to-end test to show a bug in `visit_children_set` Concretely, `rootfilesin` is completely broken with respect to `visit_children_set` optimization.

tests: add tests and document expectations from visit_children_set in rust The tests this patch are adding have the form of formal spec in invariants::visit_children_set::holds, and then a series of checks that all examples must satisfy this formal spec. I tried to make the spec consistent with how this function is used and how it was originally conceived. This is in conflict with how it's documented in Rust. Some of the implementations also fail to implement this spec, which leads to bugs, in particular when complicated patterns are used with `hg status`.

tests: add a test that demonstrates a bug in rhg status pattern handling The bug is in [visit_children_set], will be elaborated on in follow-up changes.

bundle-spec: properly parse boolean configuration as boolean Before this changesets "v2;revbranchcache=no" would actually request the addition for a revbranchcache part as the non-empty string `"0"` is `True`

bundle-spec: properly identify changegroup-less bundle It is possible to produce a bundle without changegroup. For example if we want to only send phases or obsolescence information. However that lead to crash for command that identifies bundle content. So we fix that. The test will come in the next changesets, when we fix another bug preventing to generate such bundle by hand.

perf: create the temporary target next to the source in stream-consume See inline comment for rational.

setup: display return code information about failed `hg` call This help to understand what is going wrong when things goes wrong.

bundlespec: rationalize the way we specify stream bundle version Instead of having weird dedicated option for each version (v2, v3, etc) we reuse the same "stream" parameters. This is consistent with the ability to request a stream clone using "none-v2;stream=v2". This changeset introduce no user visible change, this is pure internal cleaning.

bundle: do no check the changegroup version if no changegroup is included We don't need to check the compatibility of something we will not use. In practice this was getting in the was of `streamv2` bundles on a narrow repository as the 'cg.version=02' value was rejected by this checks.

perf-stream-consume: use the source repository config when applying This might contains critical configuration for the benchmark, like enabling of extensions like narrow.

unbundle: move most of the logic on cmdutil to help debug::unbundle reuse This make sure `hg debug::unbundle` focus on the core logic.

postincoming: move to cmdutil This looks like a good place for it to live.

postincoming: avoid computing branchhead if no report will be posted This otherwise defeat some of the branch v3 optimization.

streamclone: stop listing files for entries that have no volatile files This will save a lot of python related time. This significantly boost performance. The following number comes from a large private repository using perf::stream-locked-section: base-line: 35.04 seconds prev-change: 24.51 seconds (-30%) prev-change: 20.88 seconds (-40%) prev-change: 14.22 seconds (-60%) this-change: 11.58 seconds (-67% from baseline; -18% from prev)

stream-clone: disable gc for the initial section for the v3 format The number of small container created turn Python in a gc-frenzy that seriously impact performance. This significantly boost performance. The following number comes from a large private repository using perf::stream-locked-section: base-line: 35.04 seconds prev-change: 24.51 seconds (-30%) prev-change: 20.88 seconds (-40%) this-change: 14.22 seconds (-60% from baseline; -31% from prev)

stream-clone: disable gc for `_entries_walk` duration The number of small container created turn Python in a gc-frenzy that seriously impact performance. This significantly boost performance. The following number comes from a large private repository using perf::stream-locked-section: base-line: 35.04 seconds prev-change: 24.51 seconds (-30%) this-change: 20.88 seconds (-40% from baseline; -15% from previous changes)

nocg: make the utility work are both a decorator and context manager In some case, the context manager version will be simpler.

stream-clone: stop getting the file size of all file in v3 The point of v3 is to do less work in the locked section. It was currently not the case. This significantly boost performance. The following number comes from a large private repository using perf::stream-locked-section: base-line: 35.03 seconds this-change: 24.50 seconds (-30%)

stream: in v3, skip the "size" fast path if the entries as some unknown size We are about to prefetch size during the lock less in the v3 case. So we need to avoid trying to use that prefetched size when it is not available. See next changeset for the motivation.

perf-stream-locked-section: advertise the right version key in the help As the v3 format is still experimental, its key is "v3-exp". The help text was not pointing that out. (we also fix `perf::stream-generate` in the process)

perf-stream-locked-section: fix the call to the v3 generator That generator simply return chunks so we should not assign the return to a tuple.

perf-stream-locked-section: actually use v1 generation when requested We were fetching a v1 generator but actually using the v2 function…

branching: merge stable into default

Added signature for changeset 803e61387e86

Added tag 6.7.2 for changeset 803e61387e86

relnotes: add 6.7.2

bundle2: make the "hgtagsfnodes" part advisory This bundle2 part is about helping the client to warms its cache. There is no reason for it to be mandatory. So we mark it advisory.

branching: merge stable into default

branchcache: allow to detect "pure topological case" for branchmap We don't rum this detection every time we run the branchcache, that would be costly. However we now do it when running `hg debugupdatecache`. This will help existing repository to benefit from the fastpath when possible.

branchcache: add a "pure topological head" fast path In a narrow but actually quick common case, all topological heads are all on the same branch and all open. In this case, computing the branch map is very simple. We can quickly detect situation where this situation will not change. So we update the V3 format to be able to express this situation and upgrade the update code to detect we remains in that mode. The branch cache is populated with the actual value when the branch map is accessed, but the update_disk method can do the update without needing to populate it.

branchcache: move the processing of the new data in a dedicated method In a future changeset, this will allow the V3 of the branch cache to use a fast path when possible.

branchcache: gather newly closed head in a dedicated set This is part of a series to more clearly split the update in two step. This will allow us to introduce a fast path during update in a future changeset.

branchcache: gather new obsolete revision in a set This is part of a series to more clearly split the update in two step. This will allow us to introduce a fast path during update in a future changeset.

branchcache: filter obsolete revisions sooner Since we won't do anything with the obsolete revisions, we can just ignore them sooner.

branchcache: skip entries that are topological heads in the on disk file In the majority of cases, topological heads are also branch heads. We have efficient way to get the topological heads and efficient way to retrieve their branch information. So there is little value in putting them in the branch cache file explicitly. On the contrary, writing them explicitly tend to create very large cache file that are inefficient to read and update. So the branch cache v3 format is no longer including them. This changeset focus on the format aspect and have no focus on the performance aspect. We will cover that later.

branchcache: simplify the branch rev cache test We don't need that many content dump and this gets in the way in change in access pattern (e.g. accessing revision in a different order change the order of branches in the "names" file). So we simplify this test in advance.

branchcache: store filtered hash and obsolete hash independently for V3 This will avoid the bug covered in tests/test-branches-obsolete.t when we stop storing all heads explicitly in V3.

branchcache: show the cache file content in test-branches-obsoletes.t This help to track the changes in format between v2 and v3.

branchcache: rework the `filteredhash` logic to be more generic We now have a more flexible `key_hashes` tuple. We duplicated various logic in the V2 and V3 version of the cache as the goal is to start changing the logic for V3 in the next few changesets.

filteredhash: rename the filteredhash function The new name is less ambiguous, as we are about to introduce an alternative function it seems like a good idea to have clearer name to distinct the two.

filteredhash: split the computation of revision sets The branch2's filteredhash combines the filtered revisions and the obsolete ones, this will creates issue for implicit reference to heads we want to introduce for the v3 of the branch cache format. So we isolate this logic for alternative use.

filteredhash: move the hashing in its own function This will help us to reuse this logic in variants of the hashes used for branch cache validation.

branchcache: cleanup the final key generation after update A lot of duplicated work seemed to be done, as we already update the tiprev and tipnode when needed right before. So we simplify that part to focus on the filtered hash. See inline comment for details.

branchcache: add more test for the logic around obsolescence and branch heads While working on branch-cache-v3, we noticed some ambiguity in the filtered+obsolete hash. However this was only caught by a rebase test by chance. It seems important to explicitly tests these cases.

branchcache-v3: use more explicit header line The key-value approach is clearer and gives more rooms to have the format evolve in a clear way. It also provides extension (like topic) simpler way to extend the validation scheme. This is just a small evolution, the V3 format is still a work in progress.

branchcache-v3: introduce a v3 format For now the format is the very same, however we will start changing it in future changesets.

branchcache: use an explicit class for the v2 version This prepare the introduction of an experimental v3 format version. In the process, we move the description of the format in that new class.

branchcache: add some blank line in a test This helps each section to stand out.

phases: update the phase set as we go during retract boundary Apparently iterating over the `changed_revs` dictionary is very expensive. On mozilla-try-2019-02-18, a perf::unbundle call with a 10 000 changesets bundle gives give use the following timing. e57d4b868a3e: 4.6 seconds ac1c75188440: 102.5 seconds prev-changeset: 30.0 seconds this-changeset: 4.6 seconds So, the performance regression is gone. Once again: thanks to marvelous Python!

phases: avoid a potentially costly dictionary interation in some case If we retract for the draft phase, there is not non-public item to be retracted and we can skip this part. This part is was apparently super costly thanks to Python. On mozilla-try-2019-02-18, a perf::unbundle call with a 10 000 changesets bundle gives give use the following timing. e57d4b868a3e: 4.6 seconds ac1c75188440: 102.5 seconds this-changeset: 30.0 seconds So we recovered about ⅔ of the regression, the next changeset will give us the rest back.

branching: merge stable into default

Added signature for changeset 2e6fde2ed01e

Added tag 6.7.1 for changeset 2e6fde2ed01e

relnotes: add 6.7.1

subrepo: fix normalizing paths with scheme After revision 0afe96e374a7, subrepo paths were normalized using posixpath.normpath and that resulted in ssh paths being wrongly converted from ssh://host/path to ssh:/host/path This fix applies the same logic used in urlutil.url to split the path scheme from the rest and only use posixpath.normpath to the string after scheme://

sshpeer: fix path when handling invalid url exception In 73ed1d13c0bf the code was refactored but the error handling seems to have been missed (or maybe the object shoud have implemented __bytes__)

delta-search: fix crash caused by unbound variable This code path was apparently not tested. This fixes a crash when cloning the Tryton repo.

branching: merge stable into default

Added signature for changeset c9ceb4f60256

Added tag 6.7 for changeset c9ceb4f60256

phases: avoid N² behavior in `advanceboundary` We allowed duplicated entries in the deque, which each entry could potentially insert all its ancestors. So advancing boundary for the full repository would mean each revision would walk all its ancestors, resulting in O(N²) iteration. For repository of any decent size, N² is quickly insane. We introduce a simple set to avoid this and get back to reasonable performance.

relnotes: add 6.7

admin-commands: move the chainsaw extension to the admin commands module Activating an extension is always a little bit of a chore and the long name, options and "chainsaw" bits are deterrent enough. This also allows us to help the discoverability for people looking for repo "administration" tools, with the widest semantic of "administration".

obsutil: sort metadata before comparing in geteffectflag() This is probably less important now that we dropped Python 2. We do still support Python 3.6 though, and the dictionaries aren't ordered there either (that was a big change that came with 3.7). Still, maybe it's a good idea to sort metadata explicitly.

tests: disable revlog compression in test-generaldelta.t (issue6867) The revlog compression makes a lot of numbers unstable. Since checking revlog compression is not the goal of this test, we disable the compression to get stable numbers. This should avoid wasting more time on this kind of changes in the future.

test-general-delta: actually test optimize-delta-parent-choice=no Since the configuration was not explicit, the case stopped testing what it intended to test when the default value changed.

test-chg: stabilize the log checking The "worker process exited" line have been making the CI flaky for a long time. Lets sort this out.

tests: fix test-patchbomb-tls.t instability The flakiness on chg is caused by a client that exit faster than the server output log. So actively wait for the server to issue the expected output (with a small timeout)

test-lock: use synchronisation file instead of sleep This will prevent the test to be flaky on load.

branchcache: use update_disk to refresh 'served' and 'served.hidden' The `update_disk` method is dedicated to this kind of usecase. Now that the writting patterns are more consistent, we can use it to warm these two important cache. I am dropping the first comment about "refreshing all the others" because it is false. If a branchmap already exist for "served", none of the subset will be updated.

branchcache: explictly update disk state only if no transaction exist If a transaction exist the `write_dirty` call will eventually be done and the state will be synched on disk. It is better to no interfer with that.

branchcache: do not use `__getitem__` in updatecache The `update_disk` method uses `updatecache` and the point of `update_disk` is to be able to do alternative processing to the one we do in `__getitem__`. So we calling `__getitem__` in `updatecache` defeat this purpose. Instead we do the equivalent explicitly to preserve the spirit of `update_disk` (that we will actually put to use soon, I promise)

branchcache: explicitly track inheritence "state" We move from a binary "dirty" flag to a three value "state": "clean", "inherited", "dirty". The "inherited" means that the branch cache is not only "clean", but it is a duplicate of its parent filter. If a branch cache is "inherited", we can non only skip writing its value on disk, but it is a good idea to delete any stale value on disk, as those will just waste time (and possibly induce bug) in the future. We only do this in the update related to transaction or explicit cache update (e.g `hg debugupdatecache`). Deleting the file when we simply detected a stall cache during a read only operation seems more dangerous. We rename `copy` to `inherit_for` to clarify we associate a stronger semantic to the operation.

branchcache: stop writing more branchcache file on disk than needed Before this change, we were unconditionally writing a branchmap file for the filter level passed to `update_disk`. This is actually counter productive if no update were needed for this filter level. In many case, the branch cache for a filter level is identical to its parent "subset" and it is better to simply keep the subset update and reuse it every time instead of having to do identical work for similar subset. So we change the `update_disk` method to only write a file when that filter level differ from its parent. This removes many cases where identical files were written, requiring multiple boring update in the test suite. The only notable changes is the change to `test-strip-branch-cache.t`, this case was checking a scenario that no longer reproduce the bug as writing less branchmap file result in less stalled cache on disk. Strictly speaking, we could create a more convoluted scenario that create a similar issue. However the next changeset would also cover that scenario so we directly updated that test case to a "no longer buggy" state.

branchcache: do not copy the `_dirty` flag If the inherited branch cache is dirty, it will be written on disk, and the super-set did not need to modify it, the on disk value for the subset will be re-useable as is. So the super set does not needs to write the very same content itself.

branchcache: explicitly assert that copy is always about inheritance This would catch cases where copy is used for something else if any existed.

branchcache: stop using `copy(…)` in `replace(…)` The `copy` method is mostly used for a filter level to inherit the branchmap from a subset. So we stop using (abusing) it in "replace" to ensure `copy` is used only for inheritance purposes. Since `replace` is a method of the BranchMapCache, it seems fine to do lower level operation there.

branchcache: change the _delayed flag to an explicit `_dirty` flag This is more consistent with the logic we use for other object and it open the way to a clearer management of the cache state. Now, cache are created clean, cache update mark them dirty, writing them on disk mark them clean again.

branchcache: write branchmap in subset inheritance order This way, we can guarantee a valid subset has been written before touching the branchmap of another filter. This is especially useful as we are bout to start deleting outdated branchmap file.

branchcache: do not accept "empty update" This currently does not happens and it will be simpler that is remains that way. If all update do something, we will be able to simply declare, in a later changesets, that all update to result in a dirty branchcache.

branchcache: avoid created a `None` filter repoview when writing The repoview class is not intended to be used for unfiltered repository.

stream-clone-tests: stop filtering non existent warning This filtering was introduced in 74c004a515bc, however there is already no warning in that changeset. So I guess the warnings existed when we the patch was created but the problem was solved in another changeset that 74c004a515bc, rebased on.

stream-clone-test: simplify case testing obsolescence There is only two important things in this test: - the number of file we send, to show we picked the obsstore. - the resulting state, to show we did alter things in the process. The rest are of the number are very fragile and consume a lot of time for little value when adjusting formats, caches, and protocol.

stream-clone-test: simplify the case testing phases There is only two important things in this test: - the number of file we send, to show we picked the phase roots. - the resulting phases, to show we did not modified them. The rest are of the number are very fragile and consume a lot of time for little value when adjusting formats, caches, and protocol.

stream-clone-test: simplify bookmark clone The important things to test here is the number of file included (to catch that the bookmark file was sent). So we keep that part non glob'ed but glob the rest. The glob'ed numbers are very fragile and consume a lot of time for little value when adjusting formats, caches, and protocol.

stream-clone-test: add a verify call to the "clone while changing" case It seems useful to very that the clone did not result in a corrupted copy.

stream-clone-test: add title to various test cases These case are fine as is, but as we are adding title to all the other as we simplify them, lets add title for all cases.

stream-clone-test: simplify testing of secret cloning restriction Here, we just want to check if the streaming clone is allowed and used or not. We do not care about the details of the clone itself.

stream-clone-test: simplify the background file closing test Here we just care about the fact the background file closing logic actually ran. We don't need to check the details of the cloning. The details of the output is very fragile and consume a lot of time for little value when adjusting formats, caches, and protocol. So we filter it out.

stream-clone-test: simplify the --uncompressed alias check To check that --uncompressed is an alias we just need to check it trigger a stream clone, we don't need to check anything else.

stream-clone-test: drop an automatic pattern replacement That pattern is nice, but it prevent us to glob the number of bytes when we don't care about them. We don't care about them more often that what we currently checks so dropping this pattern will help use to simplify various tests.

stream-clone-test: simplify the test for getbundle with stream=1 The core of this tests is about checking we receive a stream bundle with such request. We don't need to look at too much of the details of the stream itself. Since the content of the stream if shifting overtime, Such check is very fragile and consume a lot of time for little value when adjusting formats, caches, and protocol. So we reduce the size of what we check to focus on "is this a stream clone" question.

stream-clone-test: factor some piece of basic clone test out Multiple parts of this case (listing cache, checking error) are common to all cases and don't need to be in the conditionnal block. This simplify the test update.

stream-clone-test: simplify the case where server disabled it We have an option to disable it, we don't need to test it with all protocol variants. In addition there is little value in looking at the bytes to bytes details of the reply. Such check is very fragile and consume a lot of time for little value when adjusting formats, caches, and protocol.

rust-matchers: raw regular expression builder Extracting this `re_builder()` from `re_matcher()` makes it reusable in more general cases than matching `HgPath` instances and would help reducing code duplication in RHGitaly.

rust-filepatterns: export glob_to_re function Making this function public should not risk freezing the internal API, and it can be useful for all downstream code that needs to perform glob matching against byte strings, such as RHGitaly where it will be useful to match on branches and tags.

repoview: prevent `None` to be passed as the filtername We let such instantiation slip in a previous commit, so we add an explicit check to prevent it to happen in the future.

branchcache: avoid created a `None` filter repoview when writing The repoview class is not intended to be used for unfiltered repository.

rust-index: don't use mutable borrow to computed filtered heads This does not need to mutate the index. This is the prime suspect for some RuntimeError raised during some pushes.

rust-index: don't use mutable borrow for head-diff computation It does not needs to mutate the index. This is one of the two suspects of RuntimeError being thrown during push.

branchcache: move head writing in a `_write_headers` method Same rational: this will help having format variants.

branchcache: move head writing in a `_write_heads` method Same rational: this will help having format variants.

branchcache: move the header loading in a `_load_header` class method This will help changing header parsing in format variants.

branchcache: simplify a long line Gratuitous change to help code readability.

branchcache: rename `load` to `_load_heads` We are about to have more similar function, we rename the existing one to a more meaningful name and mark it private in the process.

branchcache: move the filename to a class attribute This prepare the introduction of more variant of cache.

test-clonebundles: simplify matching to be less flavor depends We keep the files and bytes output for the first call, but then we mostly check that we are being served a stream-clone bundle, not the actual content and size of the bundle. That aspect being tested by the stream clone test themselves.

repoview: fix changelog.__contains__ method This have been around for ten years, so we can safely that this method have few callers. However I am about to add one.

branchcache: unconditionally write delayed branchmap

branchcache: drop the unused `_verifyclosed` This code appears dead since its introduction about 5 years ago in this three consecutive commits: - 6578654916ae → introduce the method with two calls - 7c9d4cf23adf → remove first call - be5eeaf5c24a → remove second call o changeset: be5eeaf5c24a | user: Pulkit Goyal <pulkit@yandex-team.ru> | date: Fri Apr 05 15:57:09 2019 +0300 | summary: branchcache: don't verify closed nodes in _branchtip() | o changeset: 7c9d4cf23adf | user: Pulkit Goyal <pulkit@yandex-team.ru> | date: Fri Apr 05 15:56:33 2019 +0300 | summary: branchcache: don't verify closed nodes in iteropen() | o changeset: 6578654916ae | user: Pulkit Goyal <pulkit@yandex-team.ru> ~ date: Mon Apr 01 13:56:47 2019 +0300 summary: branchcache: lazily validate nodes from the branchmap

branchcache: dispatch the code into the dedicated subclass The code useful only to the local brancache have now been moved into the dedicated subclass. This will help improving the branchcache code without subtle breaking the remote variants.

branchcache: introduce a base class for branchmap This will help define a clear boundary between the two.

branchcache: fix the copy code We copy some internal attribute along too. This should prevent inconsistency in the resulting branchmap.

branchcache: pass a "verify_node" attribut to __init__ instead of hasnode The hasnode callback cannot be inherited and is dropped on copy, which seems like a bad idea. Instead we pass the actual semantic as a parameter and let the internal logic deal with it.

branchcache: stop storing a repository instance on the cache altogether We did not really needed it and we do not needs it anymore at all. So lets make things simpler for consistency and garbage collecting and stop storing it altogether.

branchcache: pass the target repository when copying Branchmap are usually copied to be used on a different repoview using a different filter level. Passing the repository around means the repository in `branchcache._repo` will drift from the actual branchmap filter. This is currently "fine" because the repo is only used to retrieve the `nullid` value. However, this is a fairly big trap for any extension or future code using the `_repo` attribute. The replace logic is now using a copy to ensure the right repository view is used to initialized the cached value. We add a couple of assert for make sure this inconsistency does not sneak back.