branching: merge stable into default

rust: address 'error: unnecessarily eager cloning of iterator items' Build failed with a reference to https://rust-lang.github.io/rust-clippy/master/index.html#iter_overeager_cloned which seems reasonable. There doesn't seem to be any reason to not follow the advice.

utils: fix resourceutil use of deprecated importlib.resources Some importlib functionality was deprecated in 3.11 . The documentation on https://docs.python.org/3.12/library/importlib.resources.html recommends using the new .files() API that was introduced in 3.9.

utils: avoid using internal _imp.is_frozen() imp has been deprecated for a long time, and were removed in Python 3.12 . As a workaround, we started using the internal _imp. That is ugly and risky. It seems less risky to get the functionality in some other way. Here, we just inspect if 'origin' of the '__main__' module is set and 'frozen'. That seems to work and do the same, and might be better than using the internal _imp directly. This way of inspecting module attributes seems to work in some test cases, but it is a risky change. This level of importlib doesn't have much documentation, a complicated implementation, and we are dealing with some odd use cases.

cext: use sys.executable instead of deprecated Py_GetProgramFullPath Fix warning with Python 3.13: mercurial/cext/parsers.c: In function 'check_python_version': mercurial/cext/parsers.c:1243:30: warning: 'Py_GetProgramFullPath' is deprecated [-Wdeprecated-declarations] 1243 | Py_GetProgramFullPath()); | ^~~~~~~~~~~~~~~~~~~~~ In file included from /usr/include/python3.13/Python.h:119, from mercurial/cext/parsers.c:11: /usr/include/python3.13/pylifecycle.h:43:43: note: declared here 43 | Py_DEPRECATED(3.13) PyAPI_FUNC(wchar_t *) Py_GetProgramFullPath(void); | ^~~~~~~~~~~~~~~~~~~~~ At this point in time, the PyConfig struct memory has been released and the PyConfig API can't be used. https://docs.python.org/3.13/c-api/init.html#c.Py_GetProgramFullPath recommands using sys.executable instead. Let's assume that will work in all versions. It would perhaps be better to use PySys_GetObject, but I prefer to stay consistent with how the same function is retrieving sys.hexversion.

subrepoutil: pass re.sub 'count' argument by name Python 3.13 started warning: DeprecationWarning: 'count' is passed as positional argument

tests: pass re.MULTILINE to re.sub as 'flags' - not in 'count' position This bug was caught by the new Python 3.13 warning: DeprecationWarning: 'count' is passed as positional argument

tests: use packaging from setuptools instead of deprecated distutils When invoking StrictVersion in 3.12 we got: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. distutils is dead in the standard library, and we have to move towards using `setuptools` as general extern dependency. Instead of also requiring the extern `packaging`, we will just use the packaging that is vendored in setuptools.

tests: drop test-demandimport.py distutils test that failed with warnings The test would fail because warnings: /usr/lib/python3.11/site-packages/_distutils_hack/__init__.py:18: UserWarning: Distutils was imported before Setuptools, but importing Setuptools also replaces the `distutils` module in `sys.modules`. This may lead to undesirable behaviors or errors. To avoid these issues, avoid using distutils directly, ensure that setuptools is installed in the traditional way (e.g. not an editable install), and/or make sure that setuptools is always imported before distutils. warnings.warn( /usr/lib/python3.11/site-packages/_distutils_hack/__init__.py:33: UserWarning: Setuptools is replacing distutils. warnings.warn("Setuptools is replacing distutils.") The test for distutils.msvc9compiler comes from 2205d00b6d2b. But since then, distutils is going away, and this test must change somehow. It is unclear exactly how setuptools depended on msvc9compiler, but setuptools also moved forward, and this exact test no longer seems relevant. It thus seems like a fair solution to remove the test while keeping the demandimport blacklist of distutils.msvc9compiler.

utils: test coverage of makedate Explore the scenario from ae04af1ce78d to avoid future regressions. This was intended to give some coverage of the change in faccec1edc2c.

filecache: use bytes wherever possible in the tests This is closer than the actual usage, so I figured in would not hurt.

cachestat: avoid creating cachestat for http path The statichttprepo repo attemp to create cachestat for content we access through http. We modify the couple of place create cachestat object to detect this situation and avoids it. This is not marvelous, but there is few of them and the freeze is looming. This helps on Windows where calling cachestat on http path might create issues.

filecache: use binary path in the test This was overlooked when converting string. This is needed as we are about to introduce bytes specific code in the filecache code path.

branching: merge stable into default

py-3-13: stabilize the docstring output across all supported Python versions Python 3.13 now trims indents from docstrings at compilation time (to save space in .pyc), so all of our helptext is affected. The indentation has never served a user-facing purpose and was more here because nobody cared enough to remove it: we gain some screen space this way. Rather than undo the transformation (which isn't really possible since the transform also deletes leading/trailing whitespace), we align the behavior of older Python versions with that of 3.13. Unfortunately, this means breaking some of the translations. I've only touched the ones that need to work for some tooling tests to pass, but I do not have the time to fix the rest of them across all languages, since they cannot be done in an automated way. i18n updates have been basically abandonned for a good while now, hopefully someone cares enough to bring them back.

docstring: backed out changeset 51057ab0dffa In retrospect this is too much of a behavior change for stable. So I grafted the same change as 31076a2301f1 on default, and I am backing out its version on stable.

tests: stabilize `test-clonebundles-autogen.t` on Windows The problem was that the commands are spun up with `shell=True`, which uses `cmd.exe`, which doesn't understand `$foo` style variables. The HGCB variable expansion has to be delayed, because it's figured out right before launching the command. We could probably add a conditional for Windows, and rewrite the config to use `%foo%` style variables, but it's more maintainable to just wrap the command in a bash shell invocation. The forward style slashes in the path are needed to avoid accruing double backslashes (when switching between shells- the url template seems fine). Also need to strong quote the command so that the double quotes don't get stripped off of `$HGCB_BUNDLE_PATH`, which results in: sh: 1: Syntax error: Unterminated quoted string abort: command returned status 2: sh -c "cp $HGCB_BUNDLE_PATH $TESTTMP/final-upload/"

clonebundles: stop shell quoting `HGCB_BUNDLE_BASENAME` environment variable This causes problems in `test-clonebundles-autogen.t` on Windows, because the quoted path ends up being passed to the `cp` command, which fails, because quote characters are not a legal part of a file name. I don't see any quoting in environment variables on either MSYS or WSL, even with weird ones that appear to have escape sequences like `PS1=\[\033]0;$MSYSTEM:\w\007` (in MSYS). The quoting was added back in 5ae30ff79c76, and as shown here, was causing problems even on posix when a quote was slipped into the path. (The other obvious problem is that the command is spun up shell style, which invokes `cmd.exe`, which doesn't know about `$foo` style variables. That will be addressed next, but that change didn't work without this too.)

tests: add coverage to for `HGCB_BUNDLE_BASENAME` with special characters Per request on IRC, to show the behavior of dropping the quoting of `HGCB_BUNDLE_BASENAME` in the next commit. This current failure is basically the same error and output that currently happens on Windows with any path (even without the embedded quote). The only difference is Windows doesn't print the `cp: cannot stat ...` line.

tests: stabilize `test-eol-update.t` on Windows Perhaps it's better if this doesn't happen, but there are a bunch of tests that spew this, and we already have a conditional match for this in the block prior to the comment right above this section. So accept it as a possibility, and reduce the noise in the Windows tests.

tests: force `dummysmtpd.py` to write its log file with '\n' on Windows The log files were being `cat'd` in `test-patchbomb-tls.t`, and causing gratuitous failures. Since `sys.stdout` is being written to with `str` instead of `bytes`, use a `io.TextIOWrapper` to change the EOL, like 2924676d4728.

tests: raise the default value for the various `devel.sync.*-timeout` configs These are used in `mercurial.testing.wait_file()` to stall for a file to appear in the filesystem, and raise an error if the file doesn't show up before the timeout expires. The default of 2s was way too low on Windows, especially when running tests in parallel, and resulted in various timeouts in `test-dirstate-read-race.t`, `test-dirstate-status-write-race.t`, and `test-clone-stream-revlog-split.t`. The various `wait-on-file` invocations in the tests are inconsistent, and wait anywhere from 5s - 20s. I'm using 20s here because if everything is working, the timeout won't matter. Also with the default timeout being raised on Windows in f4c038081561, both `HGTEST_TIMEOUT_DEFAULT` and `HGTEST_TIMEOUT` are 1440 in the default case where the timeout is not specified on the command line of the test runner, so the timing factor that is multipled with the value is 1, resulting in no changes. (But if someone specified a lower value on the command line, that would *lower* the timeout period used.)

tests: remove deprecated test-check-py3-compat.t If our current source files were not compatible with Python 3, we would know by now. This check has not been relevant for a couple of years now and we can safely remove it.

zope-interface: add compatibility with 3.13 compiler attributes We could follow-up with an actual vendoring update from the newest version of zope-interface in the new cycle since we're dropping 3.7 and down. However we are also in the process of replacing zope-interface with Protocol, so hopefully we can simply drop the zope-interface vendoring.

py-3-13: fix traceback matching for the new Python version

py-3-13: stabilize the docstring output across all supported Python versions Python 3.13 now trims indents from docstrings at compilation time (to save space in .pyc), so all of our helptext is affected. The indentation has never served a user-facing purpose and was more here because nobody cared enough to remove it: we gain some screen space this way. Rather than undo the transformation (which isn't really possible since the transform also deletes leading/trailing whitespace), we align the behavior of older Python versions with that of 3.13. Unfortunately, this means breaking some of the translations. I've only touched the ones that need to work for some tooling tests to pass, but I do not have the time to fix the rest of them across all languages, since they cannot be done in an automated way. i18n updates have been basically abandonned for a good while now, hopefully someone cares enough to bring them back.

test-lfs-serve-access: vastly simplify the error output check The traceback differences between all supported Python versions is about to become insane to handle, so let's check what we're actually looking to check.

subrepo: propagate non-default path on outgoing There was already a fix made in 5dbff89cf107 for pull and push commands. I did the same for the outgoing command. The problem I identified is that when the parent repository has multiple paths, the outgoing command was not respecting the parent path used and was always using the default path for subrepositories.

outgoing: pre-indent some code This will make the next changeset clearer.

outgoing: move sorting in the display function This make the core code simpler.

outgoing: extract changeset display in its own function This will clarify future patches.

subrepo: move code around

merge: add a config to allow conflict-free merge of changes on adjacent lines This change adds a config to make it no longer a conflict to merge changes made on adjacent lines. The reason these changes are considered a conflict is that there's no region of text at the relevant position (sync region) that's kept unchanged by both sides of the merge. The problem can be solved by making the sync regions being a bit more powerful: we can keep a 0-length sync region if we find that a block unchanged by one side is ajacent to a block unchanged by the other side. Since these 0-length sync regions are emitted using the ~same algorithm as the normal non-empty sync regions, this change involves no arbitrary decisions and I expect it to work pretty well. 0-length sync regions do create an ambiguity in a special case where two pairs of adjacent regions "meet" at the same point. This corresponds to an insertion made at the same place by the two sides of the merge, and this still results in a conflict.

localrepo: drop the CamelCase name for `localrepo.ilocalrepositorymain` See 61557734c0ae for the reasoning. This one is slightly different, however, because the `localrepository` class already subclasses the interface class in the type checking phase.

localrepo: drop the CamelCase name for `localrepo.revlognarrowfilestorage` See 61557734c0ae for the reasoning.

localrepo: drop the CamelCase name for `localrepo.revlogfilestorage` See 61557734c0ae for the reasoning.

localrepo: drop the CamelCase name for `localrepo.locallegacypeer` See 61557734c0ae for the reasoning.

localrepo: drop the CamelCase name for `localrepo.localpeer` See 61557734c0ae for the reasoning.

localrepo: drop the CamelCase name for `localrepo.localcommandexecutor` See 61557734c0ae for the reasoning.

manifest: drop the CamelCase name for `manifest.treemanifestctx` See 61557734c0ae for the reasoning.

manifest: drop the CamelCase name for `manifest.memtreemanifestctx` See 61557734c0ae for the reasoning.

manifest: drop the CamelCase name for `manifest.manifestctx` See 61557734c0ae for the reasoning.

manifest: drop the CamelCase name for `manifest.memmanifestctx` See 61557734c0ae for the reasoning.

manifest: drop the CamelCase name for `manifest.manifestlog` See 61557734c0ae for the reasoning.

manifest: drop the CamelCase name for `manifest.manifestrevlog` See 61557734c0ae for the reasoning.

manifest: drop the CamelCase name for `manifest.treemanifest` See 61557734c0ae for the reasoning.

manifest: drop the CamelCase name for `manifest.manifestdict` See 61557734c0ae for the reasoning.

filelog: drop the CamelCase name for `filelog.filelog` See 61557734c0ae for the reasoning.

revlog: drop the CamelCase name for `revlog.revlogproblem` See 61557734c0ae for the reasoning.

revlog: drop the CamelCase name for `revlog.revlogrevisiondelta` These CamelCase names popped up this cycle because pytype was getting confused by the zope decorator, and so the decoration was decoupled from the class declaration. We're in the process of switching all of the zope interfaces to `typing.Protocol` classes, but we're up against the code freeze. It would be nice to use CamelCase for protocol classes (or classes in general), but let's reset to a consistent state and buy some more time to think about this. Since we're moving to Protocol classes and I disabled the interface tests back in ef7d85089952, I'm not bothering to re-add the decorator, and re-confuse pytype. But do place the interface next to the class, so that these places can be found with the `grep` when the time comes to subclass.

tests: use pyflakes as a tool, not a python module The usage of pyflakes as a Python module was introduced in e397c6d74652, to work around issue between Python 2 and Python 3. This issues are long behind us now and we can get beck to using pyflakes as a tool, giving us more flexibility about how we install it. The `hghave` requirements is modified to check that we have a tool available, instead of a python module.

clang-format: do not accept version above 19.x This gives me different formatting on latest version available in Debian. So disabling for now.

rust-update: add a config item to disable the Rust update fastpath Explanations inline.

update: add a Rust fast-path when updating from null (and clean) This case is easy to detect and we have all we need to generate a valid working copy and dirstate entirely in Rust, which speeds things up considerably: On my machine updating a repo of ~300k files goes from 10.00s down to 4.2s, all while consuming 50% less system time, with all caches hot. Something to note is that further improvements will probably happen with the upcoming `InnerRevlog` series that does smarter mmap hanlding, especially for filelogs. Here are benchmark numbers on a machine with only 4 cores (and no SMT enabled) ``` ### data-env-vars.name = heptapod-public-2024-03-25-ds2-pnm # benchmark.name = hg.command.update # bin-env-vars.hg.py-re2-module = default # bin-env-vars.hg.changeset.node = <this change> # benchmark.variants.atomic-update = no # benchmark.variants.scenario = null-to-tip # benchmark.variants.worker = default default: 5.328762 ~~~~~ rust: 1.308654 (-75.44%, -4.02) ### data-env-vars.name = mercurial-devel-2024-03-22-ds2-pnm # benchmark.name = hg.command.update # bin-env-vars.hg.py-re2-module = default # bin-env-vars.hg.changeset.node = <this change> # benchmark.variants.atomic-update = no # benchmark.variants.scenario = null-to-tip # benchmark.variants.worker = default default: 1.693271 ~~~~~ rust: 1.151053 (-32.02%, -0.54) ### data-env-vars.name = mozilla-unified-2024-03-22-ds2-pnm # benchmark.name = hg.command.update # bin-env-vars.hg.py-re2-module = default # bin-env-vars.hg.changeset.node = <this change> # benchmark.variants.atomic-update = no # benchmark.variants.scenario = null-to-tip # benchmark.variants.worker = default default: 38.901613 ~~~~~ rust: 11.637880 (-70.08%, -27.26) ### data-env-vars.name = netbsd-xsrc-public-2024-09-19-ds2-pnm # benchmark.name = hg.command.update # bin-env-vars.hg.py-re2-module = default # bin-env-vars.hg.changeset.node = <this change> # benchmark.variants.atomic-update = no # benchmark.variants.scenario = null-to-tip # benchmark.variants.worker = default default: 4.793727 ~~~~~ rust: 1.505905 (-68.59%, -3.29) ```

merge: improve working-copy mtime race handling Explanations inline. This also makes use of `make_mtime_reliable`, which unifies our mtime raciness logic from the status. On top of this, this fixes the handling of the pure dirstate status to better catch racy status, as we've been doing in Rust for a long time now.

merge: move the filtering of ambiguous files to a dedicated function I have multiple reasons: - The body of `_update` is way too long - This adds typing which will help our tooling and brains understand this code more easily - This function will get more nested and complex in the next patch I've taken the liberty of rewrapping and typo-passing the docstring.

timestamp: make the reliable comparison more usable from outside This is going to be used with pre-computed times unlike in status.

timestamp: add type information to the module This is easy to do and helps both Pytype and developpers understand what objects they are dealing with.

rust-dirstate: make the reliable timestamp comparison more usable from outside This is going to be used with pre-computed times unlike in status.

rust-dirstate-map: use a more precise identity This is closer to the behavior of what Python does. So far, we were checking only the inode, but this might not be good enough for the v1 case.

dirstate-map: add a missing debug wait point when accessing the v2 docket fc8e37c380d3 added synchronization points to the dirstate to allow for race condition testing without actually requiring a time-based race condition to happen. This changes adds the `pre-read-file` wait point before we read the docket, since callers might ask for the parents before anything else is read, leading to the first read being done before the wait point. This removes some differences in test output which were presumed to be speed related, but weren't.

rust-parsers: use the same error message as with the higher-level code This can happen at two places, but it's not really enough time to justify it being refactored. Let's ensure we have the same error message, the newer one being slightly more helpful.

rust-pathauditor: make sure we actually test the nested repo case This covers the *on-disk* case, where the path itself does not have a `.hg` component.

rust-pathauditor: match more of Python's behavior and display messages We will make use of the path auditor when running our update fast-path, and we want to output of it to be close enough.

rust-dirstate: pass dirstate tracked key from the requirements We will make use of this with the `hg update` fastpath

rust-files: check for empty manifests caused by narrow Explanations inline

rust-repo: don't use on-disk dirstate parents in v1 This mistake was not causing any problems yet since we were never updating parents from Rust code. This is about to change, so let's fix it.

rust-cpython: add a util to get a `Repo` from a python path I suspect this will not be the last time we need to do something like this.

rust: improve `InvalidRevision` error message I encountered this when debugging earlier and felt like we were losing some information along the way, which we were!

rust-dirstate: use a struct as arguments for the high-level `reset_state` This makes the interface a lot clearer at the call site and prevents silly mistakes, as an API with a bunch of booleans is prone to errors. This refactor adds a `from_empty` parameter for a fast-path when resetting and entry we're sure does not exist. It will be used in the upcoming update Rust fastpath, and was not split to prevent more churn.

rust: implement `From<SparseConfigWarning>` for `HgError` This will be useful in a future patch to avoid a lot of boilerplate.

rust-hg-cpython: add an `HgProgressBar` util This will be the entry point for all progress bars from a Python context in upcoming patches. Like the `Progress` trait, this is subject to change once we have more use cases, but this is good enough for now.

rust: add `Progress` trait for progress bars This will be used in the next few changes to introduce a progress bar for the `hg update` fastpath.

rust-files: separate the listing of files from a revset and a revision We won't need to parse a revset all the time, and an upcoming patch will make use of this new util.

rust-files: add a `Sync` bound to the matcher This enables us to use this code in multithreaded environements, which we very much want to do.

rust-files: also return filenode and flags when listing a revision's files This is going to be useful when implementing parts of `update` and makes it so we don't have to fetch the manifest and each entry twice.

rust-manifest: encode flags as `Option<NonZeroU8>` This makes the compiler use the niche optimization for all flags: since 0 is not a valid representation of any flags, we can use 0 as a replacement for `None`, which reduces memory footprint and could yield a little performance improvement over many iterations.

rust-repo: add a method to set the current parents This will be useful when we start writing an `hg update` fastpath.

rust-repo: add a method to get a `Node` from a `Revision` to the `Repo` This is going to be more and more useful as we start writing higher-level code

rust-utils: move the `filesystem_now` function to a util This is going to be useful for an upcoming `hg update` fastpath.

rust-dirstate: actually remember the identity This was an oversight that likely had no influence on anything since we have only been writing the dirstate for status. We will start writing the dirstate for more operations now, so we'll need this fixed.

mergestate: reduce the number of attribute lookups This code is called a lot during updates, this is a very small but also very easy thing to do.

test-lfs-serve-access: account for output differences in Python 3.12 This test is quite noisy, but the fix was very easy.

patchbomb: don't test ambiguous address This is a bug in Python's `email` package and shouldn't be relied on. Python 3.12 has fixed this problem¹ and raises an exception. We keep the multiple `-t` because this is still relevant for testing. [1] https://github.com/python/cpython/issues/102988 [2] https://docs.python.org/3/whatsnew/changelog.html

contrib: install pip 24.2 in the Windows dependency installer script One of the modern pythons was crying about the old version importing `distutils`.

contrib: install the latest py 3.11-3.13 in the Windows dependency script Note that python 3.11 adds an installer for arm64, which we ignore here because I don't have a Windows system running arm64, and it's likely we will be dropping support for some platforms before I get my hands on one.

contrib: update to the latest py3.9 and 3.10 in the Windows dependency script Python3.8 was already at its last version with an installer, FWIW.

contrib: drop python 3.7 from the Windows dependency installer script I'm tempted to drop 3.8 too, since we use 3.9 on Windows.

contrib: drop python 3.5 requirements file for Linux automation The new minimum this cycle is py3.8.

windows: implement `util.cachestat` to fix numerous dirstate problems I got here by bisecting the issue1790 related failure on Windows to keep an entry from being marked "unset" in `test-dirstate.t` back to eedbf8256263. There were a handful of other tests failing with an unexpected dirstate entry state like this, as well as numerous "skip updating dirstate: identity mismatch" messages added to various tests, as well as an issue with dirstate wrapping with the largefiles extension[1], all of which appear to be fixed by this. In total, ~25 tests are fully fixed on Windows with this change on default. This is basically a copy/paste of the posix implementation, but we drop the `st_mode` comparison- I think the only reason we care about the mode on posix is to detect +/-x mode changes, but the executable bits on Windows are synthesized based on the name of the file[2]. None of the other parts of the codebase are equipped to handle executable bits in the filesystem on Windows anyway, so it doesn't make sense to worry about them here. Note that `st_uid` and `st_gid` seem to always be 0 on Windows (and I can't find them being initialized), so they can probably be dropped from the comparison. But I doubt they matter any more on posix, since we don't track ownership. The `st_ino`, `st_dev`, and `st_nlink` attributes all seem to have reasonable values for comparing like on posix[3]. Also note that `st_ctime` is apparently deprecated in 3.12+ (for reasons I haven't explored)[4]. [1] https://foss.heptapod.net/mercurial/mercurial-devel/-/merge_requests/884 [2] https://github.com/python/cpython/blob/aab3210271136ad8e8fecd927b806602c463e1f2/Modules/posixmodule.c#L1948 [3] https://github.com/python/cpython/blob/aab3210271136ad8e8fecd927b806602c463e1f2/Python/fileutils.c#L1158 [4] https://github.com/python/cpython/blob/aab3210271136ad8e8fecd927b806602c463e1f2/Modules/posixmodule.c#L2200

extdiff: don't run gui programs when in a cli-only environment In order to provide a useful error message to override the behavior, we also need to slightly change the way that tool.gui is found in the config. Before, it had to be where tool.diffargs is located, which might not exist. Now, tool.isgui can exist on its own. A test is added for the new error message. We also need to force procutil.isgui() to return true, so we set $DISPLAY to a non-empty value before running any test expecting to have a gui.

bookflow: fix bullet list indentation in docstring

doc: generate separate commands/topics/extension pages This change modifies gendoc.py and Makefile so that individual pages for commands, help topics, and extensions can be generated. A new index page is also generated with links to all these pages. This makes it easier to look up and search the help text of a given command or topic, instead of having to deal with the giant hg.1 "all-in-one" page. Since the list of individual pages varies based on the source code, we generate a dynamic Makefile that contains this list of files as individual targets. This gives us fine-grained control over output files. However, it greatly increases the time spent generating all help pages. It's recommended to run make with -j to make use of multi-core archs. Individual man pages are produced in doc/man, and HTML ones are in doc/html

doc: refactor gendoc for better reusability This change separates the gathering of commands/topics/etc from the logic of printing their documentation out.

stream: prefer keeping an open file handle to volatile file instead of copy We will still do copy if too many file handle are open. Currently, have less than 10 volatile files in typical usage, so we should be fine. See inline documentation for details.

stream: remove __getitem__ from the VolatileManager It is no longer used and that API will get in the way of the new order.

stream: open volatile file through the manager To do more subtle things, we need more control.

stream: rename TempCopyManager to VolatileManager We are going to be more subtle in our management of volatile file, so use a more semantic name.

ci: only use the macOS runner if manually invoked Right now, we only have a single machine with a single runner, so don't create a bottleneck by default.

ci: add a runner for macos This is running from a venv, created with the packages from the requirements file in the `test-setup-future-work` topic being discussed separately. We can basically reuse the existing `test_c` configuration as-is, but there was one small issue- if the clone from the previous CI run is still present in `/tmp`, the clone for the current run fails. It's not a problem for the current setup because a new container is instantiated, so the easy fix is to simply delete the clone first. (The Windows CI avoids this by not cloning in the first place.) It looks like it takes ~80m to run the whole test suite on the current system.

branching: merge stable into default

tests: use shlex.quote instead of pipes.quote The pipes module got removed in python 3.13. https://bugs.debian.org/1084553

branchmap-v3: make sure we write the cache after detecting pure-topo mode We were properly detecting the pure topo-case but in some case the one disk cache file was never updated with that information.

demande-import-test: use `wsgiref` instead of `telnetlib` for testing The `telnetlib` module has been dropped in 3.13. The `wsgiref` module fit the same purpose of being obscure and not imported yet.

hgdemandimport: add collections.abc to the ignore list because Python 3.13 rc3 It worked in Python 3.13 rc2, but something changed for rc3. Now, when i18n.py imports typing and it touches collections.abc.Hashable , we get: ValueError: module object for collections.abc substituted in sys.modules during a lazy load This seems to be a general problem, released in the final Python 3.13 . I have not analyzed the problem in details. We *could* work around the problem by disabling demand import of typing. But that would effectively disable demand import of collections.abc too. Instead, just disable demand import for collections.abc .

run-tests: include non-activated venv packages in `PYTHONPATH` If a venv is activated since afa9d73780e1, `hghave` would see the packages installed in it, and enable related tests. If the python interpreter was launched directly however, none of them were seen. In addition to getting consistent behavior, it's also easier to manage in CI if the venv doesn't need activation.

tests: allow optional output when `test-lfs-server` is sent SIGTERM I guess this has only ever been tested on Windows, but on macOS, the test was failing with changes like: +++ /private/tmp/mercurial-ci/tests/test-lfs-test-server.t#git-server.err @@ -858,6 +858,7 @@ (Restart the server in a different location so it no longer has the content) $ "$PYTHON" $RUNTESTDIR/killdaemons.py $DAEMON_PIDS + $TESTTMP.sh: line 153: 38639 Terminated: 15 lfs-test-server > lfs-server.log 2>&1 #if hg-server $ cat $TESTTMP/access.log $TESTTMP/errors.log

tests: stabilize `test-http-bad-server.t` on macOS I'm not sure what's going on here, because I got one test run where this worked, but two where it gave this error instead. It's connection related as the comment says it should be, so good enough.

upgrade: disable using the parallel workers optimization on macOS It crashes `test-copies-chain-merge.t` for some reason[1]. It's only experimental, and already hard-disabled on Windows. [1] https://foss.heptapod.net/mercurial/mercurial-devel/-/jobs/2591045#L108

format: fix few black issues

install: static data moved from setup.py to pyproject.toml

ci: display tool version more selectively The goal here is to avoid showing tool version in all case as this create issue for non linux/docker runner that might not have some of them installed. This is currently relevant for the coming mac runner that does not have clang-format installed and does not needs its. This might also be useful in the future to use narrower docker image more specialized for each jobs.

zeroconf: fix a warning about a signature mismatch in a method override Caught by PyCharm. It looks like the `hdr` arg is mostly unused (thus why it was missing in some cases), so pass along an empty string where needed.

zeroconf: use str instead of bytes when indexing `globals()` I suppose since we set the key to bytes during init that it won't raise a KeyError, but this was very likely an oversight when mass-byteifying, rather than purposeful.

zeroconf: fix an invalid argument error on Windows The idea that pyoxidizer was triggering the problem when standing up the previous incarnation of CI for Windows was misleading- it was a Windows problem in general. See the inline bug link. Unfortunately, there's no commit referenced there, and it looks like OP closed the report himself with the suggested workaround. IOW, very modern python may not work, but it's extremely unlikely that there are any users of this extension, especially on Windows.

tests: fix hooks in `test-transaction-rollback-on-revlog-split.t` for Windows A few problems with external hooks here: 1) `cmd.exe` is blissfully unaware of the meaning of single quotes, and passes them along as part of the revision arg 2) `cmd.exe` doesn't know how to run the python script `f` with the shebang line, so it needs to be invoked with the intepreter explicitly. Then for some reason it was trying to open `$TESTTMP\troffset-computation-hooks\f`, so make it an absolute path. 3) Likewise, the shell script `wait-on-file` cannot be invoked directly by `cmd.exe`. 4) Windows python doesn't understand whatever `$TESTTMP` unrolls to (probably it has the double backslashes), and silently failed to write the lock file. I'm assuming that `pretxnclose.03-abort` is also bad (there's no `false` in `cmd.exe`), but it's not currently causing problems. Also note that the paths in this are very long, and typically exceed the `MAX_PATH` limit in Windows. It can be run with `run-tests.py --tmpdir /c/t`.

tests: use the cross platform `SIGKILL` function This partially fixes `test-transaction-rollback-on-revlog-split.t` (there are still problems related to waiting on the lock files), and completely fixes `test-fncache.t`.

tests: add a module that can perform the equivalent of `SIGKILL` on any OS I started with this being Windows specific, but let's push all of the decision making into this function so that it can just be called by the tests. The tradeoff is that this is very specific to sending `SIGKILL`- since `signal.SIGKILL` doesn't exist on Windows, the desired signal can't be passed from the caller. Maybe there's a way, but let's wait until there's a need. We don't use `killdaemons.py` unconditionally because it starts with a more graceful `SIGTERM` on posix.

tests: teach `killdaemons` on Windows to use an exit code provided by a caller Right now, there are several tests that use `signal.SIGKILL`, which isn't a thing on Windows. The `killdaemons` script approximates this by forcibly terminating the process. There's a minor difference in that `signal.SIGKILL` results in the test capturing an exit code of 137, and the `killdaemons` victim doesn't record a code (as though it exited with 0). Since the exit code line couldn't be conditionalized the last time I checked, let's just allow the caller to simulate the same exit code, and avoid conditionalizing the tests.

ci: also offer tests with Python 3.13 Python3.13 is du to be released soon. We better make sure we work with it.

run-test: add regular line break and test count in the dot output This will make it simpler to follow the actual progress. The value "75" have been picked as it is smaller or equal to 80 and give a nice round number.

clonebundle-digest: add recursion guards for Python 3.8 For Python 3.8 and 3.9, the read/readinto pair can recurse, so make sure the data is only hashed once.

tests: replace inline `waitlock()` with `wait-on-file` script The latter scales up the timeout based on the timeout value provided to the test runner, and I was seeing timeouts on Windows when running all of the tests using all CPU cores.

tests: stabilize `test-split-legacy-inline-changelog.t` on Windows The `tar` command is unable to process "C:\path\to\foo.tar" style paths, which is how `$TESTDIR` is constructed. It also didn't work with `$TESTDIR_FORWARD_SLASH`- both failed with: tar: Cannot connect to C: resolve failed [128] But `cat` can handle it if the path is quoted, and `tar` can read from stdin.

tests: stabilize `test-largefiles-cache.t` on Windows The `undo.backup.dirstate.bck` was created by `hg commit -m 'add largefile'` at line 18, and deleted in the conditional block by `hg push -q http://localhost:$HGPORT/` at line 138. That's... surprising, but probably not harmful, and can be debugged from Linux if it is a problem. This was showing up in `find src/.hg/largefiles/* | grep -E "(dirstate|$hash)"`.

tests: stabilize `test-journal.t` on Windows The file in these lines are double quoted, but single quoted on other platforms. Not sure why, other than `cmd.exe` doesn't recognize single quotes. But it's a cosmetic difference, so glob over it and move on.

tests: stabilize `test-clonebundles.t` on Windows The `remote: {foo,bar}` lines were different because `echo` in MSYS uses `\r\n`. I couldn't make it work with the previous echoing of individual lines, changing the internal `echo` to `printf "foo\n"`, because that output as "foon". This works on Linux and Windows, so I'm not thinking too hard about it.

tests: cleanup some special casing of `seq` emitting '\r\n' My guess is these predate the commit referenced in the previous commit.

tests: force `seq` to print with '\n' EOL It looks like consistent EOL is the reason for 0605726179a0, but now on py3, `print()` uses the platform EOL without regard to binary mode. The tests mostly use this to loop over a sequence of number in the shell, but there are a handful that redirect output to a file. Specifically, this fixes Windows runs of `test-bundle2-multiple-changegroups.t`, but there may be other tests this fixes. Some other `tests/*.py` files also set binary mode on stdout, but they also write bytes directly to `sys.stdout.buffer`. I'm not doing that here because PyCharm flags these write calls for passing bytes instead of str (PyCharm is likely wrong, but possibly confused because the code falls back to `sys.stdout` if there is no `.buffer` attribute), and it's annoying.

tests: stabilize `test-bundle-phase-internal.t` on Windows This is a silly difference, and we control the Windows side of things from mercurial/windows.py:195. I'll swap and quote the values to be like `no-windows` at some point, but I suspect this exception output would appear elsewhere, and don't feel like waiting for a 2h+ test run to find all of them.

tests: skip `test-wsgicgi.t` on MSYS The test is attempting to set `PATH_INFO="/rev/\xe2\x80\x94"` into the environment, which it does. The problem is that when MSYS sees a leading '/' in an environment variable, it thinks it's a unix filesystem path, so it "helpfully" prepends the Windows path to the MSYS root directory before running a non-MSYS process. hgweb would then split this value on '/', so it would get 'C:' instead of 'rev', and return a 400 since that isn't a valid web command. I tried generating a *.bat file, but had trouble running that via `cmd.exe` inside the test. I also tried generating an equivalent *.py launcher that would set the environment variables itself. But there is no `os.environb` on Windows, and the value was getting mangled when put into the script. So, I give up. If it's encoding stuff on Windows, it's probably broken.

clonebundles: allow manifest to specify sha256 digest of bundles

install: add long_description_content_type

headrevs: replace a boolean match with a if/else I missed that while doing a previous cleanup.

head-revs: add a native implementation of the `stop_rev` parameter This does not add too much complexity to the native code and help with branchmap v3 performance. Note that the final conversion of the heads from native-code to Python is still too costly, especially in Rust. In addition the current caching around headrevs is too simple and fragile. However these are an unrelated problem. ### benchmark.name = hg.command.unbundle # bin-env-vars.hg.py-re2-module = default # benchmark.variants.issue6528 = disabled # benchmark.variants.resource-usage = default # benchmark.variants.reuse-external-delta-parent = yes # benchmark.variants.revs = any-1-extra-rev # benchmark.variants.source = unbundle # benchmark.variants.validate = default # benchmark.variants.verbosity = quiet ## data-env-vars.name = netbeans-2018-08-01-zstd-sparse-revlog # bin-env-vars.hg.flavor = default branch-v2: 0.233711 ~~~~~ branch-v3 before: 0.239857 (+2.63%, +0.01) branch-v3 after: 0.239558 (+2.50%, +0.01) # bin-env-vars.hg.flavor = rust branch-v2: 0.235230 ~~~~~ branch-v3 before: 0.240972 (+2.44%, +0.01) branch-v3 after: 0.239917 (+1.99%, +0.00) ## data-env-vars.name = netbeans-2018-08-01-ds2-pnm # bin-env-vars.hg.flavor = rust branch-v2: 0.255586 ~~~~~ branch-v3 before: 0.268560 (+5.08%, +0.01) branch-v3 after: 0.262261 (+2.61%, +0.01) ## data-env-vars.name = mozilla-central-2024-03-22-zstd-sparse-revlog # bin-env-vars.hg.flavor = default branch-v2: 0.339010 ~~~~~ branch-v3 before: 0.349389 (+3.06%, +0.01) branch-v3 after: 0.348247 (+2.72%, +0.01) # bin-env-vars.hg.flavor = rust branch-v2: 0.346525 ~~~~~ branch-v3 before: 0.355661 (+2.64%, +0.01) branch-v3 after: 0.350906 (+1.26%, +0.00) ## data-env-vars.name = mozilla-central-2024-03-22-ds2-pnm # bin-env-vars.hg.flavor = rust branch-v2: 0.380202 ~~~~~ branch-v3 before: 0.408851 (+7.54%, +0.03) branch-v3 after: 0.406511 (+6.92%, +0.03) ## data-env-vars.name = mozilla-unified-2024-03-22-zstd-sparse-revlog # bin-env-vars.hg.flavor = default branch-v2: 0.412165 ~~~~~ branch-v3 before: 0.427782 (+3.79%, +0.02) branch-v3 after: 0.422595 (+2.53%, +0.01) # bin-env-vars.hg.flavor = rust branch-v2: 0.412397 ~~~~~ branch-v3 before: 0.422354 (+2.41%, +0.01) branch-v3 after: 0.421079 (+2.11%, +0.01) ## data-env-vars.name = mozilla-unified-2024-03-22-ds2-pnm # bin-env-vars.hg.flavor = rust branch-v2: 0.429501 ~~~~~ branch-v3 before: 0.443197 (+3.19%, +0.01) branch-v3 after: 0.449432 (+4.64%, +0.02) ## data-env-vars.name = mozilla-try-2024-03-26-zstd-sparse-revlog # bin-env-vars.hg.flavor = default branch-v2: 3.403171 ~~~~~ branch-v3 before: 3.819477 (+12.23%, +0.42) branch-v3 after: 3.658482 (+7.50%, +0.26) # bin-env-vars.hg.flavor = rust branch-v2: 3.454876 ~~~~~ branch-v3 before: 3.590284 (+3.92%, +0.14) branch-v3 after: 3.545843 (+2.63%, +0.09) ## data-env-vars.name = mozilla-try-2024-03-26-ds2-pnm # bin-env-vars.hg.flavor = rust branch-v2: 3.465435 ~~~~~ branch-v3 before: 3.633278 (+4.84%, +0.17) branch-v3 after: 3.556074 (+2.62%, +0.09)

head-revs: move hg-core's inner_headrevsfiltered closer to inner_headrevs This is pure code movement, it make a coming changesets significantly clearer.

branchmap-v3: introduce a "stop_rev" argument to `headsrevs` The `headsrevs` method of the revlog already have a `revs` argument to compute the headrevs of a limited set of heads. However, it disable the use of the native compiled code to compute the heads, which slows down the branchmap v3 code a lot. The branchmap v3 usage is actually quite constrained as we will always only ignores a part at the top of the graph. So we could be significantly faster. We start by making small change to the python side to improve the situation and introduce the new API. More collaboration with the native code are coming later. This massively speedup operation and close most of the remaining gaps between branchmap-v3 and branchmap-v2. especially on repository with many revs like mozilla-try. A small overhead remains mostly because the `headrevs` logic currently has some inefficiently. We will look into them from there. ### benchmark.name = hg.command.unbundle # bin-env-vars.hg.py-re2-module = default # benchmark.variants.issue6528 = disabled # benchmark.variants.resource-usage = default # benchmark.variants.reuse-external-delta-parent = yes # benchmark.variants.revs = any-1-extra-rev # benchmark.variants.source = unbundle # benchmark.variants.validate = default # benchmark.variants.verbosity = quiet ## data-env-vars.name = netbeans-2018-08-01-zstd-sparse-revlog # bin-env-vars.hg.flavor = default branch-v2: 0.233711 ~~~~~ branch-v3 before: 0.368769 (+57.79%, +0.14) branch-v3 after: 0.239857 (+2.63%, +0.01) # bin-env-vars.hg.flavor = rust branch-v2: 0.235230 ~~~~~ branch-v3 before: 0.372460 (+58.34%, +0.14) branch-v3 after: 0.240972 (+2.44%, +0.01) ## data-env-vars.name = netbeans-2018-08-01-ds2-pnm # bin-env-vars.hg.flavor = rust branch-v2: 0.255586 ~~~~~ branch-v3 before: 0.318907 (+24.78%, +0.06) branch-v3 after: 0.268560 (+5.08%, +0.01) ## data-env-vars.name = mozilla-central-2024-03-22-zstd-sparse-revlog # bin-env-vars.hg.flavor = default branch-v2: 0.339010 ~~~~~ branch-v3 before: 0.349752 (+3.17%, +0.01) branch-v3 after: 0.349389 (+3.06%, +0.01) # bin-env-vars.hg.flavor = rust branch-v2: 0.346525 ~~~~~ branch-v3 before: 0.354300 (+2.24%, +0.01) branch-v3 after: 0.355661 (+2.64%, +0.01) ## data-env-vars.name = mozilla-central-2024-03-22-ds2-pnm # bin-env-vars.hg.flavor = rust branch-v2: 0.380202 ~~~~~ branch-v3 before: 0.396293 (+4.23%, +0.02) branch-v3 after: 0.408851 (+7.54%, +0.03) ## data-env-vars.name = mozilla-unified-2024-03-22-zstd-sparse-revlog # bin-env-vars.hg.flavor = default branch-v2: 0.412165 ~~~~~ branch-v3 before: 0.424769 (+3.06%, +0.01) branch-v3 after: 0.427782 (+3.79%, +0.02) # bin-env-vars.hg.flavor = rust branch-v2: 0.412397 ~~~~~ branch-v3 before: 0.421796 (+2.28%, +0.01) branch-v3 after: 0.422354 (+2.41%, +0.01) ## data-env-vars.name = mozilla-unified-2024-03-22-ds2-pnm # bin-env-vars.hg.flavor = rust branch-v2: 0.429501 ~~~~~ branch-v3 before: 0.443849 (+3.34%, +0.01) branch-v3 after: 0.443197 (+3.19%, +0.01) ## data-env-vars.name = mozilla-try-2024-03-26-zstd-sparse-revlog # bin-env-vars.hg.flavor = default branch-v2: 3.403171 ~~~~~ branch-v3 before: 6.234055 (+83.18%, +2.83) branch-v3 after: 3.819477 (+12.23%, +0.42) # bin-env-vars.hg.flavor = rust branch-v2: 3.454876 ~~~~~ branch-v3 before: 6.307813 (+82.58%, +2.85) branch-v3 after: 3.590284 (+3.92%, +0.14) ## data-env-vars.name = mozilla-try-2024-03-26-ds2-pnm # bin-env-vars.hg.flavor = rust branch-v2: 3.465435 ~~~~~ branch-v3 before: 5.176076 (+49.36%, +1.71) branch-v3 after: 3.633278 (+4.84%, +0.17)

branchmap-v3: filter topo heads using node for performance reason The branchmap currently contains heads as nodeid. If we build a set of revnum with the topological heads, we need to turn the nodeid in the branchmap to revnum to be able to check if they are topo-heads. That nodeid → revnum lookup is "expensive" and adds up to something noticeable if you do it hundreds of thousand of time. Instead we turn all the topo-heads revnums into nodes and build a set. So we can directly test membership of the nodeids stored in the branchmap. That is much faster. Ideally we would have revnum in the branchmap and could directly test revnum against a revnum set and that would be even faster. However that's an adventure for another time. Without this change, the branchmap format "v3" was significantly slower than the "v2" format. With this changes, some of that gap is recovered With rust + persistent nodemap, this overhead was smaller because the extra lookup did not had to to build the nodemap from scratch. In addition the mozilla-unified repository is able to use the "pure_top" mode of branchmap v3, so it was not really affected by this. Future changeset will work of the remaining of the performance gap. ### benchmark.name = hg.command.unbundle # bin-env-vars.hg.py-re2-module = default # benchmark.variants.issue6528 = disabled # benchmark.variants.resource-usage = default # benchmark.variants.reuse-external-delta-parent = yes # benchmark.variants.revs = any-1-extra-rev # benchmark.variants.source = unbundle # benchmark.variants.validate = default # benchmark.variants.verbosity = quiet ## data-env-vars.name = netbeans-2018-08-01-zstd-sparse-revlog # bin-env-vars.hg.flavor = default branch-v2: 0.233711 ~~~~~ branch-v3 before: 0.380994 (+63.02%, +0.15) branch-v3 after: 0.368769 (+57.79%, +0.14) # bin-env-vars.hg.flavor = rust branch-v2: 0.235230 ~~~~~ branch-v3 before: 0.385060 (+63.70%, +0.15) branch-v3 after: 0.372460 (+58.34%, +0.14) ## data-env-vars.name = netbeans-2018-08-01-ds2-pnm # bin-env-vars.hg.flavor = rust branch-v2: 0.255586 ~~~~~ branch-v3 before: 0.317524 (+24.23%, +0.06) branch-v3 after: 0.318907 (+24.78%, +0.06) ## data-env-vars.name = mozilla-central-2024-03-22-zstd-sparse-revlog # bin-env-vars.hg.flavor = default branch-v2: 0.339010 ~~~~~ branch-v3 before: 0.410007 (+20.94%, +0.07) branch-v3 after: 0.349752 (+3.17%, +0.01) # bin-env-vars.hg.flavor = rust branch-v2: 0.346525 ~~~~~ branch-v3 before: 0.410428 (+18.44%, +0.06) branch-v3 after: 0.354300 (+2.24%, +0.01) ## data-env-vars.name = mozilla-central-2024-03-22-ds2-pnm # bin-env-vars.hg.flavor = rust branch-v2: 0.380202 ~~~~~ branch-v3 before: 0.393871 (+3.60%, +0.01) branch-v3 after: 0.396293 (+4.23%, +0.02) ## data-env-vars.name = mozilla-unified-2024-03-22-zstd-sparse-revlog # bin-env-vars.hg.flavor = default branch-v2: 0.412165 ~~~~~ branch-v3 before: 0.438105 (+6.29%, +0.03) branch-v3 after: 0.424769 (+3.06%, +0.01) # bin-env-vars.hg.flavor = rust branch-v2: 0.412397 ~~~~~ branch-v3 before: 0.438405 (+6.31%, +0.03) branch-v3 after: 0.421796 (+2.28%, +0.01) ## data-env-vars.name = mozilla-unified-2024-03-22-ds2-pnm # bin-env-vars.hg.flavor = rust branch-v2: 0.429501 ~~~~~ branch-v3 before: 0.452692 (+5.40%, +0.02) branch-v3 after: 0.443849 (+3.34%, +0.01) ## data-env-vars.name = mozilla-try-2024-03-26-zstd-sparse-revlog # bin-env-vars.hg.flavor = default branch-v2: 3.403171 ~~~~~ branch-v3 before: 6.562345 (+92.83%, +3.16) branch-v3 after: 6.234055 (+83.18%, +2.83) # bin-env-vars.hg.flavor = rust branch-v2: 3.454876 ~~~~~ branch-v3 before: 6.160248 (+78.31%, +2.71) branch-v3 after: 6.307813 (+82.58%, +2.85) ## data-env-vars.name = mozilla-try-2024-03-26-ds2-pnm # bin-env-vars.hg.flavor = rust branch-v2: 3.465435 ~~~~~ branch-v3 before: 5.381648 (+55.30%, +1.92) branch-v3 after: 5.176076 (+49.36%, +1.71)

rev-branch-cache: properly ignores unaligned trailing data Previously, trailing data could lead to crash and would be written back to disk, disaligning all new data… This is no longer the cases. This was detected while playing with branchmap-v3 that access the rev-branch-cache much more aggressively.

rev-branch-cache: stop pretending we will overwrite data when we don't We were issuing a message about overwriting data even when we were about to write 0 bytes in pratice. This is silly. Instead we point at the extra data remaining in the file (in case someone is using debug to debug something).

rev-branch-cache: fix message about overwritten data If there is data at the end we ignore, we should not count them as overwritten. This reveal that we something don't overwrite anything, this will be taken car of in the next changeset.

branchmap-v3: detect invalid headerline and raise error Otherwise, broken cache file would be accept as valid, but empty.

head-revs: merge the two inner_headrevs… variants Now that there is only one method, it does not make sense to have two different "inner" method. This is especially true as we are about to add another parameter to the method. So we clean up before that.

head-revs: move hg-cpython's inner_headrevsfiltered closer to inner_headrevs This is pure code movement, it make a coming changesets significantly clearer.

head-revs: teach the pure indexes about the `headrevs` method Having this computation done at the index level unify the API and remove revlog side complexity. It might also be a front runner of handing more responsability to the index.

head-revs: remove the `headrevsfiltered` method on the index This method is no longer called and can be removed.

head-revs: stop calling headrevsfiltered The `headrevs` method have been accepting filter since Mercurial 3.2¹. I guess we can rely on it for now. (except the Rust extension that just gained this capability, but it has it now)

rust-cpython: also accept the `filteredrevs` argument in index.headrevs The C version have been accepting this argument since Mercurial 3.2, lets align the Rust index here. This will make it possible to simplify the code in later changesets.

branchmap: use the proper experimental name in cacheutil Otherwise they are not properly copied around.

interfaces: introduce and use a protocol class for the `mpatch` module See f2832de2a46c for details when this was done for the `bdiff` module. Two things worth pointing out- 1) The `cffi` module "inherits" the `pure` implementation of `patchedsize()` because of its wildcard import. 2) It's odd that the `mpatchError` lives in both `pure` and `cext` modules. I initially thought to move the exception into the new class, and make the existing class name an alias to the class in the new location, but the exception is created in C code by the `cext` module, so that won't work. I don't think a protocol class is approriate, because there's nothing special about the class to distinguish from any other `Exception`. Fortunately, nobody is catching this exception in core, so we can kick the can down the road.

branching: merge stable into default

doctest: use the system hg to find the list of file to tests Same as what we do for the `.t` tests, but more manually.

relnotes: skip the test if the source repository is not readable The test want to run the relnot extension, with the tested mercurial, on the original repository. This is not always possible (e.g. when running with --pure and the repository use zstd for example). So we skip the test in this case.

check-code: document reason and suggest alternative to exit code negation The check was introduced in f48b075ff088. We trust the explanation from there.

ci: drop the HGMODULEPOLICY overwrite The test runner have been setting them properly for a while now. So we don't need to do it manually anymore.

test: preserve the existing module policy config for testrepohg resuse The test runner is, righfully, setting HGMODULEPOLICY to match the flavor requests to run the tests. However this mess with the `testrepohg` ability to run properly, as the newly set policy might be incompatible with its installation, or its ability to read the test repo. So we preserve the initial value in a dedicated variable and use it in the `testrepohg` helper.

module-policy: ignore empty module policy This make the variable easier to work with, the empty value is not ambiguous about not wanting to get in the way.

tests: provide an alternate fake lock for filesystems without symlink support

tests: disable `worker.backgroundclose` to stabilize a test on Windows TIL that `worker.enabled=0` doesn't prevent these workers from spinning up. At any rate, there's already a whole lot of conditionalized output following `cat client.log`, the placement of the "starting 4 threads for background file closing" message seems unstable, and we don't care about those worker threads here. Preventing the message is better for test maintenance.

tests: fix lock file path mangling in `test-racy-mutations.t` on Windows I guess `$TESTTMP_FORWARD_SLASH` gets translated by MSYS. This was in the `.foo_commit_out` file: sh: C;C:\\MinGW\\msys\\1.0\\Users\\Matt\\AppData\\Local\\Temp\\hgtests.1qc8jmdl\\child2\\test-racy-mutations.t-skip-detection\\waitlock_editor.sh: $ENOENT

tests: stabilize `test-status-eacces.t` on Windows As noted earlier, `chmod` doesn't complain in MSYS, but also doesn't alter the file permissions such that they are unreadable. I'm guessing the other lines of output in this area that are gated on `rhg` (or not) will also need this, but I don't want to dig too deeply into something that is apparently working well enough.

run-tests: bump the default timeout on Windows to 4x the normal value There are a ridiculous number of tests that timeout on Windows with the 360 sec default (~60). And because of the bug where timed out tests still run to completion before the results are thrown away[1], the timeout does nothing but waste time, so there's no reason to try to find a lower value that still works. For reference on my system: # Ran 909 tests, 116 skipped, 119 failed. python hash seed: 2052473208 real 151m44.322s user 0m0.077s sys 0m0.046s [1] I thought that I wrote a bug for this, but search isn't finding it.

run-tests: bump the minimum python to 3.8 Presumably this was an oversight when hg was updated to 3.8.

tests: stabilize `test-sparse.t` on Windows One of the reserved characters for path values is '*', so it can't be used. Fortunately, missing this seems to not get in the way of any other tests, and it is removed shortly after with `rm -r foo*bar`, and the extant 'foo-bar' matches the pattern.

tests: fix a test hang on Windows when setting a debuglock I have no idea why, but running the `hg -R auto-upgrade debuglock --set-lock` command near the end of `test-upgrade-repo.t` hangs the test. It does background the process and `killdaemons.py` runs without error, but control doesn't return to `run-tests.py` until the process is manually killed. I did notice that `$!` in MSYS is *not* the PID of the process that got backgrounded, even when a simple `sleep 60 &` is run in MSYS without the *.t file. When `killdaemons.py` is run manually with the PID in ProcessExplorer, the backgrounded process terminates immediately, and returns control to `run-tests.py`. This looks like it would be a race, but the test waits 10s for the lock file to appear before attempting to kill the process, so there's time. `hg serve` has a `--pid-file` option to write the pid to the file, but this is only a debug command, so I'm not bothering with cluttering the command line.

tests: conditionalize `chmod` usage in `test-upgrade-repo.t` While the command itself doesn't error out on Windows, it also doesn't make the filesystem readonly. Therefore the repo gets altered to drop dirstate-v2, and puts it out of sync with that happens on Linux.

tests: print the actual timeout value used in `wait-on-file` Previously, it was printing the time passed in, prior to it being scaled up to account for a longer timeout.

tests: stabilize `test-transaction-wc-rollback-race.t` on Windows MSYS has a tendency to munge C:\Dir\SubDir\File into C:DirSubDirFile unless it is quoted, and that's what was happening here- there were a lot of these failures: file not created after 5 seconds: $TESTTMP/transaction-waiting I suspect quoting is only needed in the hook script that is generated (the catting of the log file pointed me in the right direction here), but I missed a spot and trial and error got me here. The quoting elsewhere doesn't harm anything and it was taking 7+ minutes to run this test when things were timing out, so I don't feel like reducing the quoting to the minimum required.

tests: stabilize `test-merge-partial-tool.t` on Windows The test was previously failing because it was opening the shell scripts being used as an executable in a text editor, and problems cascaded from there.

tests: replace `hg id --debug -i` command substitution with non-debug command The censor and convert tests were failing on Windows because the `--debug` flag also prints debug messages, and at least some of these were outputting: skip updating dirstate: identity mismatch ${node} Obviously that causes cascading problems. The other tests were OK, but it's better to use a non debug command for stability.

tests: correct Windows output to account for putting repos in `repo` subdir These were missed in 55c6ebd11cb9, due to being conditionalized and not running in CI.

tests: use pattern matching to mask `ECONNREFUSED` messages The second and third one of these in `test-http-proxy.t` was failing on Windows. The others were found by grep and by failed tests when output was matched and an attempt was made to emit the mask pattern. The first clonebundles failure on Windows emitted: error fetching bundle: [WinError 10061] $ECONNREFUSED$ We should probably stringify that better to get rid of the "[WinError 10061]" part.

typing: add stub functions for `cext/charencoding` I'm not sure if it's better to have a separate file, and currently pytype doesn't really know how to handle these, so it's no help in figuring that out. Technically, these methods are part of the `mercurial.cext.parsers` module, so put them into the existing stub until there's a reason to split it out.

interfaces: introduce and use a protocol class for the `charencoding` module See f2832de2a46c for details when this was done for the `bdiff` module. This lets us dump the hack where the `pure` implementation was imported during the type checking phase to provide signatures for the module methods it provides. Now the protocol classes are starting to shine, because these methods are provided by `pure.charencoding` and `cext.parsers`, and references to `cffi.charencoding` and `cext.charencoding` are forwarded to them as appropriate by the `policy` module. But none of that matters, as long as the module returned provides the listed methods. The interface was copy/pasted from the `pure` module, but `jsonescapeu8fallback` is omitted because it is accessed from the `pure` module directly when the escaping fails in the primary module's `jsonescapeu8()`.

debugantivirusrunning: use bytes when opening a vfs file I noticed this when searching for "base85" to see if anything else in the previous commit needed to be annotated. This was added in 87047efbc6a6, after the mass byteification in 687b865b95ad.

interfaces: introduce and use a protocol class for the `base85` module See f2832de2a46c for details when this was done for the `bdiff` module. It looks like PEP-688 removed the special casing of `bytes` being a standin for any type of `ByteString`, and defines a `typing.Buffer` class (with a backport in `typing_extensions` for Python prior to 3.12). There's been a lot of churn in this area with pytype, but recent versions of pytype and PyCharm recognize this, and e.g. have `mercurial.node.hex()` defined as: from typing_extensions import Buffer def hex(data: Buffer, sep: str | bytes = ..., bytes_per_sep: int = ...) -> bytes This covers `bytes`, `bytearray`, and `memoryview` by default. Both of the C functions here use `y#` to parse the arguments, which means the arg is a byte-like object[2], so the args would appear to be better typed as `Buffer`. However, pytype has a bug that prevents using this from `typing_extensions`[3], and mypy complained `Unsupported left operand type for + ("memoryview")` in the pure module on line 37 (meaning it's only a subset of `Buffer`). So hold off on changing any of that for now. [1] https://peps.python.org/pep-0688/#no-special-meaning-for-bytes [2] https://docs.python.org/3/glossary.html#term-bytes-like-object [3] https://github.com/google/pytype/issues/1772

base85: avoid a spurious use-before-initialized warning in `pure` module The error wasn't possible because the only way for `acc` to not be initialized was if `len(text) == 0`. But then `0 % 5 == 0`, so no attempt at padding was done. It's a simple enough fix to not have PyCharm flag this though. The value needs to be reset on each loop iteration, so it's a line copy, not a line move.

typing: add type annotations to `mercurial/mdiff.py` We'll leave converting `diffopts` to `attrs` as another project.

mdiff: convert a few block definitions from lists to tuples These were flagged by adding type hints. Some places were using a tuple of 4 ints to define a block, and others were using a list of 4. A tuple is better for typing, because we can define the length and the type of each entry. One of the places had to redefine the tuple, since writing to a tuple at an index isn't supported. This change spills out into the tests, and archeology says it was added to the repo in this state. There was no reason given for the divergence, and I suspect it wasn't intentional. It looks like `splitblock()` is completely unused in the codebase.

interfaces: add the optional `bdiff.xdiffblocks()` method PyCharm flagged where this was called on the protocol class in `mdiff.py` in the previous commit, but pytype completely missed it. PyCharm is correct here, but I'm committing this separately to highlight this potential problem- some of the implementations don't implement _all_ of the methods the others do, and there's not a great way to indicate on a protocol class that a method or attribute is optional- that's kinda the opposite of what static typing is about. Making the method an `Optional[Callable]` attribute works here, and keeps both PyCharm and pytype happy, and the generated `mdiff.pyi` and `modules.pyi` look reasonable. We might be getting a little lucky, because the method isn't invoked directly- it is returned from another method that selects which block function to use. Except since it is declared on the protocol class, every module needs this attribute (in theory, but in practice this doesn't seem to be checked), so the check for it on the module has to change from `hasattr()` to `getattr(..., None)`. We defer defining the optional attrs to the type checking phase as an extra precaution- that way it isn't an attr with a `None` value at runtime if someone is still using `hasattr()`. As to why pytype missed this, I have no clue. The generated `mdiff.pyi` even has the global variable typed as `bdiff: intmod.BDiff`, so uses of it really should comply with what is on the class, protocol class or not.

interfaces: introduce and use a protocol class for the `bdiff` module This is allowed by PEP 544[1], and we basically follow the example there. The class here is copied from `mercurial.pure.bdiff`, and the implementation removed. There are several modules that have a few different implementations, and the implementation chosen is controlled by `HGMODULEPOLICY`. The module is loaded via `mercurial/policy.py`, and has been inferred by pytype as `Any` up to this point. Therefore it and PyCharm were blind to all functions on the module, and their signatures. Also, having multiple instances of the same module allows their signatures to get out of sync. Introducing a protocol class allows the loaded module that is stored in a variable to be given type info, which cascades through the various places it is used. This change alters 11 *.pyi files, for example. In theory, this would also allow us to ensure the various implementations of the same module are kept in alignment- simply import the module in a test module, attempt to pass it to a function that uses the corresponding protocol as an argument, and run pytype on it. In practice, this doesn't work (yet). PyCharm (erroneously) flags imported modules being passed where a protocol class is used[2]. Pytype has problems the other way- it fails to detect when a module that doesn't adhere to the protocol is passed to a protocol argument. The good news is that mypy properly detects this case. The bad news is that mypy spews a bunch of other errors when importing even simple modules, like the various `bdiff` modules. Therefore I'm punting on the tests for now because the type info around a loaded module in PyCharm is a clear win by itself. [1] https://peps.python.org/pep-0544/#modules-as-implementations-of-protocols [2] https://youtrack.jetbrains.com/issue/PY-58679/Support-modules-implementing-protocols

mdiff: tweak calls into `bdiff.fixws` to match its type hints It turns out that protocol classes can be used for modules too, which is great because all of the dynamically loaded modules (and their attributes) are currently inferred as `Any`. See the next commit for details. A protocol class for the `bdiff` module detected this (trivial) mismatch, so correct it first. The various implementations of this method are typed as taking a `bool`. The `cext` implementation parses its arguments with `PyArg_ParseTuple(args, "Sb:fixws", &s, &allws)`, which wants an `int`. But experimenting in `hg debugshell` under py38, passing `True` or `False` to `cext.fixws()` also works. We can change the implementation to use "p" (which was introduced in py33) instead of "b", but that's beyond the scope of this.

util: minor copy editing of the documentation for `mmapread()`

util: make `mmapread()` work on Windows again 522b4d729e89 started referencing `mmap.MAP_PRIVATE`, but that's not available on Windows, so `hg version` worked, but `make local` did not. That commit also started calling the constructor with the fine-grained `flags` and `prot` args, but those aren't available on Windows either[1] (though the backing C code doesn't seem conditionalized to disallow usage of them). I assume the change away from from the `access` arg was to provide the same options, plus `MAP_POPULATE`. Looking at the source code[2], they're not quite the same- `ACCESS_READ` is equivalent to `flags = MAP_SHARED` and `prot = PROT_READ`. `MAP_PRIVATE` is only used with `ACCESS_COPY`, which allows read and write. Therefore, we can't quite get the same baseline flags on Windows, but this was the status quo ante and `MAP_POPULATE` is a Linux thing, so presumably it works. I realize that typically the OS differences are abstracted into the platform modules, but I'm leaving it here so that it is obvious what the differences are between the platforms. [1] https://docs.python.org/3/library/mmap.html#mmap.mmap [2] https://github.com/python/cpython/blob/5e0abb47886bc665eefdcc19fde985f803e49d4c/Modules/mmapmodule.c#L1539

typing: add type annotations to the dirstate classes The basic procedure here was to use `merge-pyi` to merge the `git/dirstate.pyi` file in (after renaming the interface class to match), cleaning up the import statement mess, and then repeating the procedure for `mercurial/dirstate.pyi`. Surprisingly, git's dirstate had more hints inferred in its *.pyi file. After that, it was a manual examination of each method in the interface, and how they were implemented in the core and git classes to verify what was inferred by pytype, and fill in the missing gaps. Since this involved jumping around between three different files, I applied the same type info to all three at the same time. Complex types I rolled up into type aliases in the interface module, and used that as needed. That way if it changes, there's one place to edit. There are some hints still missing, and some documentation that doesn't match the signatures. They should all be marked with TODOs. There are also a bunch of methods on the core class that aren't on the Protocol class that seem like maybe they should be (like `set_tracked()`). There are even more methods missing from the git class. But that's a project for another time.

interfaces: change a couple of dirstate fields to `@property` As I was adding type hints here and to the concrete classes, PyCharm flagged the property in the core class as not being compatible with the base class's version.

git: make `dirstate.parents()` return a list like the core class The core class returned a list, so that's how I type annotated it, and this got flagged. I suppose we could annotate it as a `Sequence[bytes]`, but it's a trivial difference.

typing: add type hints for the overloads of `matchmod.readpatternfile()` The return type is conditional on an argument passed, and it very much confused both pytype and PyCharm inside `dirstate._ignorefileandline()` after adding type hints for the return value there.

dirstate: subclass the new dirstate Protocol class Behold the chaos that ensues. We'll use the generated *.pyi files to apply type annotations to the interface, and see how much agrees with the documentation. Since the CamelCase name was used to try to work around pytype issues with zope interfaces and is a new innovation this cycle (see c1d7ac70980b), drop the CamelCase name. I think the Protocol classes *should* be CamelCase, but that can be done later in one pass. For now, the CamelCase alias is extra noise in the *.pyi files.

git: correct some signature mismatches between dirstate and the Protocol class These were flagged by PyCharm when subclassing the Protocol class. Note that both `is_changing_xxx` were only flagged when the Protocol class used a plain field, as mentioned in the previous commit. After converting those attrs in the Protocol class to @property to match the regular dirstate class, it stopped flagging these. But I don't think that makes sense- `@property` should look like an attribute to the outside world, not a callable.

interfaces: convert the zope `Attribute` attrs to regular fields At this point, we should have a useful protocol class. The file syntax requires the type to be supplied for any fields that are declared, but we'll leave the complex ones partially unspecified for now, for simplicity. (Also, the things documented as `Callable` are really as future type annotating worked showed- roll with it for now, but they're marked as TODO for fixing later.) All of the fields and all of the attrs will need type annotations, or the type rules say they are considered to be `Any`. That can be done in a separate pass, possibly applying the `dirstate.pyi` file generated from the concrete class. The first cut of this turned the `interfaceutil.Attribute` fields into plain fields, and thus the types on them. PyCharm flagged a few things as having incompatible signatures when the concrete dirstate class subclassed this, when the concrete class has them declared as `@property`. So they've been changed to `@property` here in those cases. The remaining fields that are decorated in the concrete class have comments noting the differences. We'll see if they need to be changed going forward, but leave them for now. We'll be in trouble if the `@util.propertycache` is needed, because we can't import that module here at runtime, due to circular imports.

interfaces: add the missing `self` arg to the dirstate Protocol class This clears all of the errors that PyCharm has been flagging in this file, since the zope interface was declared here.

interfaces: convert the dirstate zope interface to a Protocol class This is a small trial run for converting the repository interfaces enmasse, in the same series of steps. I'm not sure that this current code is valid (it has zope attribute fields, and it's missing all of the `self` args on its functions, but that was the previous state of things, and made PyCharm really unhappy). But it will be easier to review the repository interface changes if this change is separate from adding `self` and dropping the zope attributes all over. Having an empty constructor in a protocol is weird. I'm not sure if these args should be converted to fields that all subclasses would have, and comments around existing attributes say some should be going away. Comment it out for now so that it's not in the way, but also not forgotten.

tests: disable `test-check-interfaces.py` while converting to protocols The goal is to convert everything, so get it all out of the way. The interfaces don't get that much maintenance that this needs to be tested right now.

tests: always access the mercurial repo through `helpers-testrepo.sh` In some contexts the mercurial repo needs to be accessed through system hg. That's what `helpers-testrepo.sh` enforces, but some tests incorrectly use the mercurial repo without going through that script. This patch fixes those tests.

tests: in helpers-testrepo.sh switch from shell aliases to functions The reason is that I want this script to work in non-interactive shells too. (will be used in the next commit)

rust: fix the deprecation warning in NaiveDateTime::from_timestamp This warning appears between chrono 0.4.34 and 0.4.38, so isn't affecting the current lock file, but it would come when we upgraded the version.

run-tests: ensure that --no-rust do not use rust Since having a valid value in HGWITHRUSTEXT is enough to trigger the use of rust, we need to unset it before install to be sure.

revlogutils: teach issue6528 filtering about grandparents During dynamic filtering, we should assume that the current repository is correct. Therefore the parents of the delta base can tell us if that parent has metadata without having to build the whole text.

revlogutils: remember known metadata parents for issue6528 In the cases where the parent revs tell us for sure that the parent has metadata, remember this fact to avoid content recomputations later.

revlogutils: for issue6528 fix, pre-cache nullrev as metadata-free

revlogutils: for issue6528 fix, cache results for null changes

revlogutils: fix _chunk() reference _chunk is only found in the inner revlog object and not directly exposed outside.

rev-branch-cache: reenable memory mapping of the revision data Now that we are no longer truncating it, we can mmap it again. This provide a sizeable speedup on repository with a very large amount of revision for example for a mozilla-try clone with 5 793 383 revisions, this provide a speedup of 5ms - 10ms. Since they happens within the "critical" locked path during push. These miliseconds are important. In addition, the v3 branchmap format is use the rev-branch-cache more than the v2 branchmap cache so this will be important. On smaller repository we consistently see an improvement of one or two percents, but the gain in absolute time is usually < 10 ms. #### benchmark.name = hg.command.unbundle # benchmark.variants.issue6528 = disabled # benchmark.variants.reuse-external-delta-parent = yes # benchmark.variants.revs = any-1-extra-rev # benchmark.variants.source = unbundle # benchmark.variants.verbosity = quiet ### data-env-vars.name = mozilla-try-2024-03-26-zstd-sparse-revlog ## bin-env-vars.hg.flavor = default e51161b12c7e: 3.527923 ebdcfe85b070: 3.468178 (-1.69%, -0.06) ## bin-env-vars.hg.flavor = rust e51161b12c7e: 3.580158 ebdcfe85b070: 3.480564 (-2.78%, -0.10) ### data-env-vars.name = mozilla-try-2024-03-26-ds2-pnm ## bin-env-vars.hg.flavor = rust e51161b12c7e: 3.527923 ebdcfe85b070: 3.468178 (-1.69%, -0.06)

rev-branch-cache: have debugupdatecache warm rbc too Since the "v2" format can be more performant than the "v1" format (thanks to mmap), it is useful to be able to make sure it is present

rev-branch-cache: schedule a write of the "v2" format if we read from "v1" The new file can be memorymapped, while the old one cannot. So there is value in having the v2 format around as soon a possible.

rev-branch-cache: fallback on "v1" data if no v2 is found This will help smooth the transition to the v2 format for existing large repository.

rev-branch-cache: increment the version to "v2" We want to ensure no older clients will truncate the file under us. So we need to change their name. We don't change the rest of the format (unfortunaly).

rev-branch-cache: stop truncating cache file Truncating the file prevent the safe use of mmap. So instead of overwrite the existing data. If more than 20% of the file is to be overwritten, we rewrite the whole file instead. Such whole rewrite is done by replacing the old one with a new one, so mmap of the old file would be affected. This prepare a more aggressive use of mmap in later patches.

rev-branch-cache: make sure we close the name file we open We were various opening without with or try. Adding a try would not hurt.

rev-branch-cache: add a way to force rewrite of the cache This seems useful to be able to do this, for example during strip. This align with the intended expressed in the `test-branches.t` test. This will help use being more confident about future changes in the series.

rev-branch-cache: issue more truthful "truncating" message First, don't pretend it truncate to 40 when it actually truncate to 0. Second, don't pretend to truncate to 0 when the file is already empty/missing.

rev-branch-cache: move the code in a dedicated module The branchmap module is getting huge and the rev branch cache is fully independent, lets move it elsewhere.

statichttprepo: stop shadowing the `bytes` builtin PyCharm flagged it, but I also misunderstood when looking at the code, because the name implied a byte string, not a number.

statichttprepo: fix `httprangereader.read()` for py3 It looks like there were a bunch of problems, not all of them py3 related: 1) The signature of BinaryIO.read() is -1, not None 2) The `end` variable can't be bytes and interpolate into str with "%s" 3) The `end` variable can't be an int and interpolate into str with "%s" 4) The result slicing could be out of bounds if more is requested than returned I guess if somebody would have called `read(-1)` (either directly or because a wrapper defaults to that), it wouldn't have been handled correctly. The fact that it is a valid value meaning to read everything requires some additional changes later in the method around when it slices the byte string that was read, but that seems to have already been broken.

statichttprepo: use a context manager to handle a file descriptor I'm not sure if this should be reduced to `vfs.exists()`. That would seem to be equivalent code (since the result of the read is ignored, so we can't tell if the file actually has content, which has been the state of things going back to 98b6c3dde237), but this is at least safer file descriptor handling.

profiling: pass bytes to `_()` and `error.Abort()` And of course `other_tool_name` is str too, so that needs to be converted. The type hints from PyCharm say `sys.monitoring.get_tool()` can return None, so handle that case explicitly before it trips up pytype.

exchange: improve computation of relevant markers for large repos Compute the candidate nodes with relevant markers directly from keys of the predecessors/successors/children dictionaries of obsstore. This is faster than iterating over all nodes directly. This test could be further improved for repositories with relative few markers compared to the repository size, but this is no longer hot already. With the current loop structure, the obshashrange use works as well as before as it passes lists with a single node. Adjust the interface by allowing revision lists as well as node lists. This helps cases that computes ancestors as it reduces the materialisation cost. Use this in _pushdiscoveryobsmarker and _getbundleobsmarkerpart. Improve the latter further by directly using ancestors(). Performance benchmarks show notable and welcome improvement to no-op push and pull (that would also apply to other push/pull). This apply to push and pull done without evolve. ### push/pull Benchmark parameter # bin-env-vars.hg.flavor = default # benchmark.variants.explicit-rev = none # benchmark.variants.protocol = ssh # benchmark.variants.revs = none ## benchmark.name = hg.command.pull # data-env-vars.name = mercurial-devel-2024-03-22-zstd-sparse-revlog before: 5.968537 seconds after: 5.668507 seconds (-5.03%, -0.30) # data-env-vars.name = tryton-devel-2024-03-22-zstd-sparse-revlog before: 1.446232 seconds after: 0.835553 seconds (-42.23%, -0.61) # data-env-vars.name = netbsd-src-draft-2024-09-19-zstd-sparse-revlog before: 5.777412 seconds after: 2.523454 seconds (-56.32%, -3.25) ## benchmark.name = hg.command.push # data-env-vars.name = mercurial-devel-2024-03-22-zstd-sparse-revlog before: 6.155501 seconds after: 5.885072 seconds (-4.39%, -0.27) # data-env-vars.name = tryton-devel-2024-03-22-zstd-sparse-revlog before: 1.491054 seconds after: 0.934882 seconds (-37.30%, -0.56) # data-env-vars.name = netbsd-src-draft-2024-09-19-zstd-sparse-revlog before: 5.902494 seconds after: 2.957644 seconds (-49.89%, -2.94) There is not notable different in these result using the "rust" flavor instead of the "default". The performance impact on the same operation when using evolve were also tested and no impact was noted.

typing: make the localrepo classes known to pytype 9d4ad05bc91c and 1b17309cdaab both mentioned making `bundlerepository` and `unionrepository` subclass `localrepository` during the type checking phase, but that didn't apply to pytype in practice. See bcaa5d408657 and friends for how the zope interfaces confuse pytype, and end up converting the classes they decorate into `Any`. This commit is slightly more complex though, because `localrepository` has mixin classes applied to it when it is instantiated. Specifically, `RevlogFileStorage` is added, which adds `def file(f)` (which isn't defined on `localrepository`). Therefore a list of `localrepository` superclasses is provided during type checking to account for the mixins. Without this, the `bundlerepository` class gets flagged when it attempts to call its superclass implementation of `file()`. Note that pytype doesn't understand these mixin superclasses (it marks the superclass of `localrepository` as `Any`, because they are zope interfaces it doesn't understand), but that's enough to get it to not flag `bundlerepository`. PyCharm also stops flagging it as a missing function, though it seems like it is able to handle the zope interfaces.

typing: add a handful more annotations to `mercurial/vfs.py` These came out of refactoring into a protocol class, but they can stand on their own. The `audit` callback is kinda screwy because the internal lambda and the callable for `pathutil.pathauditor` have different args and a different return type. It's conditionalized where it is called, and can be cleaned up later if desired.

typing: make `vfs.isfileorlink_checkdir()` path arg required The only caller to this is `merge._checkunknownfile()`, which supplies a value. That's good, because `util.localpath()` immediately uses the value to call a method on it on Windows. The posix implementation returns the value unaltered, but then `pathutil.finddirs_rev_noroot()` would have exploded.

typing: manually add type annotations to `mercurial/vfs.py` This isn't everything, but hopefully it's close enough to hack on a protocol class.

typing: correct pytype mistakes in `mercurial/vfs.py` With the previous changes in this series (prior to merging the *.pyi file), this wasn't too bad- the only definitively wrong things were the `data` argument to `writelines()`, and the return type on `backgroundclosing()` (both of these errors were dropped in the previous commit; for some reason pytype doesn't like `contextlib._GeneratorContextManager`, even though that's what it determined it is): File "/mnt/c/Users/Matt/hg/mercurial/vfs.py", line 411, in abstractvfs: Bad return type 'contextlib._GeneratorContextManager' for generator function abstractvfs.backgroundclosing [bad-yield-annotation] Expected Generator, Iterable or Iterator PyCharm thinks this is `Generator[backgroundfilecloser], Any, None]`, which can be reduced to `Iterator[backgroundfilecloser]`, but pytype flagged the line that calls `yield` without an argument unless it's also `Optional`. PyCharm is happy either way. For some reason, `Iterable` didn't work for pytype: File "/mnt/c/Users/Matt/hg/mercurial/vfs.py", line 390, in abstractvfs: Function contextlib.contextmanager was called with the wrong arguments [wrong-arg-types] Expected: (func: Callable[[Any], Iterator]) Actually passed: (func: Callable[[Any, Any, Any], Iterable[Optional[Any]]]) Attributes of protocol Iterator[_T_co] are not implemented on Iterable[Optional[Any]]: __next__

typing: run `merge-pyi` on `mercurial/vfs.py` The *.pyi file was generated with pytype 2023.11.21. There were a few things here that were wrong (e.g. `writelines()` takes an `Iterable[bytes]`, not `bytes`, or inexplicable errors like importing several of the vfs classes from this very module), and those changes have been dropped manually here.

typing: add type annotations to `mercurial.util.makelock()` This bubbles up into the `vfs` classes, so get this out of the way.

util: avoid a leaked file descriptor in `util.makelock()` exceptional case

typing: add type annotations to the `mercurial.util.filestat` class It's referenced in the `vfs` classes, so get this out of the way to help there. The `TypeVar` definition and its usage was copied from the existing `util.pyi` file.

vfs: do minor copyediting on comments and doc strings These were flagged by PyCharm, so clear them from the gutter.

vfs: simplify the `abstractvfs.rename()` implementation PyCharm was yapping about `util.rename()` not returning anything, because it is typed to return `None`, but the value was captured and returned after calling `_avoidambig()`. Instead, drop all of that, unconditionally rename, and then call `_avoidambig()` if appropriate. While we're here, convert the ersatz ternary operator into a modern one to help pytype. When a variable is initialized the old way, pytype tends to assign the type of the LHS of the `and`. In this case, that's a bool, and it will get confused that bool doesn't have a `stat` attribute once this method gets more type annotations. (Currently it thinks the `checkambig` arg is `Any`, so it doesn't care.)

vfs: use @abstractmethod instead of homebrewing abstract methods The latter confuses PyCharm after adding more type annotations when, for example, `abstractvfs.rename()` calls `_auditpath()`- the latter unconditionally raised an error, so PyCharm thought the code that came after is unreachable. It also tricked pytype into marking the return type as `Never`, which isn't available until Python 3.11 (outside of `typing_extensions`). This also avoid PyCharm warnings that the call to the superclass constructor was missed (it couldn't be called because it raised an error to prevent instantiation). The statichttprepo module needed to be given an override for one of the abstract methods, so that it can be instantiated. In `abstractvfs`, this method is only called by `rename()`, so I think we can leave this empty. We raise an error in case somebody accidentally calls it in the future- it would have raised this same error prior to this change. I couldn't wrangle `import-checker.py` into accepting importing `ABC` and `abstractmethod`- for each subsequent import, it reports something like: stdlib import "contextlib" follows local import: abc I suspect the problem is that near the `if fullname != '__future__'` check, if the module doesn't fall into the error case, `seenlocal` gets set to the module name. That causes it to be treated like a local module on the next iteration, even though it is in `stdlib_modules`.

vfs: modernize the detection of the main thread There weren't a lot of good choices when py27 was supported, but starting with py34, `threading.main_thread()` is available. This gets us away from an undocumented, internal symbol, and drops a pytype suppression statement. It is also apparently no longer reliable after a process fork.[1][2] [1] https://stackoverflow.com/a/23207116 [2] https://github.com/python/cpython/blob/v3.6.3/Lib/threading.py#L1334

store: fix a signature mismatch for a vfs subclass This was flagged by PyCharm. I'm not sure why pytype doesn't catch this- it's not excluded from the modules that are currently checked.

lfs: fix various signature mismatches for vfs subclasses These were flagged by PyCharm. I'm not sure why pytype doesn't catch these- only `hgext/lfs/__init__.py` in the lfs extension is excluded from being checked. I'm not sure if the `*insidef` arg to `join()` was meant as an internal convencience, because I see another class that gets flagged for the same signature problem (to be fixed next). But I don't feel bold enough to make this an internal function, and provide a simplified public `join()` on the `vfs` classes. That can still be done later, if desired. For now, process the additional args and pass them along, even though there don't appear to be any current callers that provide extra args to these classes. We need all of the subclasses to agree on the signature, or they won't be considered to implement the `Vfs` protocol being developed. While we're copy/pasting from the base class, bring the type annotations along for the ride.

util: add a comment to suppress a PyCharm warning about a PEP 8 violation Slowly trying to get rid of silly warnings, so that real problems aren't hidden.

keepalive: fix a signature mismatch for a http.client.HTTPResponse subclass Also flagged by PyCharm. This is checked by pytype too, so I'm not sure why it misses this. I verified in py36 that this argument is documented for the function, so maybe this is py2 legacy.

cbor: drop a duplicate dictionary initialization entry Flagged by PyCharm.

profiling: document the py-spy value for `profiling.type` The feature was not visible otherwise.

tests: enable pytype checking on `mercurial/unionrepo.py`

unionrepo: fix mismatches with revlog classes This is a subset of cfd30df0f8e4, applied to `unionrepository`. There are none of the `write()` method overrides here, like `bundlerepository`. With these changes, pytype flags the `unionrevlog` constructor: File "/mnt/c/Users/Matt/hg/mercurial/unionrepo.py", line 55, in __init__: No attribute '_revlog' on mercurial.changelog.changelog [attribute-error] Called from (traceback): line 207, in __init__ File "/mnt/c/Users/Matt/hg/mercurial/unionrepo.py", line 55, in __init__: No attribute '_revlog' on mercurial.revlog.revlog [attribute-error] Called from (traceback): line 232, in __init__ But it turns out that both `changelog.changelog` and `revlog.revlog` do have a `target` attribute, so they wouldn't trip over this. It seems weird that the second caller to be flagged is passing the private `_revlog`, but maybe that's how it needs to be.

typing: make `unionrepository` subclass `localrepository` while type checking This is the same change as 9d4ad05bc91c made for `bundlerepository`, for the same reasons. Also, add a comment here to suppress the PyCharm warning that the superclass constructor is not called, that is new now that there's a simulated superclass. That lack of a call is by design- `makeunionrepository()` does magic that PyCharm isn't aware of. But PyCharm has been better at catching problems than pytype in a lot of cases, so I'd like to reduce the bogus things it flags, to make the real issues stand out.