tests: stabilize `test-clonebundles.t` on Windows The `remote: {foo,bar}` lines were different because `echo` in MSYS uses `\r\n`. I couldn't make it work with the previous echoing of individual lines, changing the internal `echo` to `printf "foo\n"`, because that output as "foon". This works on Linux and Windows, so I'm not thinking too hard about it.

tests: cleanup some special casing of `seq` emitting '\r\n' My guess is these predate the commit referenced in the previous commit.

tests: force `seq` to print with '\n' EOL It looks like consistent EOL is the reason for 0605726179a0, but now on py3, `print()` uses the platform EOL without regard to binary mode. The tests mostly use this to loop over a sequence of number in the shell, but there are a handful that redirect output to a file. Specifically, this fixes Windows runs of `test-bundle2-multiple-changegroups.t`, but there may be other tests this fixes. Some other `tests/*.py` files also set binary mode on stdout, but they also write bytes directly to `sys.stdout.buffer`. I'm not doing that here because PyCharm flags these write calls for passing bytes instead of str (PyCharm is likely wrong, but possibly confused because the code falls back to `sys.stdout` if there is no `.buffer` attribute), and it's annoying.

tests: stabilize `test-bundle-phase-internal.t` on Windows This is a silly difference, and we control the Windows side of things from mercurial/windows.py:195. I'll swap and quote the values to be like `no-windows` at some point, but I suspect this exception output would appear elsewhere, and don't feel like waiting for a 2h+ test run to find all of them.

tests: skip `test-wsgicgi.t` on MSYS The test is attempting to set `PATH_INFO="/rev/\xe2\x80\x94"` into the environment, which it does. The problem is that when MSYS sees a leading '/' in an environment variable, it thinks it's a unix filesystem path, so it "helpfully" prepends the Windows path to the MSYS root directory before running a non-MSYS process. hgweb would then split this value on '/', so it would get 'C:' instead of 'rev', and return a 400 since that isn't a valid web command. I tried generating a *.bat file, but had trouble running that via `cmd.exe` inside the test. I also tried generating an equivalent *.py launcher that would set the environment variables itself. But there is no `os.environb` on Windows, and the value was getting mangled when put into the script. So, I give up. If it's encoding stuff on Windows, it's probably broken.

clonebundles: allow manifest to specify sha256 digest of bundles

install: add long_description_content_type

headrevs: replace a boolean match with a if/else I missed that while doing a previous cleanup.

head-revs: add a native implementation of the `stop_rev` parameter This does not add too much complexity to the native code and help with branchmap v3 performance. Note that the final conversion of the heads from native-code to Python is still too costly, especially in Rust. In addition the current caching around headrevs is too simple and fragile. However these are an unrelated problem. ### benchmark.name = hg.command.unbundle # bin-env-vars.hg.py-re2-module = default # benchmark.variants.issue6528 = disabled # benchmark.variants.resource-usage = default # benchmark.variants.reuse-external-delta-parent = yes # benchmark.variants.revs = any-1-extra-rev # benchmark.variants.source = unbundle # benchmark.variants.validate = default # benchmark.variants.verbosity = quiet ## data-env-vars.name = netbeans-2018-08-01-zstd-sparse-revlog # bin-env-vars.hg.flavor = default branch-v2: 0.233711 ~~~~~ branch-v3 before: 0.239857 (+2.63%, +0.01) branch-v3 after: 0.239558 (+2.50%, +0.01) # bin-env-vars.hg.flavor = rust branch-v2: 0.235230 ~~~~~ branch-v3 before: 0.240972 (+2.44%, +0.01) branch-v3 after: 0.239917 (+1.99%, +0.00) ## data-env-vars.name = netbeans-2018-08-01-ds2-pnm # bin-env-vars.hg.flavor = rust branch-v2: 0.255586 ~~~~~ branch-v3 before: 0.268560 (+5.08%, +0.01) branch-v3 after: 0.262261 (+2.61%, +0.01) ## data-env-vars.name = mozilla-central-2024-03-22-zstd-sparse-revlog # bin-env-vars.hg.flavor = default branch-v2: 0.339010 ~~~~~ branch-v3 before: 0.349389 (+3.06%, +0.01) branch-v3 after: 0.348247 (+2.72%, +0.01) # bin-env-vars.hg.flavor = rust branch-v2: 0.346525 ~~~~~ branch-v3 before: 0.355661 (+2.64%, +0.01) branch-v3 after: 0.350906 (+1.26%, +0.00) ## data-env-vars.name = mozilla-central-2024-03-22-ds2-pnm # bin-env-vars.hg.flavor = rust branch-v2: 0.380202 ~~~~~ branch-v3 before: 0.408851 (+7.54%, +0.03) branch-v3 after: 0.406511 (+6.92%, +0.03) ## data-env-vars.name = mozilla-unified-2024-03-22-zstd-sparse-revlog # bin-env-vars.hg.flavor = default branch-v2: 0.412165 ~~~~~ branch-v3 before: 0.427782 (+3.79%, +0.02) branch-v3 after: 0.422595 (+2.53%, +0.01) # bin-env-vars.hg.flavor = rust branch-v2: 0.412397 ~~~~~ branch-v3 before: 0.422354 (+2.41%, +0.01) branch-v3 after: 0.421079 (+2.11%, +0.01) ## data-env-vars.name = mozilla-unified-2024-03-22-ds2-pnm # bin-env-vars.hg.flavor = rust branch-v2: 0.429501 ~~~~~ branch-v3 before: 0.443197 (+3.19%, +0.01) branch-v3 after: 0.449432 (+4.64%, +0.02) ## data-env-vars.name = mozilla-try-2024-03-26-zstd-sparse-revlog # bin-env-vars.hg.flavor = default branch-v2: 3.403171 ~~~~~ branch-v3 before: 3.819477 (+12.23%, +0.42) branch-v3 after: 3.658482 (+7.50%, +0.26) # bin-env-vars.hg.flavor = rust branch-v2: 3.454876 ~~~~~ branch-v3 before: 3.590284 (+3.92%, +0.14) branch-v3 after: 3.545843 (+2.63%, +0.09) ## data-env-vars.name = mozilla-try-2024-03-26-ds2-pnm # bin-env-vars.hg.flavor = rust branch-v2: 3.465435 ~~~~~ branch-v3 before: 3.633278 (+4.84%, +0.17) branch-v3 after: 3.556074 (+2.62%, +0.09)

head-revs: move hg-core's inner_headrevsfiltered closer to inner_headrevs This is pure code movement, it make a coming changesets significantly clearer.

branchmap-v3: introduce a "stop_rev" argument to `headsrevs` The `headsrevs` method of the revlog already have a `revs` argument to compute the headrevs of a limited set of heads. However, it disable the use of the native compiled code to compute the heads, which slows down the branchmap v3 code a lot. The branchmap v3 usage is actually quite constrained as we will always only ignores a part at the top of the graph. So we could be significantly faster. We start by making small change to the python side to improve the situation and introduce the new API. More collaboration with the native code are coming later. This massively speedup operation and close most of the remaining gaps between branchmap-v3 and branchmap-v2. especially on repository with many revs like mozilla-try. A small overhead remains mostly because the `headrevs` logic currently has some inefficiently. We will look into them from there. ### benchmark.name = hg.command.unbundle # bin-env-vars.hg.py-re2-module = default # benchmark.variants.issue6528 = disabled # benchmark.variants.resource-usage = default # benchmark.variants.reuse-external-delta-parent = yes # benchmark.variants.revs = any-1-extra-rev # benchmark.variants.source = unbundle # benchmark.variants.validate = default # benchmark.variants.verbosity = quiet ## data-env-vars.name = netbeans-2018-08-01-zstd-sparse-revlog # bin-env-vars.hg.flavor = default branch-v2: 0.233711 ~~~~~ branch-v3 before: 0.368769 (+57.79%, +0.14) branch-v3 after: 0.239857 (+2.63%, +0.01) # bin-env-vars.hg.flavor = rust branch-v2: 0.235230 ~~~~~ branch-v3 before: 0.372460 (+58.34%, +0.14) branch-v3 after: 0.240972 (+2.44%, +0.01) ## data-env-vars.name = netbeans-2018-08-01-ds2-pnm # bin-env-vars.hg.flavor = rust branch-v2: 0.255586 ~~~~~ branch-v3 before: 0.318907 (+24.78%, +0.06) branch-v3 after: 0.268560 (+5.08%, +0.01) ## data-env-vars.name = mozilla-central-2024-03-22-zstd-sparse-revlog # bin-env-vars.hg.flavor = default branch-v2: 0.339010 ~~~~~ branch-v3 before: 0.349752 (+3.17%, +0.01) branch-v3 after: 0.349389 (+3.06%, +0.01) # bin-env-vars.hg.flavor = rust branch-v2: 0.346525 ~~~~~ branch-v3 before: 0.354300 (+2.24%, +0.01) branch-v3 after: 0.355661 (+2.64%, +0.01) ## data-env-vars.name = mozilla-central-2024-03-22-ds2-pnm # bin-env-vars.hg.flavor = rust branch-v2: 0.380202 ~~~~~ branch-v3 before: 0.396293 (+4.23%, +0.02) branch-v3 after: 0.408851 (+7.54%, +0.03) ## data-env-vars.name = mozilla-unified-2024-03-22-zstd-sparse-revlog # bin-env-vars.hg.flavor = default branch-v2: 0.412165 ~~~~~ branch-v3 before: 0.424769 (+3.06%, +0.01) branch-v3 after: 0.427782 (+3.79%, +0.02) # bin-env-vars.hg.flavor = rust branch-v2: 0.412397 ~~~~~ branch-v3 before: 0.421796 (+2.28%, +0.01) branch-v3 after: 0.422354 (+2.41%, +0.01) ## data-env-vars.name = mozilla-unified-2024-03-22-ds2-pnm # bin-env-vars.hg.flavor = rust branch-v2: 0.429501 ~~~~~ branch-v3 before: 0.443849 (+3.34%, +0.01) branch-v3 after: 0.443197 (+3.19%, +0.01) ## data-env-vars.name = mozilla-try-2024-03-26-zstd-sparse-revlog # bin-env-vars.hg.flavor = default branch-v2: 3.403171 ~~~~~ branch-v3 before: 6.234055 (+83.18%, +2.83) branch-v3 after: 3.819477 (+12.23%, +0.42) # bin-env-vars.hg.flavor = rust branch-v2: 3.454876 ~~~~~ branch-v3 before: 6.307813 (+82.58%, +2.85) branch-v3 after: 3.590284 (+3.92%, +0.14) ## data-env-vars.name = mozilla-try-2024-03-26-ds2-pnm # bin-env-vars.hg.flavor = rust branch-v2: 3.465435 ~~~~~ branch-v3 before: 5.176076 (+49.36%, +1.71) branch-v3 after: 3.633278 (+4.84%, +0.17)

branchmap-v3: filter topo heads using node for performance reason The branchmap currently contains heads as nodeid. If we build a set of revnum with the topological heads, we need to turn the nodeid in the branchmap to revnum to be able to check if they are topo-heads. That nodeid → revnum lookup is "expensive" and adds up to something noticeable if you do it hundreds of thousand of time. Instead we turn all the topo-heads revnums into nodes and build a set. So we can directly test membership of the nodeids stored in the branchmap. That is much faster. Ideally we would have revnum in the branchmap and could directly test revnum against a revnum set and that would be even faster. However that's an adventure for another time. Without this change, the branchmap format "v3" was significantly slower than the "v2" format. With this changes, some of that gap is recovered With rust + persistent nodemap, this overhead was smaller because the extra lookup did not had to to build the nodemap from scratch. In addition the mozilla-unified repository is able to use the "pure_top" mode of branchmap v3, so it was not really affected by this. Future changeset will work of the remaining of the performance gap. ### benchmark.name = hg.command.unbundle # bin-env-vars.hg.py-re2-module = default # benchmark.variants.issue6528 = disabled # benchmark.variants.resource-usage = default # benchmark.variants.reuse-external-delta-parent = yes # benchmark.variants.revs = any-1-extra-rev # benchmark.variants.source = unbundle # benchmark.variants.validate = default # benchmark.variants.verbosity = quiet ## data-env-vars.name = netbeans-2018-08-01-zstd-sparse-revlog # bin-env-vars.hg.flavor = default branch-v2: 0.233711 ~~~~~ branch-v3 before: 0.380994 (+63.02%, +0.15) branch-v3 after: 0.368769 (+57.79%, +0.14) # bin-env-vars.hg.flavor = rust branch-v2: 0.235230 ~~~~~ branch-v3 before: 0.385060 (+63.70%, +0.15) branch-v3 after: 0.372460 (+58.34%, +0.14) ## data-env-vars.name = netbeans-2018-08-01-ds2-pnm # bin-env-vars.hg.flavor = rust branch-v2: 0.255586 ~~~~~ branch-v3 before: 0.317524 (+24.23%, +0.06) branch-v3 after: 0.318907 (+24.78%, +0.06) ## data-env-vars.name = mozilla-central-2024-03-22-zstd-sparse-revlog # bin-env-vars.hg.flavor = default branch-v2: 0.339010 ~~~~~ branch-v3 before: 0.410007 (+20.94%, +0.07) branch-v3 after: 0.349752 (+3.17%, +0.01) # bin-env-vars.hg.flavor = rust branch-v2: 0.346525 ~~~~~ branch-v3 before: 0.410428 (+18.44%, +0.06) branch-v3 after: 0.354300 (+2.24%, +0.01) ## data-env-vars.name = mozilla-central-2024-03-22-ds2-pnm # bin-env-vars.hg.flavor = rust branch-v2: 0.380202 ~~~~~ branch-v3 before: 0.393871 (+3.60%, +0.01) branch-v3 after: 0.396293 (+4.23%, +0.02) ## data-env-vars.name = mozilla-unified-2024-03-22-zstd-sparse-revlog # bin-env-vars.hg.flavor = default branch-v2: 0.412165 ~~~~~ branch-v3 before: 0.438105 (+6.29%, +0.03) branch-v3 after: 0.424769 (+3.06%, +0.01) # bin-env-vars.hg.flavor = rust branch-v2: 0.412397 ~~~~~ branch-v3 before: 0.438405 (+6.31%, +0.03) branch-v3 after: 0.421796 (+2.28%, +0.01) ## data-env-vars.name = mozilla-unified-2024-03-22-ds2-pnm # bin-env-vars.hg.flavor = rust branch-v2: 0.429501 ~~~~~ branch-v3 before: 0.452692 (+5.40%, +0.02) branch-v3 after: 0.443849 (+3.34%, +0.01) ## data-env-vars.name = mozilla-try-2024-03-26-zstd-sparse-revlog # bin-env-vars.hg.flavor = default branch-v2: 3.403171 ~~~~~ branch-v3 before: 6.562345 (+92.83%, +3.16) branch-v3 after: 6.234055 (+83.18%, +2.83) # bin-env-vars.hg.flavor = rust branch-v2: 3.454876 ~~~~~ branch-v3 before: 6.160248 (+78.31%, +2.71) branch-v3 after: 6.307813 (+82.58%, +2.85) ## data-env-vars.name = mozilla-try-2024-03-26-ds2-pnm # bin-env-vars.hg.flavor = rust branch-v2: 3.465435 ~~~~~ branch-v3 before: 5.381648 (+55.30%, +1.92) branch-v3 after: 5.176076 (+49.36%, +1.71)

rev-branch-cache: properly ignores unaligned trailing data Previously, trailing data could lead to crash and would be written back to disk, disaligning all new data… This is no longer the cases. This was detected while playing with branchmap-v3 that access the rev-branch-cache much more aggressively.

rev-branch-cache: stop pretending we will overwrite data when we don't We were issuing a message about overwriting data even when we were about to write 0 bytes in pratice. This is silly. Instead we point at the extra data remaining in the file (in case someone is using debug to debug something).

rev-branch-cache: fix message about overwritten data If there is data at the end we ignore, we should not count them as overwritten. This reveal that we something don't overwrite anything, this will be taken car of in the next changeset.

branchmap-v3: detect invalid headerline and raise error Otherwise, broken cache file would be accept as valid, but empty.

head-revs: merge the two inner_headrevs… variants Now that there is only one method, it does not make sense to have two different "inner" method. This is especially true as we are about to add another parameter to the method. So we clean up before that.

head-revs: move hg-cpython's inner_headrevsfiltered closer to inner_headrevs This is pure code movement, it make a coming changesets significantly clearer.

head-revs: teach the pure indexes about the `headrevs` method Having this computation done at the index level unify the API and remove revlog side complexity. It might also be a front runner of handing more responsability to the index.

head-revs: remove the `headrevsfiltered` method on the index This method is no longer called and can be removed.

head-revs: stop calling headrevsfiltered The `headrevs` method have been accepting filter since Mercurial 3.2¹. I guess we can rely on it for now. (except the Rust extension that just gained this capability, but it has it now)

rust-cpython: also accept the `filteredrevs` argument in index.headrevs The C version have been accepting this argument since Mercurial 3.2, lets align the Rust index here. This will make it possible to simplify the code in later changesets.

branchmap: use the proper experimental name in cacheutil Otherwise they are not properly copied around.

interfaces: introduce and use a protocol class for the `mpatch` module See f2832de2a46c for details when this was done for the `bdiff` module. Two things worth pointing out- 1) The `cffi` module "inherits" the `pure` implementation of `patchedsize()` because of its wildcard import. 2) It's odd that the `mpatchError` lives in both `pure` and `cext` modules. I initially thought to move the exception into the new class, and make the existing class name an alias to the class in the new location, but the exception is created in C code by the `cext` module, so that won't work. I don't think a protocol class is approriate, because there's nothing special about the class to distinguish from any other `Exception`. Fortunately, nobody is catching this exception in core, so we can kick the can down the road.

branching: merge stable into default

doctest: use the system hg to find the list of file to tests Same as what we do for the `.t` tests, but more manually.

relnotes: skip the test if the source repository is not readable The test want to run the relnot extension, with the tested mercurial, on the original repository. This is not always possible (e.g. when running with --pure and the repository use zstd for example). So we skip the test in this case.

check-code: document reason and suggest alternative to exit code negation The check was introduced in f48b075ff088. We trust the explanation from there.

ci: drop the HGMODULEPOLICY overwrite The test runner have been setting them properly for a while now. So we don't need to do it manually anymore.

test: preserve the existing module policy config for testrepohg resuse The test runner is, righfully, setting HGMODULEPOLICY to match the flavor requests to run the tests. However this mess with the `testrepohg` ability to run properly, as the newly set policy might be incompatible with its installation, or its ability to read the test repo. So we preserve the initial value in a dedicated variable and use it in the `testrepohg` helper.

module-policy: ignore empty module policy This make the variable easier to work with, the empty value is not ambiguous about not wanting to get in the way.

tests: provide an alternate fake lock for filesystems without symlink support

tests: disable `worker.backgroundclose` to stabilize a test on Windows TIL that `worker.enabled=0` doesn't prevent these workers from spinning up. At any rate, there's already a whole lot of conditionalized output following `cat client.log`, the placement of the "starting 4 threads for background file closing" message seems unstable, and we don't care about those worker threads here. Preventing the message is better for test maintenance.

tests: fix lock file path mangling in `test-racy-mutations.t` on Windows I guess `$TESTTMP_FORWARD_SLASH` gets translated by MSYS. This was in the `.foo_commit_out` file: sh: C;C:\\MinGW\\msys\\1.0\\Users\\Matt\\AppData\\Local\\Temp\\hgtests.1qc8jmdl\\child2\\test-racy-mutations.t-skip-detection\\waitlock_editor.sh: $ENOENT

tests: stabilize `test-status-eacces.t` on Windows As noted earlier, `chmod` doesn't complain in MSYS, but also doesn't alter the file permissions such that they are unreadable. I'm guessing the other lines of output in this area that are gated on `rhg` (or not) will also need this, but I don't want to dig too deeply into something that is apparently working well enough.

run-tests: bump the default timeout on Windows to 4x the normal value There are a ridiculous number of tests that timeout on Windows with the 360 sec default (~60). And because of the bug where timed out tests still run to completion before the results are thrown away[1], the timeout does nothing but waste time, so there's no reason to try to find a lower value that still works. For reference on my system: # Ran 909 tests, 116 skipped, 119 failed. python hash seed: 2052473208 real 151m44.322s user 0m0.077s sys 0m0.046s [1] I thought that I wrote a bug for this, but search isn't finding it.

run-tests: bump the minimum python to 3.8 Presumably this was an oversight when hg was updated to 3.8.

tests: stabilize `test-sparse.t` on Windows One of the reserved characters for path values is '*', so it can't be used. Fortunately, missing this seems to not get in the way of any other tests, and it is removed shortly after with `rm -r foo*bar`, and the extant 'foo-bar' matches the pattern.

tests: fix a test hang on Windows when setting a debuglock I have no idea why, but running the `hg -R auto-upgrade debuglock --set-lock` command near the end of `test-upgrade-repo.t` hangs the test. It does background the process and `killdaemons.py` runs without error, but control doesn't return to `run-tests.py` until the process is manually killed. I did notice that `$!` in MSYS is *not* the PID of the process that got backgrounded, even when a simple `sleep 60 &` is run in MSYS without the *.t file. When `killdaemons.py` is run manually with the PID in ProcessExplorer, the backgrounded process terminates immediately, and returns control to `run-tests.py`. This looks like it would be a race, but the test waits 10s for the lock file to appear before attempting to kill the process, so there's time. `hg serve` has a `--pid-file` option to write the pid to the file, but this is only a debug command, so I'm not bothering with cluttering the command line.

tests: conditionalize `chmod` usage in `test-upgrade-repo.t` While the command itself doesn't error out on Windows, it also doesn't make the filesystem readonly. Therefore the repo gets altered to drop dirstate-v2, and puts it out of sync with that happens on Linux.

tests: print the actual timeout value used in `wait-on-file` Previously, it was printing the time passed in, prior to it being scaled up to account for a longer timeout.

tests: stabilize `test-transaction-wc-rollback-race.t` on Windows MSYS has a tendency to munge C:\Dir\SubDir\File into C:DirSubDirFile unless it is quoted, and that's what was happening here- there were a lot of these failures: file not created after 5 seconds: $TESTTMP/transaction-waiting I suspect quoting is only needed in the hook script that is generated (the catting of the log file pointed me in the right direction here), but I missed a spot and trial and error got me here. The quoting elsewhere doesn't harm anything and it was taking 7+ minutes to run this test when things were timing out, so I don't feel like reducing the quoting to the minimum required.

tests: stabilize `test-merge-partial-tool.t` on Windows The test was previously failing because it was opening the shell scripts being used as an executable in a text editor, and problems cascaded from there.

tests: replace `hg id --debug -i` command substitution with non-debug command The censor and convert tests were failing on Windows because the `--debug` flag also prints debug messages, and at least some of these were outputting: skip updating dirstate: identity mismatch ${node} Obviously that causes cascading problems. The other tests were OK, but it's better to use a non debug command for stability.

tests: correct Windows output to account for putting repos in `repo` subdir These were missed in 55c6ebd11cb9, due to being conditionalized and not running in CI.

tests: use pattern matching to mask `ECONNREFUSED` messages The second and third one of these in `test-http-proxy.t` was failing on Windows. The others were found by grep and by failed tests when output was matched and an attempt was made to emit the mask pattern. The first clonebundles failure on Windows emitted: error fetching bundle: [WinError 10061] $ECONNREFUSED$ We should probably stringify that better to get rid of the "[WinError 10061]" part.

typing: add stub functions for `cext/charencoding` I'm not sure if it's better to have a separate file, and currently pytype doesn't really know how to handle these, so it's no help in figuring that out. Technically, these methods are part of the `mercurial.cext.parsers` module, so put them into the existing stub until there's a reason to split it out.

interfaces: introduce and use a protocol class for the `charencoding` module See f2832de2a46c for details when this was done for the `bdiff` module. This lets us dump the hack where the `pure` implementation was imported during the type checking phase to provide signatures for the module methods it provides. Now the protocol classes are starting to shine, because these methods are provided by `pure.charencoding` and `cext.parsers`, and references to `cffi.charencoding` and `cext.charencoding` are forwarded to them as appropriate by the `policy` module. But none of that matters, as long as the module returned provides the listed methods. The interface was copy/pasted from the `pure` module, but `jsonescapeu8fallback` is omitted because it is accessed from the `pure` module directly when the escaping fails in the primary module's `jsonescapeu8()`.

debugantivirusrunning: use bytes when opening a vfs file I noticed this when searching for "base85" to see if anything else in the previous commit needed to be annotated. This was added in 87047efbc6a6, after the mass byteification in 687b865b95ad.

interfaces: introduce and use a protocol class for the `base85` module See f2832de2a46c for details when this was done for the `bdiff` module. It looks like PEP-688 removed the special casing of `bytes` being a standin for any type of `ByteString`, and defines a `typing.Buffer` class (with a backport in `typing_extensions` for Python prior to 3.12). There's been a lot of churn in this area with pytype, but recent versions of pytype and PyCharm recognize this, and e.g. have `mercurial.node.hex()` defined as: from typing_extensions import Buffer def hex(data: Buffer, sep: str | bytes = ..., bytes_per_sep: int = ...) -> bytes This covers `bytes`, `bytearray`, and `memoryview` by default. Both of the C functions here use `y#` to parse the arguments, which means the arg is a byte-like object[2], so the args would appear to be better typed as `Buffer`. However, pytype has a bug that prevents using this from `typing_extensions`[3], and mypy complained `Unsupported left operand type for + ("memoryview")` in the pure module on line 37 (meaning it's only a subset of `Buffer`). So hold off on changing any of that for now. [1] https://peps.python.org/pep-0688/#no-special-meaning-for-bytes [2] https://docs.python.org/3/glossary.html#term-bytes-like-object [3] https://github.com/google/pytype/issues/1772

base85: avoid a spurious use-before-initialized warning in `pure` module The error wasn't possible because the only way for `acc` to not be initialized was if `len(text) == 0`. But then `0 % 5 == 0`, so no attempt at padding was done. It's a simple enough fix to not have PyCharm flag this though. The value needs to be reset on each loop iteration, so it's a line copy, not a line move.

typing: add type annotations to `mercurial/mdiff.py` We'll leave converting `diffopts` to `attrs` as another project.

mdiff: convert a few block definitions from lists to tuples These were flagged by adding type hints. Some places were using a tuple of 4 ints to define a block, and others were using a list of 4. A tuple is better for typing, because we can define the length and the type of each entry. One of the places had to redefine the tuple, since writing to a tuple at an index isn't supported. This change spills out into the tests, and archeology says it was added to the repo in this state. There was no reason given for the divergence, and I suspect it wasn't intentional. It looks like `splitblock()` is completely unused in the codebase.

interfaces: add the optional `bdiff.xdiffblocks()` method PyCharm flagged where this was called on the protocol class in `mdiff.py` in the previous commit, but pytype completely missed it. PyCharm is correct here, but I'm committing this separately to highlight this potential problem- some of the implementations don't implement _all_ of the methods the others do, and there's not a great way to indicate on a protocol class that a method or attribute is optional- that's kinda the opposite of what static typing is about. Making the method an `Optional[Callable]` attribute works here, and keeps both PyCharm and pytype happy, and the generated `mdiff.pyi` and `modules.pyi` look reasonable. We might be getting a little lucky, because the method isn't invoked directly- it is returned from another method that selects which block function to use. Except since it is declared on the protocol class, every module needs this attribute (in theory, but in practice this doesn't seem to be checked), so the check for it on the module has to change from `hasattr()` to `getattr(..., None)`. We defer defining the optional attrs to the type checking phase as an extra precaution- that way it isn't an attr with a `None` value at runtime if someone is still using `hasattr()`. As to why pytype missed this, I have no clue. The generated `mdiff.pyi` even has the global variable typed as `bdiff: intmod.BDiff`, so uses of it really should comply with what is on the class, protocol class or not.

interfaces: introduce and use a protocol class for the `bdiff` module This is allowed by PEP 544[1], and we basically follow the example there. The class here is copied from `mercurial.pure.bdiff`, and the implementation removed. There are several modules that have a few different implementations, and the implementation chosen is controlled by `HGMODULEPOLICY`. The module is loaded via `mercurial/policy.py`, and has been inferred by pytype as `Any` up to this point. Therefore it and PyCharm were blind to all functions on the module, and their signatures. Also, having multiple instances of the same module allows their signatures to get out of sync. Introducing a protocol class allows the loaded module that is stored in a variable to be given type info, which cascades through the various places it is used. This change alters 11 *.pyi files, for example. In theory, this would also allow us to ensure the various implementations of the same module are kept in alignment- simply import the module in a test module, attempt to pass it to a function that uses the corresponding protocol as an argument, and run pytype on it. In practice, this doesn't work (yet). PyCharm (erroneously) flags imported modules being passed where a protocol class is used[2]. Pytype has problems the other way- it fails to detect when a module that doesn't adhere to the protocol is passed to a protocol argument. The good news is that mypy properly detects this case. The bad news is that mypy spews a bunch of other errors when importing even simple modules, like the various `bdiff` modules. Therefore I'm punting on the tests for now because the type info around a loaded module in PyCharm is a clear win by itself. [1] https://peps.python.org/pep-0544/#modules-as-implementations-of-protocols [2] https://youtrack.jetbrains.com/issue/PY-58679/Support-modules-implementing-protocols

mdiff: tweak calls into `bdiff.fixws` to match its type hints It turns out that protocol classes can be used for modules too, which is great because all of the dynamically loaded modules (and their attributes) are currently inferred as `Any`. See the next commit for details. A protocol class for the `bdiff` module detected this (trivial) mismatch, so correct it first. The various implementations of this method are typed as taking a `bool`. The `cext` implementation parses its arguments with `PyArg_ParseTuple(args, "Sb:fixws", &s, &allws)`, which wants an `int`. But experimenting in `hg debugshell` under py38, passing `True` or `False` to `cext.fixws()` also works. We can change the implementation to use "p" (which was introduced in py33) instead of "b", but that's beyond the scope of this.

util: minor copy editing of the documentation for `mmapread()`

util: make `mmapread()` work on Windows again 522b4d729e89 started referencing `mmap.MAP_PRIVATE`, but that's not available on Windows, so `hg version` worked, but `make local` did not. That commit also started calling the constructor with the fine-grained `flags` and `prot` args, but those aren't available on Windows either[1] (though the backing C code doesn't seem conditionalized to disallow usage of them). I assume the change away from from the `access` arg was to provide the same options, plus `MAP_POPULATE`. Looking at the source code[2], they're not quite the same- `ACCESS_READ` is equivalent to `flags = MAP_SHARED` and `prot = PROT_READ`. `MAP_PRIVATE` is only used with `ACCESS_COPY`, which allows read and write. Therefore, we can't quite get the same baseline flags on Windows, but this was the status quo ante and `MAP_POPULATE` is a Linux thing, so presumably it works. I realize that typically the OS differences are abstracted into the platform modules, but I'm leaving it here so that it is obvious what the differences are between the platforms. [1] https://docs.python.org/3/library/mmap.html#mmap.mmap [2] https://github.com/python/cpython/blob/5e0abb47886bc665eefdcc19fde985f803e49d4c/Modules/mmapmodule.c#L1539

typing: add type annotations to the dirstate classes The basic procedure here was to use `merge-pyi` to merge the `git/dirstate.pyi` file in (after renaming the interface class to match), cleaning up the import statement mess, and then repeating the procedure for `mercurial/dirstate.pyi`. Surprisingly, git's dirstate had more hints inferred in its *.pyi file. After that, it was a manual examination of each method in the interface, and how they were implemented in the core and git classes to verify what was inferred by pytype, and fill in the missing gaps. Since this involved jumping around between three different files, I applied the same type info to all three at the same time. Complex types I rolled up into type aliases in the interface module, and used that as needed. That way if it changes, there's one place to edit. There are some hints still missing, and some documentation that doesn't match the signatures. They should all be marked with TODOs. There are also a bunch of methods on the core class that aren't on the Protocol class that seem like maybe they should be (like `set_tracked()`). There are even more methods missing from the git class. But that's a project for another time.

interfaces: change a couple of dirstate fields to `@property` As I was adding type hints here and to the concrete classes, PyCharm flagged the property in the core class as not being compatible with the base class's version.