manifest: add many type annotations to the manifest module This help to clarify the API a bit, this caught various bug in the process and will help to catch more in the future. This also make large refactoring significantly simpler.

manifest: help pytype to understant `writesubtrees`'s `getnode` type Since we provide a default, the return of `_lazydirs.get` is cannot be None. We help pytype to understand that.

manifest: use explicit None checking in `_loaddifflazy` This helps pytype to understand what is going here with `v2` type.

manifest: use explicit None checking in `_loadlazy` This help pytype to understand what is going on with `v` type.

manifest: clear `_lazydirs` in place in `_loadalllazy` This avoid resetting the type of the dictionary in pytype eyes. This is consistent with the way the dictionary is cleared bits by bits in `_loadalllazy` Having more accurate code will help pytype. We do it in advance to help bisecting and avoid drowning them in the future type annotation noise.

manifest: use tuple for `delta` in `fastdelta` This make the list content consistent and will help type annotation.

manifest: expose a version of the Class without interface decorator The decorator confuse Pytype. Having the "raw" python class exposed will also helps pytype when it get replaced by a native implementation. At least until we start using `typing.Protocol` in the future.

pytype: stop ignoring manifest.py pytype no longer complains about the file contents.

manifest: align some vfs option access on the fact we might not have options This make the usage consistent with the other option. Caught by pytype.

manifest: adds some type things for manifestdict.added This appeases pytype.

manifest: type and fix unhexlify Some part of that function seems to date back from Python 2. It raise question about whether this function is useful or not, but let us just fix it for now. This was caught by pytype.

docker-pytype: use version v2.1 of the CI image It use a more recent pytype as far as I understand.

context: some gratuitous documentation improvement I wrote it as I was reading the code.

profiling: add a py-spy profiling backend The recommended way to use this backend is by setting the config `profiling.output` to point to a file because py-spy output is not human-readable.

copytracing: fix a bug in an edge case in metadata.compute_all_files_changes

rhg: ignore readonly FS error when saving dirstate The error is already ignored when the .hg directory is read-only, so this is only fair. (the python hg is silent on readonly fs, too)

commit: set whole manifest entries at once (node with its associated flags) Add a new function manifest.set that sets whole manifest entries at once, so the caller doesn't have to do two separate operations: m[p] = n m.set_flags(f) becomes: m.set(p, n, f) This obviously saves an extra lookup by path, and it also lets the underlying manifest implementation to be more efficient as it doesn't have to deal with partially-specified entries. It makes the interaction conceptually simpler, as well, since we don't have to go through an intermediate state of incorrect partially-written entry. (the real motivation for this change is an alternative manifest implementation where we batch pending writes, and dealing with fully defined entries makes the batching logic muchsimpler while avoiding slowdown due to alternating writes and reads)

typing: add type hints around the matcher for subrepo archiving Mostly this is meant to try to smoke out any other potential issues around the matcher, since these args were mostly previously treated as `Any`, and therefore checking wasn't done.

subrepo: drop the default value of None for the archive matcher This was flagged by pytype after adding hints to `match.subdirmatcher` that it takes a non-optional matcher. That matcher argument is used without a guard in the subdirmatcher constructor, so that's the correct restriction. I don't think this fixes a bug in practice because the only way these are invoked is either by a parent `hgsubrepo.archive()`, `archival.archive()`, or the largefiles override of these. The `hgsubrepo.archive()` case (and the largefiles override) uses what the caller provided, so the caller will eventually be `archival.archive()` (or the largfiles override) up the call chain. The `archival.archive()` method also has None for its matcher's default arg. However, the three callers of that (`commands.archive()`, `webcommands.archive()`, and `extdiff.snapshot()`) all provide a matcher argument, so the None case can never occur unless a 3rd party extension swaps it for None. Sadly, we can't make the argument on the `archival.archive()` non-optional because there is a kwarg prior to it. Even though the largefiles override of `archival.archive()` is provided a valid matcher, we duplicate the internal creation of the matcher that the original `archival.archive()` does for consistency. By eliminating an impossible to hit case, we can simplify some of the subrepo code too, by dropping unreachable code.

branching: merge stable into default Post 6.8.1 release.

Added signature for changeset 11a9e2fc0caf

Added tag 6.8.1 for changeset 11a9e2fc0caf

relnotes: add 6.8.1

rhg: expand user and environment variables in ignore includes This was reported by a user, and was a TODO long overdue.

utils: avoid using internal _imp.is_frozen() imp has been deprecated for a long time, and were removed in Python 3.12 . As a workaround, we started using the internal _imp. That is ugly and risky. It seems less risky to get the functionality in some other way. Here, we just inspect if 'origin' of the '__main__' module is set and 'frozen'. That seems to work and do the same, and might be better than using the internal _imp directly. This way of inspecting module attributes seems to work in some test cases, but it is a risky change. This level of importlib doesn't have much documentation, a complicated implementation, and we are dealing with some odd use cases.

utils: fix resourceutil use of deprecated importlib.resources Some importlib functionality was deprecated in 3.11 . The documentation on https://docs.python.org/3.12/library/importlib.resources.html recommends using the new .files() API that was introduced in 3.9.

cext: use sys.executable instead of deprecated Py_GetProgramFullPath Fix warning with Python 3.13: mercurial/cext/parsers.c: In function 'check_python_version': mercurial/cext/parsers.c:1243:30: warning: 'Py_GetProgramFullPath' is deprecated [-Wdeprecated-declarations] 1243 | Py_GetProgramFullPath()); | ^~~~~~~~~~~~~~~~~~~~~ In file included from /usr/include/python3.13/Python.h:119, from mercurial/cext/parsers.c:11: /usr/include/python3.13/pylifecycle.h:43:43: note: declared here 43 | Py_DEPRECATED(3.13) PyAPI_FUNC(wchar_t *) Py_GetProgramFullPath(void); | ^~~~~~~~~~~~~~~~~~~~~ At this point in time, the PyConfig struct memory has been released and the PyConfig API can't be used. https://docs.python.org/3.13/c-api/init.html#c.Py_GetProgramFullPath recommands using sys.executable instead. Let's assume that will work in all versions. It would perhaps be better to use PySys_GetObject, but I prefer to stay consistent with how the same function is retrieving sys.hexversion.

subrepoutil: pass re.sub 'count' argument by name Python 3.13 started warning: DeprecationWarning: 'count' is passed as positional argument

tests: pass re.MULTILINE to re.sub as 'flags' - not in 'count' position This bug was caught by the new Python 3.13 warning: DeprecationWarning: 'count' is passed as positional argument

tests: use packaging from setuptools instead of deprecated distutils When invoking StrictVersion in 3.12 we got: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. distutils is dead in the standard library, and we have to move towards using `setuptools` as general extern dependency. Instead of also requiring the extern `packaging`, we will just use the packaging that is vendored in setuptools.

tests: drop test-demandimport.py distutils test that failed with warnings The test would fail because warnings: /usr/lib/python3.11/site-packages/_distutils_hack/__init__.py:18: UserWarning: Distutils was imported before Setuptools, but importing Setuptools also replaces the `distutils` module in `sys.modules`. This may lead to undesirable behaviors or errors. To avoid these issues, avoid using distutils directly, ensure that setuptools is installed in the traditional way (e.g. not an editable install), and/or make sure that setuptools is always imported before distutils. warnings.warn( /usr/lib/python3.11/site-packages/_distutils_hack/__init__.py:33: UserWarning: Setuptools is replacing distutils. warnings.warn("Setuptools is replacing distutils.") The test for distutils.msvc9compiler comes from 2205d00b6d2b. But since then, distutils is going away, and this test must change somehow. It is unclear exactly how setuptools depended on msvc9compiler, but setuptools also moved forward, and this exact test no longer seems relevant. It thus seems like a fair solution to remove the test while keeping the demandimport blacklist of distutils.msvc9compiler.

utils: test coverage of makedate Explore the scenario from ae04af1ce78d to avoid future regressions. This was intended to give some coverage of the change in faccec1edc2c.

mmap: populate mapping in a background thread When possible, we populate the memory mapping in a second thread. The mmap population does not only read the data from disk to memory. It also actually fill the memory mapping between process memory address and the physical memory used by the file system cache containing the mmap'ed data. Doing so buy back the slowdown from pre-population when it matters. When most data is accessed, only a few page fault will occurs, while the background thread fill the memory controller. When few data is accessed, the non-blocking mmap won't have to wait for all data to be populated. Here is a few example of improvement seen in benchmark around unbundle and push: ### data-env-vars.name = netbeans-2018-08-01-zstd-sparse-revlog # benchmark.name = hg.command.unbundle # benchmark.variants.issue6528 = disabled # benchmark.variants.reuse-external-delta-parent = yes # benchmark.variants.revs = any-100-extra-rev before: 0.758101 after: 0.732129 (-3.43%, -0.03) ## data-env-vars.name = mozilla-try-2019-02-18-zstd-sparse-revlog before: 1.519941 after: 1.503473 (-1.08%, -0.02) ### data-env-vars.name = mozilla-try-2019-02-18-zstd-sparse-revlog # benchmark.name = hg.command.push # bin-env-vars.hg.flavor = default # benchmark.variants.issue6528 = disabled # benchmark.variants.protocol = ssh # benchmark.variants.reuse-external-delta-parent = yes # benchmark.variants.revs = any-1-extra-rev before: 4.801442 after: 4.695810 (-1.46%, -0.07) # benchmark.variants.revs = any-100-extra-rev before: 4.848596 after: 4.794075 (-1.12%, -0.05) # bin-env-vars.hg.flavor = rust # benchmark.variants.revs = any-1-extra-rev before: 4.818410 after: 4.700053 (-2.46%, -0.12)

pure: stringify builtin exception messages Builtin exceptions usually want strings, and display with a wierd b'' prefix if given bytes.

httppeer: avoid another bad reference before assignment warning This wasn't a problem, because `b''` from the `AttributeError` handler is in `bundle2.bundletypes`, so the following loop and conditional always run at least once. But PyCharm can't figure that out on its own, and it took a little exploring to figure out it wasn't a problem. The usage in `bundle2.writebundle` is to look it up in the map of bundle types, so it will break in a more obvious way in the unlikely event that the empty string is removed from the map in the future.

httppeer: move a variable to avoid a bad reference before assignment warning No actual bug here, because the conditional used to assign is the same as the conditional in the `finally` block that guards the reference.

httppeer: simplify two-way stream cleanup No need to conditionalize the cleanup if the filename is assigned outside the exception handler. I suppose `fd` leaks if `os.fdopen()` fails, but that was the case before too (and may trigger another exception in the `finally` block on Windows, when the file is still open).

rustfmt: update expected Rust edition In this case it doesn't change anything, but we've been using 2021 for a while now.

hghave: update expected rustfmt version We still use nightly, but have moved to a newer nightly after the last CI image upgrade in 74f1bf147a6d and 3876d4c6c79e.

rustfmt: apply formatting expected by newer nightly version This was missed in 3876d4c6c79ec5c71e8c51b876cc157e93a5eaac somehow.

tests: stop skipping `mercurial/pure/osutil.py` during pytype runs Not sure when the original issue(s) were fixed, but it works for me now.

largefiles: avoid a potentially undefined variable in exception case The `wlock` variable is used to release the lock in the `finally` block, so it would be undefined if `repo.wlock()` itself failed. Caught by pytype 2024.04.11 with py3.10.11.

typing: add trivial type hints to `mercurial.scmutil` There's still a lot to go, but there's a lot here already, so I tried to keep it to obvious/trivial things. I didn't bother with contexts, matchers, and revisions that can be `bytes | int | None`.

typing: narrow the scope of some recent disabled import warnings These comments were added in 39e2b2d062c1, but had the effect of changing the known type to `Any`, which cascaded through a few function signatures. Just ignore the import error instead.

demandimport: don't delay threading import A recent cpython change breaks demandimport by importing threading locally in importlib.util.LazyLoader.exec_module; add it (plus warnings and _weakrefset, which are imported by threading) to demandimport's ignore list. ``` Traceback (most recent call last): File "/usr/bin/hg", line 57, in <module> from mercurial import dispatch File "<frozen importlib._bootstrap>", line 1360, in _find_and_load File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 935, in _load_unlocked File "/usr/lib/python3/dist-packages/hgdemandimport/demandimportpy3.py", line 52, in exec_module super().exec_module(module) File "<frozen importlib.util>", line 257, in exec_module File "<frozen importlib._bootstrap>", line 1360, in _find_and_load File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 935, in _load_unlocked File "/usr/lib/python3/dist-packages/hgdemandimport/demandimportpy3.py", line 52, in exec_module super().exec_module(module) File "<frozen importlib.util>", line 267, in exec_module AttributeError: partially initialized module 'threading' has no attribute 'RLock' (most likely due to a circular import) ``` Ref: https://github.com/python/cpython/issues/117983 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1076449 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1076747

typing: induce pytype to use the standard `attr` instead of the vendored copy What was previously happening with the vendored copy was that pytype would stub out all(?) classes that were decorated with `@attr.s` as `Any`. After this, we get a ton of classes defined, and numerous fields and methods now have proper types.

typing: disable some pytype errors in `mercurial.store` These seem to be legitimate errors, since one of the two callers (`encodedstore.data_entries()`) was previously typed to generate `BaseStoreEntry`. However, that and the other caller only pass `RevlogStoreEntry` objects. I can't tell if this was a WIP or what, but don't want to get side tracked on this. So flag as a TODO for later.

linelog: correct the default value of `annotateresult.lines` This was flagged by pytype once it was tricked into using the standard `attr` package instead of the vendored copy.

phabricator: correct the default value of `phabhunk.corpus` There's only one caller to this constructor (which does provide this argument), and no direct assignments, so there's no runtime bug here. However, when pytype is tricked into using the standard `attr` package instead of the vendored version, it flags this because bytes is passed to the one constructor invocation. Tricking pytype into using the standard package will generate many more type hints, noteably around `@attr.s` decorated things.

rust-changelog: accessing the index The `Index` object is currently the one providing all DAG related algorithms, starting with simple ancestors iteration up to more advanced ones (ranges, common ancestors…). From pure Rust code, there was no way to access the changelog index for a given `Repository`, probably because `rhg` does not use any such algorithm yet.

typing: add type hints to `mercurial.policy` Mostly trivial, but this seems like the logical module to use to inject the hints from `cext`, `pure`, etc, given that this file has the fallback policy. This is a first step. There doesn't appear to be a predefined type for a module in py3.7, so those are omitted for now.

cext: correct the argument handling of `b85encode()` The type stub indicated that this argument is `Optional`, which implies None is allowed. I don't see in the documentation where that's the case for `i`[1], and trying it in `hg debugshell` resulted in the method failing with a TypeError. I guess it was typed as an `int` argument because the `p` format unit wasn't added until Python 3.3[2]. In any event, 2 clients in core (`pvec` and `obsolete`) call this with no argument supplied, and `mdiff` calls it with True. So I guess we've avoided the None arg case, and when no arg is supplied, it defaults to the 0 initialization of the `pad` variable in C. Since the `p` format unit accepts both `int` and None, as well as `bool`, I'm not bothering to bump the module version- this code is more permissive than it was, in addition to being more correct. Interestingly, when I first imported the `cext` and `pure` methods in the same manner as the previous commit, it dropped the `Optional` part of the argument type when generating `util.pyi`. No idea why. [1] https://docs.python.org/3/c-api/arg.html#numbers [2] https://docs.python.org/3/c-api/arg.html#other-objects

typing: add type hints to the `charencode` module Since this module is dynamically imported from either `mercurial.pure` or `mercurial.cext`, these hints aren't detected in `mercurial.encoding`, and need to be imported directly there during the type-checking phase. This keeps the runtime selection via the policy config in place, but allows pytype to see these as functions with proper signatures instead of just `Any`. We don't attempt to import the `mercurial.cext` version yet because there's no types stubs for that module, but this will get the ball rolling. I thought this would spill over into other modules from there, but the only two *.pyi files that changed were for `encoding` and `charencode`. Applying this to other dynamically selected modules will clean some things up in other files, so this is a start. I had originally redefined the functions in the type-checking block (like some of the `os.path` aliasing in `mercurial.util`), but this is better because we won't have another duplication of the definitions that may get out of date.

typing: explicitly type some `mercurial.util` eol code to avoid @overload Unlike the previous commit, this makes a material difference in the generated stub file- the `pycompat.identity()` aliases generated an @overload like this: @overload def fromnativeeol(a: _T0) -> _T0: ... ... which might fail to detect a bad argument, like str. This drops the @overload for the 3 related methods, so there's a single definition for each. The `typelib.BinaryIO_Proxy` is used for subclassing (the same as was done in 8147abc05794), so that it is a `BinaryIO` type during type checking, but still inherits `object` at runtime. That way, we don't need to implement unused abstract methods.

typing: avoid some useless @overload definitions in `mercurial.util` Apparently pytype considered the name as well as the type of each argument, and generates @overload definitions if they don't match. At best this is clutter, and can easily be removed.

dirstate: stringify a few exception messages Built in exceptions want str, and ProgrammingError converts bytes to str internally (because it subclasses RuntimeError).

typing: add type hints to `mercurial.verify._normpath()` Since 10db46e128d4, pytype almost figured this out, going from `Any` -> `_T0`, but the intent is obvious.

typing: add type hints to `i18n._msgcache` Since 10db46e128d4, pytype stopped inferring that the key is bytes.

typing: add type hints to `mercurial.dirstatemap` Somewhere since 10db46e128d4, pytype stopped being able to infer the type of the `identity` field. Fill in some obvious other hints along the way. These hints caused pytype to flag a missing attribute: File "/mnt/c/Users/Matt/hg/mercurial/dirstatemap.py", line 714, in _v1_map: No attribute 'stat' on mercurial.windows.cachestat [attribute-error] In Union[Any, mercurial.posix.cachestat, mercurial.windows.cachestat] File "/mnt/c/Users/Matt/hg/mercurial/dirstatemap.py", line 715, in _v1_map: No attribute 'stat' on mercurial.windows.cachestat [attribute-error] In Union[Any, mercurial.posix.cachestat, mercurial.windows.cachestat] In practice, the `identity` field is NOT replaced with None if it isn't cacheable, so it's probably safer to just add the field and set it to None, since that check is already in place on line 715.

typing: add type hints to `cmdutil.findrepo()` Since 10db46e128d4, pytype almost figured this out, going from `Any` -> `_T0`, but the intent is obvious.

typing: add some type hints to fastannotate that have decayed in the last year Somewhere since 10db46e128d4, `_knownopts` decayed to `set` for unknown reasons. Also, `annotateopts.default` changed from bytes to str. While that is correct, I noticed that PyCharm was flagging the member fields as undefined in `shortstr()`, so add those to keep it happy. (There are no complaints from pytype because that module is excluded, due to the missing reference to `linelog.copyfrom()` that I'm not sure how to fix.)

heptapod-ci: use new v2.1 image This is finally catching up to ~3 years of tech debt.

heptapod-ci: move version prints closer to the start This makes debugging a lot easier if anything is to go wrong, and shows output earlier.

pytype: only try the hacky way of finding PYTHON if not provided This allows us to work in more environments, like when using pyenv. This syntax is compatible with all POSIX shells.