revlogutils: for issue6528 fix, pre-cache nullrev as metadata-free

revlogutils: for issue6528 fix, cache results for null changes

revlogutils: fix _chunk() reference _chunk is only found in the inner revlog object and not directly exposed outside.

rev-branch-cache: reenable memory mapping of the revision data Now that we are no longer truncating it, we can mmap it again. This provide a sizeable speedup on repository with a very large amount of revision for example for a mozilla-try clone with 5 793 383 revisions, this provide a speedup of 5ms - 10ms. Since they happens within the "critical" locked path during push. These miliseconds are important. In addition, the v3 branchmap format is use the rev-branch-cache more than the v2 branchmap cache so this will be important. On smaller repository we consistently see an improvement of one or two percents, but the gain in absolute time is usually < 10 ms. #### benchmark.name = hg.command.unbundle # benchmark.variants.issue6528 = disabled # benchmark.variants.reuse-external-delta-parent = yes # benchmark.variants.revs = any-1-extra-rev # benchmark.variants.source = unbundle # benchmark.variants.verbosity = quiet ### data-env-vars.name = mozilla-try-2024-03-26-zstd-sparse-revlog ## bin-env-vars.hg.flavor = default e51161b12c7e: 3.527923 ebdcfe85b070: 3.468178 (-1.69%, -0.06) ## bin-env-vars.hg.flavor = rust e51161b12c7e: 3.580158 ebdcfe85b070: 3.480564 (-2.78%, -0.10) ### data-env-vars.name = mozilla-try-2024-03-26-ds2-pnm ## bin-env-vars.hg.flavor = rust e51161b12c7e: 3.527923 ebdcfe85b070: 3.468178 (-1.69%, -0.06)

rev-branch-cache: have debugupdatecache warm rbc too Since the "v2" format can be more performant than the "v1" format (thanks to mmap), it is useful to be able to make sure it is present

rev-branch-cache: schedule a write of the "v2" format if we read from "v1" The new file can be memorymapped, while the old one cannot. So there is value in having the v2 format around as soon a possible.

rev-branch-cache: fallback on "v1" data if no v2 is found This will help smooth the transition to the v2 format for existing large repository.

rev-branch-cache: increment the version to "v2" We want to ensure no older clients will truncate the file under us. So we need to change their name. We don't change the rest of the format (unfortunaly).

rev-branch-cache: stop truncating cache file Truncating the file prevent the safe use of mmap. So instead of overwrite the existing data. If more than 20% of the file is to be overwritten, we rewrite the whole file instead. Such whole rewrite is done by replacing the old one with a new one, so mmap of the old file would be affected. This prepare a more aggressive use of mmap in later patches.

rev-branch-cache: make sure we close the name file we open We were various opening without with or try. Adding a try would not hurt.

rev-branch-cache: add a way to force rewrite of the cache This seems useful to be able to do this, for example during strip. This align with the intended expressed in the `test-branches.t` test. This will help use being more confident about future changes in the series.

rev-branch-cache: issue more truthful "truncating" message First, don't pretend it truncate to 40 when it actually truncate to 0. Second, don't pretend to truncate to 0 when the file is already empty/missing.

rev-branch-cache: move the code in a dedicated module The branchmap module is getting huge and the rev branch cache is fully independent, lets move it elsewhere.

statichttprepo: stop shadowing the `bytes` builtin PyCharm flagged it, but I also misunderstood when looking at the code, because the name implied a byte string, not a number.

statichttprepo: fix `httprangereader.read()` for py3 It looks like there were a bunch of problems, not all of them py3 related: 1) The signature of BinaryIO.read() is -1, not None 2) The `end` variable can't be bytes and interpolate into str with "%s" 3) The `end` variable can't be an int and interpolate into str with "%s" 4) The result slicing could be out of bounds if more is requested than returned I guess if somebody would have called `read(-1)` (either directly or because a wrapper defaults to that), it wouldn't have been handled correctly. The fact that it is a valid value meaning to read everything requires some additional changes later in the method around when it slices the byte string that was read, but that seems to have already been broken.

statichttprepo: use a context manager to handle a file descriptor I'm not sure if this should be reduced to `vfs.exists()`. That would seem to be equivalent code (since the result of the read is ignored, so we can't tell if the file actually has content, which has been the state of things going back to 98b6c3dde237), but this is at least safer file descriptor handling.

profiling: pass bytes to `_()` and `error.Abort()` And of course `other_tool_name` is str too, so that needs to be converted. The type hints from PyCharm say `sys.monitoring.get_tool()` can return None, so handle that case explicitly before it trips up pytype.

exchange: improve computation of relevant markers for large repos Compute the candidate nodes with relevant markers directly from keys of the predecessors/successors/children dictionaries of obsstore. This is faster than iterating over all nodes directly. This test could be further improved for repositories with relative few markers compared to the repository size, but this is no longer hot already. With the current loop structure, the obshashrange use works as well as before as it passes lists with a single node. Adjust the interface by allowing revision lists as well as node lists. This helps cases that computes ancestors as it reduces the materialisation cost. Use this in _pushdiscoveryobsmarker and _getbundleobsmarkerpart. Improve the latter further by directly using ancestors(). Performance benchmarks show notable and welcome improvement to no-op push and pull (that would also apply to other push/pull). This apply to push and pull done without evolve. ### push/pull Benchmark parameter # bin-env-vars.hg.flavor = default # benchmark.variants.explicit-rev = none # benchmark.variants.protocol = ssh # benchmark.variants.revs = none ## benchmark.name = hg.command.pull # data-env-vars.name = mercurial-devel-2024-03-22-zstd-sparse-revlog before: 5.968537 seconds after: 5.668507 seconds (-5.03%, -0.30) # data-env-vars.name = tryton-devel-2024-03-22-zstd-sparse-revlog before: 1.446232 seconds after: 0.835553 seconds (-42.23%, -0.61) # data-env-vars.name = netbsd-src-draft-2024-09-19-zstd-sparse-revlog before: 5.777412 seconds after: 2.523454 seconds (-56.32%, -3.25) ## benchmark.name = hg.command.push # data-env-vars.name = mercurial-devel-2024-03-22-zstd-sparse-revlog before: 6.155501 seconds after: 5.885072 seconds (-4.39%, -0.27) # data-env-vars.name = tryton-devel-2024-03-22-zstd-sparse-revlog before: 1.491054 seconds after: 0.934882 seconds (-37.30%, -0.56) # data-env-vars.name = netbsd-src-draft-2024-09-19-zstd-sparse-revlog before: 5.902494 seconds after: 2.957644 seconds (-49.89%, -2.94) There is not notable different in these result using the "rust" flavor instead of the "default". The performance impact on the same operation when using evolve were also tested and no impact was noted.

typing: make the localrepo classes known to pytype 9d4ad05bc91c and 1b17309cdaab both mentioned making `bundlerepository` and `unionrepository` subclass `localrepository` during the type checking phase, but that didn't apply to pytype in practice. See bcaa5d408657 and friends for how the zope interfaces confuse pytype, and end up converting the classes they decorate into `Any`. This commit is slightly more complex though, because `localrepository` has mixin classes applied to it when it is instantiated. Specifically, `RevlogFileStorage` is added, which adds `def file(f)` (which isn't defined on `localrepository`). Therefore a list of `localrepository` superclasses is provided during type checking to account for the mixins. Without this, the `bundlerepository` class gets flagged when it attempts to call its superclass implementation of `file()`. Note that pytype doesn't understand these mixin superclasses (it marks the superclass of `localrepository` as `Any`, because they are zope interfaces it doesn't understand), but that's enough to get it to not flag `bundlerepository`. PyCharm also stops flagging it as a missing function, though it seems like it is able to handle the zope interfaces.

typing: add a handful more annotations to `mercurial/vfs.py` These came out of refactoring into a protocol class, but they can stand on their own. The `audit` callback is kinda screwy because the internal lambda and the callable for `pathutil.pathauditor` have different args and a different return type. It's conditionalized where it is called, and can be cleaned up later if desired.

typing: make `vfs.isfileorlink_checkdir()` path arg required The only caller to this is `merge._checkunknownfile()`, which supplies a value. That's good, because `util.localpath()` immediately uses the value to call a method on it on Windows. The posix implementation returns the value unaltered, but then `pathutil.finddirs_rev_noroot()` would have exploded.

typing: manually add type annotations to `mercurial/vfs.py` This isn't everything, but hopefully it's close enough to hack on a protocol class.

typing: correct pytype mistakes in `mercurial/vfs.py` With the previous changes in this series (prior to merging the *.pyi file), this wasn't too bad- the only definitively wrong things were the `data` argument to `writelines()`, and the return type on `backgroundclosing()` (both of these errors were dropped in the previous commit; for some reason pytype doesn't like `contextlib._GeneratorContextManager`, even though that's what it determined it is): File "/mnt/c/Users/Matt/hg/mercurial/vfs.py", line 411, in abstractvfs: Bad return type 'contextlib._GeneratorContextManager' for generator function abstractvfs.backgroundclosing [bad-yield-annotation] Expected Generator, Iterable or Iterator PyCharm thinks this is `Generator[backgroundfilecloser], Any, None]`, which can be reduced to `Iterator[backgroundfilecloser]`, but pytype flagged the line that calls `yield` without an argument unless it's also `Optional`. PyCharm is happy either way. For some reason, `Iterable` didn't work for pytype: File "/mnt/c/Users/Matt/hg/mercurial/vfs.py", line 390, in abstractvfs: Function contextlib.contextmanager was called with the wrong arguments [wrong-arg-types] Expected: (func: Callable[[Any], Iterator]) Actually passed: (func: Callable[[Any, Any, Any], Iterable[Optional[Any]]]) Attributes of protocol Iterator[_T_co] are not implemented on Iterable[Optional[Any]]: __next__

typing: run `merge-pyi` on `mercurial/vfs.py` The *.pyi file was generated with pytype 2023.11.21. There were a few things here that were wrong (e.g. `writelines()` takes an `Iterable[bytes]`, not `bytes`, or inexplicable errors like importing several of the vfs classes from this very module), and those changes have been dropped manually here.

typing: add type annotations to `mercurial.util.makelock()` This bubbles up into the `vfs` classes, so get this out of the way.

util: avoid a leaked file descriptor in `util.makelock()` exceptional case

typing: add type annotations to the `mercurial.util.filestat` class It's referenced in the `vfs` classes, so get this out of the way to help there. The `TypeVar` definition and its usage was copied from the existing `util.pyi` file.

vfs: do minor copyediting on comments and doc strings These were flagged by PyCharm, so clear them from the gutter.

vfs: simplify the `abstractvfs.rename()` implementation PyCharm was yapping about `util.rename()` not returning anything, because it is typed to return `None`, but the value was captured and returned after calling `_avoidambig()`. Instead, drop all of that, unconditionally rename, and then call `_avoidambig()` if appropriate. While we're here, convert the ersatz ternary operator into a modern one to help pytype. When a variable is initialized the old way, pytype tends to assign the type of the LHS of the `and`. In this case, that's a bool, and it will get confused that bool doesn't have a `stat` attribute once this method gets more type annotations. (Currently it thinks the `checkambig` arg is `Any`, so it doesn't care.)

vfs: use @abstractmethod instead of homebrewing abstract methods The latter confuses PyCharm after adding more type annotations when, for example, `abstractvfs.rename()` calls `_auditpath()`- the latter unconditionally raised an error, so PyCharm thought the code that came after is unreachable. It also tricked pytype into marking the return type as `Never`, which isn't available until Python 3.11 (outside of `typing_extensions`). This also avoid PyCharm warnings that the call to the superclass constructor was missed (it couldn't be called because it raised an error to prevent instantiation). The statichttprepo module needed to be given an override for one of the abstract methods, so that it can be instantiated. In `abstractvfs`, this method is only called by `rename()`, so I think we can leave this empty. We raise an error in case somebody accidentally calls it in the future- it would have raised this same error prior to this change. I couldn't wrangle `import-checker.py` into accepting importing `ABC` and `abstractmethod`- for each subsequent import, it reports something like: stdlib import "contextlib" follows local import: abc I suspect the problem is that near the `if fullname != '__future__'` check, if the module doesn't fall into the error case, `seenlocal` gets set to the module name. That causes it to be treated like a local module on the next iteration, even though it is in `stdlib_modules`.

vfs: modernize the detection of the main thread There weren't a lot of good choices when py27 was supported, but starting with py34, `threading.main_thread()` is available. This gets us away from an undocumented, internal symbol, and drops a pytype suppression statement. It is also apparently no longer reliable after a process fork.[1][2] [1] https://stackoverflow.com/a/23207116 [2] https://github.com/python/cpython/blob/v3.6.3/Lib/threading.py#L1334

store: fix a signature mismatch for a vfs subclass This was flagged by PyCharm. I'm not sure why pytype doesn't catch this- it's not excluded from the modules that are currently checked.

lfs: fix various signature mismatches for vfs subclasses These were flagged by PyCharm. I'm not sure why pytype doesn't catch these- only `hgext/lfs/__init__.py` in the lfs extension is excluded from being checked. I'm not sure if the `*insidef` arg to `join()` was meant as an internal convencience, because I see another class that gets flagged for the same signature problem (to be fixed next). But I don't feel bold enough to make this an internal function, and provide a simplified public `join()` on the `vfs` classes. That can still be done later, if desired. For now, process the additional args and pass them along, even though there don't appear to be any current callers that provide extra args to these classes. We need all of the subclasses to agree on the signature, or they won't be considered to implement the `Vfs` protocol being developed. While we're copy/pasting from the base class, bring the type annotations along for the ride.

util: add a comment to suppress a PyCharm warning about a PEP 8 violation Slowly trying to get rid of silly warnings, so that real problems aren't hidden.

keepalive: fix a signature mismatch for a http.client.HTTPResponse subclass Also flagged by PyCharm. This is checked by pytype too, so I'm not sure why it misses this. I verified in py36 that this argument is documented for the function, so maybe this is py2 legacy.

cbor: drop a duplicate dictionary initialization entry Flagged by PyCharm.

profiling: document the py-spy value for `profiling.type` The feature was not visible otherwise.

tests: enable pytype checking on `mercurial/unionrepo.py`

unionrepo: fix mismatches with revlog classes This is a subset of cfd30df0f8e4, applied to `unionrepository`. There are none of the `write()` method overrides here, like `bundlerepository`. With these changes, pytype flags the `unionrevlog` constructor: File "/mnt/c/Users/Matt/hg/mercurial/unionrepo.py", line 55, in __init__: No attribute '_revlog' on mercurial.changelog.changelog [attribute-error] Called from (traceback): line 207, in __init__ File "/mnt/c/Users/Matt/hg/mercurial/unionrepo.py", line 55, in __init__: No attribute '_revlog' on mercurial.revlog.revlog [attribute-error] Called from (traceback): line 232, in __init__ But it turns out that both `changelog.changelog` and `revlog.revlog` do have a `target` attribute, so they wouldn't trip over this. It seems weird that the second caller to be flagged is passing the private `_revlog`, but maybe that's how it needs to be.

typing: make `unionrepository` subclass `localrepository` while type checking This is the same change as 9d4ad05bc91c made for `bundlerepository`, for the same reasons. Also, add a comment here to suppress the PyCharm warning that the superclass constructor is not called, that is new now that there's a simulated superclass. That lack of a call is by design- `makeunionrepository()` does magic that PyCharm isn't aware of. But PyCharm has been better at catching problems than pytype in a lot of cases, so I'd like to reduce the bogus things it flags, to make the real issues stand out.

tests: enable pytype checking on `mercurial/bundlerepo.py`

revlog: make `clearcaches()` signature consistent with ManifestRevlog I'm not sure if this a newly added bug, because of using a different version of pytype, or if the recent work around avoiding the zope interface types in the type checking phase (see 5eb98ea78fd7 and friends)... but pytype 2023.11.21 started flagging this series since it was last pushed ~6 weeks ago: File "/mnt/c/Users/Matt/hg/mercurial/bundlerepo.py", line 204, in <module>: Overriding method signature mismatch [signature-mismatch] Base signature: 'def mercurial.manifest.ManifestRevlog.clearcaches(self, clear_persisted_data: Any = ...) -> None'. Subclass signature: 'def mercurial.revlog.revlog.clearcaches(self) -> None'. Not enough positional parameters in overriding method. Maybe the multiple inheritance in `bundlerepo.bundlemanifest` is bad, but it seems like a `ManifestRevlog` is-a `revlog`, even though the class hierarchy isn't coded that way. Additionally, it looks like `revlog.clearcaches()` is dealing with some persistent data, so maybe this is useful to have there anyway. Also sprinkle some trivial type hints on the method, because there are other `clearcaches()` definitions in the codebase with these hints, and I don't feel like waiting for another pytype run to see if it cares that specifically about the signature matching.

bundlerepo: fix mismatches with repository and revlog classes Both pytype and PyCharm complained that `write()` and `_write()` in the bundlephasecache class aren't proper overrides- indeed they seem to be missing an argument that the base class has. PyCharm and pytype also complained that the `revlog.revlog` class doesn't have a `_chunk()` method. That looks like it was moved from revlog to `_InnerRevlog` back in e8ad6d8de8b8, and wasn't caught because this module wasn't type checked. However, I couldn't figure out a syntax with `revlog.revlog._inner._chunk(self, rev)`, as it complained about passing too many args. `bundlerevlog._rawtext()` uses this `super(...)` style to call the super class, so hopefully that works, even with the wonky dynamic subclassing. The revlog class needed the `_InnerRevlog` field typed because it isn't set in the constructor. Finally, the vfs type hints look broken. This initially failed with: File "/mnt/c/Users/Matt/hg/mercurial/bundlerepo.py", line 65, in __init__: Function readonlyvfs.__init__ was called with the wrong arguments [wrong-arg-types] Expected: (self, vfs: mercurial.vfs.vfs) Actually passed: (self, vfs: Callable) Called from (traceback): line 232, in dirlog line 214, in __init__ I don't see a raw Callable, but I tried changing some of the vfs args to be typed as `vfsmod.abstractvfs`, but that class doesn't have `options`, so it failed elsewhere. `readonlyvfs` isn't a subclass of `vfs` (it's a subclass of `abstractvfs`), so I'm not sure how to handle that. It would be a shame to have to make a union of vfs subclasses (but not all of them have `options` either).

typing: make `bundlerepository` subclass `localrepository` while type checking Currently, `mercurial/bundlerepo.py` is excluded from pytype, mostly because it complains that various `ui` and `vfs` fields in `localrepository` are missing. (`bundlerepository` dynamically subclasses `localrepository` when it is instantiated, so it works at runtime.) This makes that class hierarchy known to pytype. Having a protocol for `Repository` is probably the right thing to do, but that will be a lot of work and this still reflects the class at runtime. Subclassing also has the benefit of making sure any method overrides have a matching signature, so maybe this is a situation where we do both of these things. (I'm not sure how clear the diagnostics are if a class *almost* implements a protocol, but is missing a method argument or similar.) The subclassing is not done outside of type checking runs to avoid any side effects on already complex code.

rust: bump rust-cpython version to 0.7.2 This version supports Python 3.12 while 0.7.1 did not.

rust: add Vfs trait This will allow for the use of multiple vfs like in the Python implementation, as well as hiding the details of the upcoming Python vfs wrapper to hg-core.

rust: use new revlog configs in all revlog opening code This centralizes the more complex logic needed for the upcoming code and creates stronger APIs with fewer booleans. We also reuse `RevlogType` where needed.

rust-revlog: don't try to open the data file if the index is empty This will cover the case where the data file is not present.

rust-revlog: add revlog-specific config objects These will be used by the upcoming Rust `InnerRevlog` to better centralize config information that is relevant to revlogs.

typing: add `from __future__ import annotations` to remaining source files Most of these look newer than when the original imports referenced in the previous commit were dropped, so these weren't covered by the backout. These were found with: hg files mercurial hgext hgext3rd -I '**.py' -X '**/thirdparty' \ | xargs grep -L 'from __future__ import annotations' All of the `__init__.py` files that finds are empty, so those were ignored and the rest manually edited.

typing: add `from __future__ import annotations` to most files Now that py36 is no longer supported, we can postpone annotation evaluation. This means that the quoting is usually optional (for things imported under the guard of `if typing.TYPE_CHECKING:` to avoid circular imports), and there's less overhead on startup[1]. There may be some missing here. I backed out 6000f5b25c9b (which removed the `from __future__ import ...` that was supporting py2), reverted the changes in `contrib/`, `doc/`, and `tests/`, and then ran: $ hg status -n --change . | \ xargs sed -i -e 's/from __future__ import .*$/from __future__ import annotations/' There were some minor tweaks needed when reviewing (mostly making the spacing around the import consistent, and `mercurial/testing/__init__.py` had a multiline import that wasn't fully rewritten. [1] https://docs.python.org/3/whatsnew/3.7.html#pep-563-postponed-evaluation-of-annotations

format: add many "missing" comma Black was not adding them until the next changeset introduced a bunch of `from __future__ import annotations` to most file. Since it make the next changeset hard to read we introduce them in advance.

typing: simplify archive.gz writing and drop a few pytype suppressions I was waiting until 3.8 to use `Literal` to fix this, but there's also the ":" and "|" characters that are passed along here, meant only for the non-gz archive types. But manipulating what the local caller passes is silly- we know we're writing, so just open it for writing. As an added bonus, PyCharm stops flagging the call too (since it doesn't know about pytype suppression comments).

typing: explicitly set the return type of `_InnerRevLog.raw_text()` Somewhere between cd72a88c5599 and 2fd44b3dcc33, pytype changed the return type from `Tuple[_T1, Any, bool]` to `Any`. Both are wrong. `mdiff.patches()` is an alias for `mpatch.patches()`, which is selected via module policy (and breaks the ability to infer the types). However, `cext`, `cffi`, and `pure` implementations all agree it returns bytes.

typing: add explicit hints for recent pytype regressions Somewhere between 454feddab720 and cd72a88c5599, pytype changed how it inferred the return type in `extdiff.py` from Tuple[Any, List[Tuple[bytes, Any, os.stat_result]]] to Tuple[Any, List[nothing]] It also changed the return type in `archival.py` from `Any` to `NoReturn`. Fix those up, and also the obvious parameter types while we're here.

revlog: use the method to check if the revlog is being written to This was probably fine, but it could become not fine at some point.

revlog: add an early return for getting sidedata No point in trying to fetch sidedata if there isn't a sidedata file.

revlog: simplify rawtext return value We're always returning a tuple even though only the raw text is being used, and we're rebuilding another tuple again higher. As a bonus, this will remove one tuple creation and deletion per `raw_text` call, hence fewer gc calls, etc.

revlog: cleanup some outdated docstrings

rust-inner-revlog: always inline `get_entry` This is a very hot function.

rust-inner-revlog: derive Debug for IndexHeaderFlags

rust-inner-revlog: drop some outdated comment

rust-config: add more ways of reading the config These will be needed for future patches of this series to interpret more complex/different config values.

util: make buffer readonly There is no use of writable buffers anywhere in the code, and this helps us make sure we don't get into unsound territory when sharing memory with Rust. This `toreadonly` method was not available in Python 3.6, but we dropped the support for it earlier that week, so no need for any compatibility code.

setup: avoid the deprecated `distutils.spawn.find_executable` I noticed this was flagged with `DeprecationWarning` in py3.12 with `setuptools` 74.1.2, and it suggested `shutil.which()` instead. The signatures aren't the same, but the additional `mode` argument in the middle of the latter defaults to `os.F_OK | os.X_OK`, which maintains the same semantics.

setup: drop the hack to disable linker warning 4197 on Windows I don't see this when building on Windows with py3.8 or py3.12, so either the code was fixed, or (more likely) the compiler stopped warning about it some time after VS 2008. If we do have to put this back, it would probably be better to put a `#pragma` in a header file somewhere, and avoid `setuptools` technical debt.

ci: also offer to test 3.12 with rust The rust-cpython binding got 3.12 support very recently, it is worse keeping on a tighter watch.

ci: add the option to test more Python versions It seems like a good idea to be able to test the lowest version we support. And there have been enougth issue with 3.12 that we need to be able to make sur we do not break it. We should probably get a matrix setup for more version and flavor, but that is a simple and efficient start.

censor: document the censor.policy option (issue6909) Censor is not marked as experimental and should be documented I am not doing this on stable because the help markup change it is using seems more suitable for default.

help: add :config-doc:`section.key` shorthand to insert documentation The config items defined in the configitems.toml file can already hold their documentation. Having some way to automatically insert it was a long standing low hanging fruit. So I did a first implementation on that. It fairly simple, but it open the door to more. It will be used in the next changeset.

bzr: attempt to stabilize the test The test has flakyness where the order of a few commit swap. This is an attempt at avoiding that.

branching: merge with stable

profiling: use "stat" profiler to profile individual request The ls profiler no longer works for that. As the lsprof profiler is not default and not great is general, lets side step the issue for now.

profiling: improve 3.12 error message for calling lsprof twice Python 3.12 prevent lsprof to be enabled if it is already enabled. This break the use of lsprof in `hg serve` as both the initial `serve` command and the request serving want to profile. The "stat" profiler (the default) does not have this problem, so we focus on improving the error message for now.

test: display server error log in test-profile.t This will help us to catch error with Python 3.12

archive: defer opening the output until a file is matched Before, if no file is matched, an error is thrown, but the archive is created anyway. When using hgweb, an error 500 is returned as the response body already exists when the error is seen. Afterwards, the archive is created before the first match is emitted. If no match is found, no archive is created. This is more consistent behavior as an empty archive is not a representable in all output formats, e.g. tar archives.

run-tests: add color to the progress output More color is useful to me.

python-compat: drop support for Python3.6 and 3.7 As discussed on the mailing list¹, these are old version that seems safe to drop. Python 3.8 comes with various improvement especially regarding typing capabilities. [1] https://lists.mercurial-scm.org/pipermail/mercurial-devel/2024-July/297998.html

ci: drop path manipulation that we do not need anymore The CI image has a squarer setup now.

brancing: merge stable into default

setup: handle removal of old MSVC compiler from setuptools 65.0 (issue6910) It was removed a few years ago[1]. When trying to reproduce locally using a clean py3.12 as called out in the bug report, `setuptools` wasn't installed at all, and needed a `pip install` to fix a `ModuleNotFoundError` when building locally. Maybe that needs to be in the requirements clause now. It looks like this "private" module was added in setuptools 48.0.[2] I can't find a changelog of what version was included in which version of python, and the changelog for pip has a huge gap between when it called out 67.6.1 in `pip` 23.1 (2023-04-15), and 41.4.0 in `pip` 19.3 (2019-10-14).[3] So, we'll just add to the existing code instead of replacing it, for safety. [1] https://github.com/pypa/setuptools/commit/cc017c77948737d131f683e0c25cd37bc639b8fc [2] https://github.com/pypa/setuptools/commit/d034a5ec7f707499139f90eb846b9e720923124c [3] https://pip.pypa.io/en/stable/news/

utils: accept bytearray arguments for escapestr

http: simplify

http: use urllib's cookie handler Split the logic for loading the cookies based on the configuration in a helper function and otherwise use the library implementation directly.

http: reuse Python's implementation of read/readline/readinto Since Python 3 already provides a working implementation of readline, there is no need for our own buffering implementation. Reduce the code to transfer accounting only.

debugwireproto: redo logging to also work for https

urllib2: redo response.readlines addition via class patching

typing: lock in new pytype gains from making revlog related classes typeable These were pretty clean changes in the pyi files from earlier in this series, so add them to the code to make it more understandable. There's one more trivial hint that can be added to the return of `mercurial.revlogutils.rewrite._filelog_from_filename()`, however it needs to be imported from '..' under the conditional of `typing.TYPE_CHECKING`, and that seems to confuse the import checker- possibly because there's already an import block from that level. (I would have expected a message about multiple import statements in this case, but got one about higher level imports should come first, no matter where I put the import statement.)

typing: add types to `revlog.revlogproblem` These attrs showed as `Any` after the previous commit made the class visible to pytype.

typing: make the revlog classes known to pytype These are the same changes as c1d7ac70980b and 45270e286bdc made to dirstate, for the same reasons.

typing: make the manifest classes known to pytype These are the same changes as c1d7ac70980b and 45270e286bdc made to dirstate, for the same reasons. The migration away from decorating the classes with `@interfaceutil.implementer` was started back in 3e9a660b074a, but missed one.

typing: make the filelog class known to pytype These are the same changes as c1d7ac70980b and 45270e286bdc made to dirstate, for the same reasons.

remotefilelog: adapt the `debugindex` command to past API changes Pytype was missing these problems because it's currently inferring the classes for `filelog` and `revlog` to be `Any`. When that's fixed, these were flagged, so fix these first. The `filelog` class used to subclass `revlog`, but that was changed back in 1541e1a8e87d (with most or all of the "lost" attributes being forwarded to the embedded `revlog` attribute at that time). These forwarded references were dropped over time, and this command has been broken at least as far back as 68282a7b29a7 when the `version` field was dropped. Most of the fixes were as simple as calling the accessor for the embedded `revlog` member, but the general delta feature detection was a bit more involved- I copied the detection for it from `mercurial.revlogutils.debug.debug_revlog()`.

typing: add type hints to the `opener` attributes and arguments of revlog When making revlog and filelog classes visible to pytype, it got confused quite a bit in `mercurial/revlogutils/rewrite.py`, thinking it had a plain `Callable`, and flagging additional methods on it like `join()` and `rename()`. I couldn't figure out how it reduced to that (and PyCharm flagged `opener` references as `Any`), but this makes it happy. So make this change before making the classes visible. The vfs class hierarchy is a bit wonky (e.g. `filteredvfs` is not a `vfs`), so this may need to be revisited with a Protocol class that covers all of the `vfs` classes. But for now, everything works.

remotefilelog: honor the `--format` arg of the `debugindex` command Flagged by PyCharm while investigating pytype spew. The other `**opts` above are already accessed as str. I've never used remotefilelog, and don't have a repo to test this on, so I'm trusting the nearby code.

merge: sort filemap only if requested by the caller The name `sorted` refers to a built-in function, which is always true, so the else branch of this if statement was dead code. Because, with this fix, the function can iterate over the dict items while yielding values, the dict should not change size while the generator is running. Because of that, it is required to re-introduce code that makes a caller copy the filemap before modification, which was removed in 3c783ff08d40cbaf36eb27ffe1d296718c0f1d77 (that changeset also introduced the filemap() method including the bug that’s being fixed by this changeset).

shelve: consistently convert exception to bytes via `stringutil.forcebytestr` The other two places in this module use this, and past experience shows that this method does a nicer job. I'm not sure why we're converting to bytes here- `KeyError` is built-in and will have str attrs, and `RepoLookupError` is a subclass of the built-in `Exception` class (not `errors.Error`, which is allegedly the baseclass for all Mercurial exceptions).

typing: add type hints to `mercurial.shelve` Pytype wasn't flagging anything here yet, but PyCharm was really unhappy about the usage of `state` objects being passed to various methods that accessed attrs on it, without any obvious attrs on the class because there's no contructor. Filling that out made PyCharm happy, and a few other things needed to be filled in to make that easier, so I made a pass over the whole file and filled in the trivial hints. The other repo, ui, context, matcher, and pats items can be filled in after the context and match modules are typed.

typing: lock in correct changes from pytype 2023.04.11 -> 2023.06.16 There were a handful of other changes to the pyi files generated when updating pytype locally (and jumping from python 3.8.0 to python 3.10.11), but they were not as clear (e.g. the embedded type in a list changing from `nothing` to `Any` or similar). These looked obviously correct, and agreed with PyCharm's thoughts on the signatures. Oddly, even though pytype starting inferring `obsutil._getfilteredreason()` as returning bytes, it (correctly) complained about the None path when it was typed that way. Instead, raise a ProgrammingError if an unhandled fate is calculated. (Currently, all possibilities are handled, so this isn't reachable unless another fate is added in the future.)

monotone: replace %s interpolation with appropriate numeric specifiers The length is an int, and the version is a float. Neither work with bytes on py3. This was noticed when looking at nearby code after updating pytype changed some signatures.

shelve: raise an error when loading a corrupt state file in an impossible case The old return statement was flagged by pytype 2023.06.16 running under python 3.10.11. No idea why it isn't caught in CI running the same pytype with py3.7. This function is only called by `unshelvecmd()` (which first checks that either `--abort` or `--continue` is specified), and `hgabortunshelve()` and `hgcontinueunshelve()`, which locally apply `--abort` or `--continue` respectively. Therefore, there is no other way to call this, and this error should never be seen, but pytype can't figure that out on its own. Given that the abort case clears the state, it seems reasonable to defensively code this and not make that a blanket `else` case, on the off chance a 3rd way of calling this appears in the future.

contrib: print the version of pytype used to do the type checking This will help with CI. I don't see a way to print the version of python that's running it. When I tried `head -n 1 $(which pytype)`, the CI run printed: #!/usr/bin/env bash Locally, that gives the path to the python interpreter in the venv, so IDK what's different.

typing: create an @overload of `phasecache` ctor to handle the copy case In `phasecache.copy()`, it calls `self.__class__(None, None, _load=False)`, but the constuctor is typed to take a non-None repository. For the `_load=False` case, all args are ignored (and the copy function itself populates the attrs on the new object), so this isn't an error. For the default `_load=True` case, it needs a non-None repository. This is the simplest way to handle that duality. The reason this wasn't being detected is because pytype is confused by the interface decorators on the `localrepository` class, and is inferring the whole class as `Any`. (See 3e9a660b074a or c1d7ac70980b) Therefore, the type hint of `localrepo.localrepository` here was also effectively `Any`, which disabled the type checking entirely. This is the first foray into using `typing_extensions` to unlock future typing features. I think this is safe and reasonable because 1) it is only imported in the type checking phase (so no need to vendor our own copy), and 2) pytype has its own copy of `typing_extensions` bundled with it, so no need to alter the test environment. When run with a version of python that supports the symbol(s) natively, `typing_extensions` simply re-exports from `typing`, so there shouldn't be any future headaches with this.

typing: declare the `_phasesets` member of `phasecache` to be `Optional` Something in this area got flagged while making the repository class visible to pytype (instead of being typed as `Any`). A None assignment to something not optional is wrong, and when I tried setting it to `{}` to keep it non-Optional, some tests failed. There are checks for the attr being None elsewhere, so this seems to have just been an oversight.

typing: hide the interface version of `dirstate` during type checking As noted in the previous commit, the `dirstate` type is still inferred as `Any` by pytype, including where it is used as a base class for the largefiles dirstate. That effectively disables most type checking. The problems fixed two commits ago were flagged by this change. I'm not at all clear what the benefit of the original type is, but that was what was used at runtime, so I don't want to change the largefiles base class to the raw class. Having both a lowercase and camelcase name for the same thing isn't great, but given that this trivially finds problems without worrying about which symbol clients may be using, and the non-raw type is useless to pytype anyway, I'm not going to worry about it.

dirstate: remove the interface decorator to help pytype This is the same change that was made for some of the manifest classes in 3e9a660b074a. Note that `dirstate` is still inferred as `Any`, but at least we have `DirState` with all of the expected attributes.

largefiles: sync up `largefilesdirstate` methods with `dirstate` base class As it currently stands, pytype infers the `dirstate` class (and anything else decorated with `@interfaceutil.implementer`) as `Any`. When that is worked around, it suddenly noticed that most of these methods don't exist in the `dirstate` class anymore. Since they only called into the missing methods and there's no test failures, we can assume these are never called, and they can be dropped. In addition, PyCharm flagged `set_tracked()` and `_ignore()` as not overriding a superclass method with the same arguments. The missing default parameter for the former was the obvious issue. I'm guessing that the latter was named wrong because while there is `_ignore()` in the base class, it takes no arguments and returns a matcher. The `_ignorefiles()` superclass method also takes no args, and returns a list of bytes. The `_ignorefileandline()` superclass method DOES take a file, but returns a tuple. Therefore, the closest match is `_dirignore()`, which takes a file AND returns a bool. No idea why this needs to be overridden though.

sparse: reliably avoid writing to store without a lock With the code as written before this patch we can still end up writing to store in `debugsparse`. Obviously we'll write to it if by accident a store requirement is modified, but more importantly we write to it if another concurrent transaction modifies the requirements file on disk. We can't rule this out since we're not holding the store lock, so it's better to explicitly pass a permission to write instead of inferring it based on file contents.

debugsparse: stop taking the store lock debugsparse is a workspace-only opperation, or it better be workspace-only. Let's make it to stop taking the store lock.

scmutils: read the requires file before writing to avoid unnecessary rewrite This lets us get away without the repo lock in situations where we need to write requirements, but we know we're not changing the store requirements.

localrepo: remove _readrequires function in favor of scmutil.readrequires

scmutil: add `readrequires` next to `writerequires` The code is copied from localrepo.py.

typing: correct a type hint in `mercurial.manifest` Obvious typo that was flagged by PyCharm.

typing: add hints to `mercurial.util.mktempcopy()` Might as well, now that the previous commit indicated what types are required.

typing: fix the hint for the `mode` argument of `platform.copymode()` The posix module is doing a bitwise AND with this and an integer, so it can't be bytes. The only caller that provides the argument is `util.mktempcopy()`, and pytype infers the type as Any, which explains why this wasn't caught.

largefiles: fix check that ensures that --all-largefiles is only used locally Previously, the command added in the test failed with “abort: --all-largefiles is incompatible with non-local destination existing_destination”. The reason for the buggy behavior was the use of hg.islocal(), which does “return true if repo (or path pointing to repo) is local” and, for local paths, assumes that the path is actually pointing to an existing repository and returns whether the path is not a regular file (in which case it assumes that it is a bundlerepo, which are considered non-local).

exchange: trivial simplification Both sides of the condition do essentially the same thing, except one with fastpath=True. No functional changes.

import: fix erroneous comparison of str with bytes

histedit: create state and acquire locks earlier This makes chistedit (histedit with curses UI) not write any files inside repo without wlock. It also makes sense to wrap the entire process of preparing commands inside the curses UI inside locks because we don't want anything else to touch wdir or repo during this time.

py3: use str literal instead of bytes literal