rust-status: save new dircache even if just invalidated There was a functional race between invalidating the cache (not acted upon until the end of the status algorithm) and populating the new cache (which relies upon an up-to-date version of the cache). We simply inform the cache populating function that we've just invalidated the cache for this particular directory since the information is present in the same scope.

rust-status: fix typos and add docstrings to dircache related fields

rhg: show a bug where repeated [hg status] is needed to cache everything

upgrade: no longer keep all revlogs in memory at any point Keeping all object open is unsustainable, so we will open them on demand. This mean opening them multiple times, but this is a lesser evil. Each revlog consume a small amount of memory (index content, associated nodemap, etc). While there are few "big" revlog, the sheer amount of small filelog can become a significant issue memory wise, consuming multiple GB of memory. If you combines this extra usage with the use of multiprocessing, this usage can quickly get out of control. This can effectively block the upgrade of larger repository. This changeset fixes this issue.

demandimport: convert ignored modules from bytes -> str in extensions The default list of ignored modules are str, and test for bypassing the lazy import is `module.__name__ in ignores`, so these were effectively NOT ignored. Most of these date back to the grand byteification in 687b865b95ad, with some subsequent additions that followed the existing example. I have no idea if these modules in fact need to bypass lazy importing, but at least it follows the intent of the code.

dirstate-v2: fix edge case where entries aren't sorted See previous commit for more details.

dirstate-v2: highlight a bug when Python-packed but used in `rhg` The Python packer creates unsorted entries in the edge case that a file starts with the same name as a sibling folder. This bug has no effect on the Python `hg status` since Python ignores directories. `rhg` assumes that all on-disk entries are sorted (which is a property of the format) including folder, hence the issue highlighted. This is also technically broken in Rust-augmented `hg status`, but it makes setting up the test more complex than necessary, since it requires the packing to be Python only (which it isn't if you have Rust extensions). Fix is in the next commit.

dirstate-v2: correct documented return values of `pack_dirstate`

dirstate-v2: fix typos in docstrings

dirstate-v2: update constant that wasn't kept in sync Despite the best efforts of the comment, this constant wasn't kept in sync when the flags were being rewritten. The fact that this doesn't actually break anything in the Rust implementation too much (which does use directories) relies on the fact that all nodes can have children and that dirstate traversal is not based on that flag, but for metadata in optimizations. However the bug could become more serious should we start encoding stronger guarantees using a combination of flags including this one.

lfs: avoid closing connections when the worker doesn't fork Probably not much more than an minor optimization, but could be useful in the case of `hg verify` where missing blobs are fetched one at a time.

lfs: fix blob corruption when tranferring with workers on posix The problem seems to be that the connection used to request the location of the blobs is sitting in the connection pool, and then when workers are forked, they all see and attempt to use the same connection. This garbles everything. I have no clue how this ever worked reliably (but it seems to, even on Linux, with SCM Manager 1.58). See previous discussion when worker support was added[1]. It shouldn't be a problem on Windows, since the workers are just threads in the same process, and can see which connections are marked available and which are in use. (The fact that `mercurial.keepalive.ConnectionManager.set_ready()` doesn't acquire a lock does give me some pause though.) [1] https://phab.mercurial-scm.org/D1568#31621

keepalive: add `__repr__()` to the HTTPConnection class to ease debugging By default, this just printed the class name and memory address. By displaying the address and ports on both sides of the socket, it makes it easier to figure out what's in the ConnectionManager, and correlate with WireShark traces. It looks like the two connections mentioned in the previous commit come about because the LFS POST request to access the blobs opens connection 1, and gets a 401. Then for some reason, the follow up with credentials opens a new socket, instead of using the existing one in the pool. I have no clue why. This can be seen with something like this in the blobstore: ``` for h in self.urlopener.handlers: if hasattr(h, "close_all"): print('open connections on %s in pid %d' % (type(h), os.getpid())) for host, conns in h._cm.get_all().items(): for c in conns: print('connection: %r' % c) ```

keepalive: ensure `close_all()` actually closes all cached connections While debugging why LFS blob downloads are getting corrupted with workers, I noticed that prior to spinning up the workers, the ConnectionManager has 2 connections to the server and calling `KeepAliveHandler.close_all()` left one behind. The reason is the value component of `self._cm.get_all().items()` is a list, and `self._cm.remove()` modifies said list while the caller is iterating over it. Now `get_all()` is a deep copy of both the dict and lists in all cases.

localrepo: byteify the requirements.DIRSTATE_TRACKED_HINT_Vx warning message Flagged by PyCharm.

rhg: fallback to slow path on invalid patterns in hgignore

rhg: add a test involving hgignore lookaround

relnotes: add 6.3

Added signature for changeset a3356ab610fc

Added tag 6.3rc0 for changeset a3356ab610fc

branching: merge default into stable This marks the feature freeze for the 6.3 release

shelve: re-wrap now that the line fits

shelve: avoid setting overloading tmpwctx

dirstate-v2: skip evaluation of hgignore regex on cached directories By making the computation of [has_ignored_ancestor] lazy we're eliding its computation in the common case when none of its descendants have changed on disk. On a ~400k files repo, with a cached status, we saw a ~64% reduction in CPU time, resulting in a speedup of ~10-15% (on ZFS), and a speedup of ~38% of XFS (XFS has faster stat operations for some reason).

releasenotes: use re.MULTILINE mode when checking admonitions Release note admonitions must start at the beginning of a line within the changeset description: .. admonitions:: The checkadmonitions function search for and validates admonitions. Unfortunately, since the ctx.description is multi-line, the regex search always fails unless the admonition is on the first line. This changeset adds re.MULTILINE to the re.compile to make the re opbject multi-line.

rhg: parallellize computation of [unsure_is_modified] [unsure_is_modified] is called for every file for which we can't determine its status based on its size and mtime alone. In particular, this happens if the mtime of the file changes without its contents changing. Parallellizing this improves performance significantly when we have many of these files. Here's an example run (on a repo with ~400k files after dropping FS caches) ``` before: real 0m53.901s user 0m27.806s sys 0m31.325s after: real 0m32.017s user 0m34.277s sys 1m26.250s ``` Another example run (a different FS): ``` before: real 3m28.479s user 0m31.800s sys 0m25.324s after: real 0m29.751s user 0m41.814s sys 1m15.387s ```

rhg: enable in case ui.statuscopies=True rhg already has code to support ui.statuscopies, but it's disabled, for seemingly no good reason.

rhg: share some code

rhg: support tweakdefaults

rhg: centralize PlainInfo

rhg: central treatment of PLAIN and PLAINEXCEPT

revset: handle wdir() in `sort(..., -topo)` The last apparent usage of `repo.changelog.parentrevs` in revsets is in `children()`, but since the sets being operated on never include wdir(), it's never called with `wdirrev` and the wdir() arg on the command line is effectively ignored instead of aborting there. I'm not sure how to fix that. Before (on a clone of hg): $ python3.8 hg perf::revset --config extensions.perf=contrib/perf.py 'sort(all(), -topo)' ! wall 0.123663 comb 0.130000 user 0.130000 sys 0.000000 (best of 76) After: $ python3.8 hg perf::revset --config extensions.perf=contrib/perf.py 'sort(all(), -topo)' ! wall 0.123838 comb 0.130000 user 0.130000 sys 0.000000 (best of 75)

revset: handle wdir() in `roots()` This is already handled in `heads()`, and both are needed to determine if a set is contiguous. I'm guessing the `0 <= p` check was to try to filter out the null revision, but it looks like that comes through in the corner case of a new repo with no commits. But that was already the case, as shown by the tests. Before (on a clone of hg): $ python3.8 hg perf::revset --config extensions.perf=contrib/perf.py 'roots(all())' ! wall 0.059301 comb 0.040000 user 0.040000 sys 0.000000 (best of 100) After: $ python3.8 hg perf::revset --config extensions.perf=contrib/perf.py 'roots(all())' ! wall 0.059387 comb 0.060000 user 0.060000 sys 0.000000 (best of 100)

pull_logger: add basic log file rotation based on size

contrib: add pull_logger extension This extension logs the pull parameters, i.e. the remote and common heads, when pulling from the local repository. The collected data should give an idea of the state of a pair of repositories and allow replaying past synchronisations between them. This is particularly useful for working on data exchange, bundling and caching-related optimisations.

branching: merge stable into default

mergetools: don't let meld open all changed files on startup In meld 3.16, a multi-file change with option '-a' results in an overview list being opened. In meld 3.20, a multi-file change with option '-a' results in an overview list AND every changed file being opened. Simply removing '-a' seems to work fine for both cases and also behaves the same as before for single-file changes. As per the release notes, this is due to the following change in meld 3.19.0: "Make the --auto-compare command line flag work again (Kai Willadsen)" See also https://gitlab.gnome.org/GNOME/meld/-/issues/516

tests: fix http-bad-server expected errors for python 3.10 (issue6643) The format of the error message changed with this version of Python. This also removes obsolete Python 3 checks.

status: let `--no-copies` override `ui.statuscopies`

run-tests: display the time it took to install Mercurial It will help make people aware of this critical step and to assess the time it takes in various options (like a CI run for example).

run-tests: deal with distutil deprecation PEP 632 recommend the use of `packaging.version` to replace the deprecated `distutil.version`. So lets do it.

fsmonitor: migrate Python ABCs from collections to collections.abc The Collections Abstract Base Classes in the collections module are deprecated since Python 3.3 in favor of collections.abc, and removed in Python 3.10.

templates: add filter to reverse list The filter supports only lists because for lists, it’s straightforward to implement. Reversing text doesn’t seem very useful and is hard to implement. Reversing the bytes would break multi-bytes encodings. Reversing the code points would break characters consisting of multiple code points. Reversing graphemes is non-trivial without using a library not included in the standard library.

requires: re-use vfs.tryread for simplicity Avoids calling `set` twice or having to re-raise an exception and implements the routine with a single return expression.

bisect: avoid copying ancestor list for non-merge commits During a bisection, hg needs to compute a list of all ancestors for every candidate commit. This is accomplished via a bottom-up traversal of the set of candidates, during which each revision's ancestor list is populated using the ancestor list of its parent(s). Previously, this involved copying the entire list, which could be very long in if the bisection range was large. To help improve this, we can observe that each candidate commit is visited exactly once, at which point its ancestor list is copied into its children's lists and then dropped. In the case of non-merge commits, a commit's ancestor list consists exactly of its parent's list plus itself. This means that we can trivially reuse the parent's existing list for one of its non-merge children, which avoids copying entirely if that commit is the parent's only child. This makes bisections over linear ranges of commits much faster. During some informal testing in the large publicly-available `mozilla-central` repository, this noticeably sped up bisections over large ranges of history: Setup: $ cd mozilla-central $ hg bisect --reset $ hg bisect --good 0 $ hg log -r tip -T '{rev}\n' 628417 Test: $ time hg bisect --bad tip --noupdate Before: real 3m35.927s user 3m35.553s sys 0m0.319s After: real 1m41.142s user 1m40.810s sys 0m0.285s

rhg-status: add support for narrow clones

rust: add support for hints in error messages This will be used by the narrow support code in the next commit.

rust: add Debug constraint to Matcher trait This makes sure we can easily debug which Matcher we're looking at when using trait objects, and is just generally useful. Effort to make the debugging output nicer has been kept to a minimum, please feel free to improve.

rhg: add sparse support

rhg: add debugrhgsparse command to help figure out bugs in rhg

rhg-status: extract a function for printing pattern file warnings This will be reused for the warnings produced by the sparse file parsing functions.

rust-filepatterns: allow overriding default syntax This will be used when parsing pattern files other than .hgignore like the sparse spec.

rhg: don't fallback if `strip` or `rebase` are activated Neither of these extensions do anything other than add commands, so ignoring them opens up more of the test suite to rhg.

rhg: fallback in `debugdata` if repo has `narrow` Narrow uses ellipsis nodes and debugdata does not understand them yet.

rust-status: expose DifferenceMatcher from Rust to Python

rust-matchers: implement DifferenceMatcher This can be generated by the sparse matcher.

rhg: support "!" syntax for disabling extensions This makes it so that calls in test-log.t do not fall back immediately because of the disabled extension, instead going through the CLI parsing code, which breaks because of invalid UTF-8 in a flag. I *think* clap 3.x+ supports this? I'm not sure, and we have to upgrade the minimum Rust version to use clap 3.x anyway which is out of scope for this series, so let's just kick that can down the road a little bit.

revlog: finer computation of "issnapshot" If the parent had an empty diff, we skip of it to compute a diff against the parent base. This create shorter and simpler chain. However these case could be wrongly detected as snapshot. So we improve the code doing this detection. In practice nobody care as when tried on a copy of mozilla-try and we got the same number of snapshot (1315) in both case. Performance where equivalent.

perf: make perf::bundle compatible down to 5.2 A another small change to make it compatible with a wider set of revision. I did not check compatibility in the python-2 territory yet.

perf: make perf::bundle compatible before 61ba04693d65 A small change to make it compatible with a wider set of revision.

branching: merge stable into default

compare-disco: support for `file` nodes specification This leverage the `nodefromfile` feature in core. This make it possible for callers to no longer pay the subset computation cost (and to make sure the subset is the right one, even when the base repository is different)

compare-disco: move case parsing into its own function This is open the way to the next changeset that will allow to specify a list of heads.

compare-disco: add an option to skip the case If we already know the context, we can save a lot of display space by skipping the case. This also open the way to speedup to the way we specify the subsets. (the code is hacky, but this is a quicky and dirty debug script)