rust-status: fix thread count ceiling This was forcing 16 threads instead of creating a ceiling, which is wrong when either the available parallelism of the platform is lower or when the user wants to set it explicitly (like we do in `run-tests.py`)

tests: use an all too familiar executable in test-run-tests.t (issue6661) true(1) sometimes lives in /usr/bin/, and so this test fails on such systems. We also can't use which(1), because it's apparently not POSIX and Debian complains about it [1]. We also cannot really create a script and chmod +x it, because this is a symlink case, execbit case is slightly below. So let's use something that we know is executable, but not hg itself.

tests: add the missing space to test-hghave.t (issue6762) log() in run-tests.py add a space to the end of all messages that it prints, and I missed this fact in my previous patch.

tests: expect the message from 1baf0fffd82f in test-hghave.t (issue6762) Couldn't reproduce locally, but apparently this message may occasionally pop up when running this test.

test: adjust test-push-race.t timeout's to overall test timeout The generic `tests/testlib/wait-on-file` mechanism scale its timeout with the value of `HGTEST_TIMEOUT`, the `delaypush.py` in `test-push-race.t` is not doing this, and we have been seeing more and more timeout from loaded CI worker lately. Adding this timeout scaling should help with that.

python-compat: adapt to Python 3.11 BC breakage with `random.sample` As per https://docs.python.org/3/whatsnew/3.11.html#porting-to-python-3-11: "The population parameter of `random.sample()` must be a sequence, and automatic conversion of sets to lists is no longer supported. Also, if the sample size is larger than the population size, a `ValueError` is raised"

match: make the FLAG_RE pattern a raw string PyCharm was complaining about invalid escape sequences since this was added recently in 3eda36e9b3d6.

help: fix a py3 error interpolating Set into b'%s' I can't reproduce it, but a coworker hit this with `hg help -v` with 6.2.3: ... File "mercurial\help.pyc", line 865, in helplist TypeError: %b requires a bytes-like object, or an object that implements __bytes__, not 'set' I can confirm that the original expression fails in `hg debugshell`, and the new one works. The second instance was found by searching for "%s", but PyCharm detects a lot of variables as Any type, so I have no idea if there are other lurking problems.

Added signature for changeset c890d8b8bc59

Added tag 6.3.1 for changeset c890d8b8bc59

relnotes: add 6.3.1

memory-usage: fix `hg log --follow --rev R F` space complexity When running `hg log --follow --rev REVS FILES`, the log code will walk the history of all FILES starting from the file revisions that exists in each REVS. Before doing so, it looks if the files actually exists in the target revisions. To do so, it opens the manifest of each revision in REVS to look up if we find the associated items in FILES. Before this changeset this was done in a way that created a changectx for each target revision, keeping them in memory while we look into each file. If the set of REVS is large, this means keeping the manifest for each entry in REVS in memory. That can be large… if REV is in the form `::X`, this can quickly become huge and saturate the memory. We have seen usage allocating 2GB per second until memory runs out. So this changeset invert the two loop so that only one revision is kept in memory during the operation. This solve the memory explosion issue.

tests: run many tests in $TESTTMP/repo instead of $TESTTMP This is useful so we can store other files in $TESTTMP (in particular tests that use docket files (nodemap, dirstate-v2) keep file uids in $TESTTMP/UID)

tests: fix the detection of dirstate-v2 in hghave.py

dirstate-v2: do not put the dirstate data file in a transaction, since the transaction reverts the store, while the dirstate is stored separately

commit: properly consider file include and exclude options when closing branch It looks like this is meant to prevent adding another commit that does nothing but close a branch on top of a commit that already closed the branch. The matcher building functions want `Dict[bytes, Any]`, not `Dict[str, Any]`, which was found by adding type hints to the matcher related methods in scmutil.

tests: demonstrate a bug blocking a redundant branch close

tests: stop creating temporary files in TESTDIR

cffi: fix a bytes vs str issue on macOS when listing directories This code hasn't been touched in recent years, and the other implementation return bytes for the filename, so I assume this is a holdover from the py2 days. I was unable to test it on mac though, because the `_osutil` import failed.

packaging: refresh dependency hashes (issue6750) Also, add some documentation to the `.in` files.

matcher: do not prepend '.*' to pattern using ^ after flags Since the previous commit (fixing wider issue), the code generated strange regex. This is now fixed and tested.

matcher: fix the issue with regex inline-flag in rust oo Same problem same solution.

matcher: fix issues regex flag contained in pattern (issue6759) Python 3.11 is now enforcing that flag must be at the beginning of the regex This creates a serious regression for people using Python 3.11 with an hgignore using flag in a "relre" pattern. We now detect any flags in such pattern and "prepend" our ".*" pattern after them. In addition, we now insert the flag in the regexp to only affect the pattern we are rewriting. Otherwise, the regex built from the combined pattern would these flags in the middle of it anyway. As a side effect of this last change, we fix a bug… before this change regex flag in a pattern would affect all combined patterns. That was bad and is not longer the case. The Rust code needs to be updated to fix that very bug, but we will do it in another changeset.

release: removed the 6.3.0 tag The revision having two tags confuses some of the build script trying to fetch the version. So, remove the "bad" tag for now.

hg-core: relax dependencies pinning Being this rigid makes packagers' job more difficult since they might not carry the exact version. This hard pinning was introduced in eb02decdf but wasn't strictly necessary to achieve its compatibility goal.

tests: make running ls in a no longer existing directory more portable On Linux, ls -A simply returns nothing and the exit code is 0. On NetBSD, ls -A complains that . doesn't exist and the exit code is 1. Sadly, it's not possible to do something like "[1] (?)", so " || true" is the best I could come up with.

tests: move some lines inside #if windows-#else block test-removeemptydirs.t This is done to make it's clear that windows is not affected by this test case, IOW windows and non-windows platforms are now tested separately, because their results are very different.

tests: use ls -A instead of ls -1 in test-removeemptydirs.t In case the tests are run as root, ls assumes -A by default on some systems (e.g. NetBSD). Tests probably shouldn't be run as root, but let's use -A just in case, for convenience.

release: add the "6.3" to the 6.3 release The release was tagged "6.3.0" while the custom have been to omit the third ".0" for initial release for the history of the project. For consistency, we add the customary version of the tag to the 6.3 release.

tests: fix new git protocol policy in convert-git.t Recent versions of git restrict the use of the "file" protocol for security reasons (https://github.com/git/git/commit/a1d4f67). This broke this test, which failed with the following hidden error: "fatal: transport 'file' not allowed". This patch relaxes the git configuration for the test to solve this.

relnotes: add final 6.3 relnotes

Added signature for changeset 04f1dba53c96

Added tag 6.3.0 for changeset 04f1dba53c96

rust: create wrapper struct to reduce `regex` contention issues Explanations inline.

tags-fnode-cache: skip building a changectx in getfnode Building a changectx object is costly, doing it just to retrieve the revision number is suboptimal. Directly fetching the revision number from the changelog provide a sizeable speedup to `hg debugupdatecache`. ### data-env-vars.name = mercurial-2018-08-01-zstd-sparse-revlog # benchmark.name = debug-update-cache # benchmark.variants.pre-state = warm before: 0.213229 seconds after: 0.165577 seconds (-22.35%) # data-env-vars.name = mercurial-filtered-2019-11-22-zstd-sparse-revlog before: 1.200383 seconds after: 1.071618 seconds (-10.73%) # data-env-vars.name = mozilla-central-2018-08-01-zstd-sparse-revlog before: 1.465735 seconds after: 0.923128 seconds (-37.02%) # data-env-vars.name = mozilla-try-2019-02-18-zstd-sparse-revlog before: 6.511771 seconds after: 4.507316 seconds (-30.78%) # data-env-vars.name = netbeans-2018-08-01-zstd-sparse-revlog before: 1.023007 seconds after: 0.645026 seconds (-36.95%) # data-env-vars.name = pypy-2018-08-01-zstd-sparse-revlog before: 0.381141 seconds after: 0.268654 seconds (-29.51%)

tags-fnode-cache: do not repeatedly open the filelog in a loop While getting multiple hgtagsfnodecache entries, we were opening (and closing) the `.hgtags` filelog for each iteration. The meant repeatedly reading and parsing the version same information from disk. A quite costly operation. We no longer do this, leading to a sizable improvement in `hg debugupdatecache` run for an already warm repositories. ### data-env-vars.name = mercurial-2018-08-01-zstd-sparse-revlog # benchmark.name = debug-update-cache # benchmark.variants.pre-state = warm before: 1.711778 seconds after: 0.213229 seconds (-87.54%) # data-env-vars.name = pypy-2018-08-01-zstd-sparse-revlog before: 4.010817 seconds after: 0.381141 seconds (-90.50%) # data-env-vars.name = netbeans-2018-08-01-zstd-sparse-revlog before: 13.574141 after: 1.023007 seconds (-92.46%) # data-env-vars.name = mozilla-central-2018-08-01-zstd-sparse-revlog before: 18.884656 after: 1.465735 seconds (-92.24%) # data-env-vars.name = mozilla-try-2019-02-18-zstd-sparse-revlog before: 88.924823 after: 6.511771 seconds (-92.68%)

profile: prevent a crash when line number is unknown For some unknown reason, `self.lineno` can be None. The previous code crashed in such case, we now ignore the case, as we do for other error in this function. We also fallback to using `-1` in the output when this lack of line number makes it to the display code. The reason of unknown line-numbers is… unknown.

upgrade: byteify requirement changes written to output ui.write() expects bytes, and internally uses `b''` as the default when getting the `label` keyword from `*args`. So either we're missing test coverage, or there's some very subtle conversion from unicode somewhere. Also, slip in a type hint to flag this in the future.

rhg: add a config option to fall back immediately This is useful for debugging the behavior of the "default" `hg` in tests without having to manually substitute the fallback path.

rhg: stop shadowing `exit` function This will be useful for the next patch which needs it.

config: add alias from `hg help rhg` to `hg help rust` This will make using `rhg` more user-friendly and features more discoverable.

rhg: add `config.rhg` helptext This will make using `rhg` more user-friendly and features more discoverable.

config: fix indentation of some`share-safe` options This makes the output much more readable.

perf-unbundle: do a quick and dirty fix to make it run on more commit Without this change, the perf commands fails within the f67741e8264b::18415fc918a1 range (boundary excluded). Check inline comment for details. With this fix, the command is able to run on this range, with a slightly different behavior (as no revset is "uninlined"). However this is still much better than not being able to run anything in this range. Especially because that range do see some performance regression for unbundle.

perf-unbundle: pre-indent the main block in per::unbundle This makes the next changeset clearer.

shelve: handle empty parents and nodestoremove in shelvedstate (issue6748)

rhg: fallback when encountering ellipsis revisions Ellipsis revisions are still experimental and buggy in non-trivial histories. We currently have no plans to improve this situation nor to add support for ellipsis revisions in `rhg`. Falling back should be done carefully (since we may have already done some work that is visible to the user), but in this case it's highly unlikely that we're doing anything useful with a repo with ellipsis revisions, so this should be strictly better, also since the error message is more explicit.

dirstate-v2: hash the source of the ignore patterns as well Fixes the test introduced in the last changeset. This caused the hash to change, which means that the check in the test had to be adapted. Since this hash is only done as a caching mechanism, invalidation does not pose any backwards compatibility issues.

dirstate-v2: add test that shows a collision in ignore patterns hash This hash is used for optimizing dirstate `status`. We demonstrate that the hash is incorrectly ignoring the changes to the semantics of the ignore files just because the contents (but not their source) haven't changed. This is fixed in the next changeset.

testlib: add `--raw-sha1` option to `f` This will be used in the patch fixing the ignore hash collision in dirstate-v2 to check the behavior of the new hashing function.

rust-status: save new dircache even if just invalidated There was a functional race between invalidating the cache (not acted upon until the end of the status algorithm) and populating the new cache (which relies upon an up-to-date version of the cache). We simply inform the cache populating function that we've just invalidated the cache for this particular directory since the information is present in the same scope.

rust-status: fix typos and add docstrings to dircache related fields

rhg: show a bug where repeated [hg status] is needed to cache everything

upgrade: no longer keep all revlogs in memory at any point Keeping all object open is unsustainable, so we will open them on demand. This mean opening them multiple times, but this is a lesser evil. Each revlog consume a small amount of memory (index content, associated nodemap, etc). While there are few "big" revlog, the sheer amount of small filelog can become a significant issue memory wise, consuming multiple GB of memory. If you combines this extra usage with the use of multiprocessing, this usage can quickly get out of control. This can effectively block the upgrade of larger repository. This changeset fixes this issue.

demandimport: convert ignored modules from bytes -> str in extensions The default list of ignored modules are str, and test for bypassing the lazy import is `module.__name__ in ignores`, so these were effectively NOT ignored. Most of these date back to the grand byteification in 687b865b95ad, with some subsequent additions that followed the existing example. I have no idea if these modules in fact need to bypass lazy importing, but at least it follows the intent of the code.

dirstate-v2: fix edge case where entries aren't sorted See previous commit for more details.

dirstate-v2: highlight a bug when Python-packed but used in `rhg` The Python packer creates unsorted entries in the edge case that a file starts with the same name as a sibling folder. This bug has no effect on the Python `hg status` since Python ignores directories. `rhg` assumes that all on-disk entries are sorted (which is a property of the format) including folder, hence the issue highlighted. This is also technically broken in Rust-augmented `hg status`, but it makes setting up the test more complex than necessary, since it requires the packing to be Python only (which it isn't if you have Rust extensions). Fix is in the next commit.

dirstate-v2: correct documented return values of `pack_dirstate`

dirstate-v2: fix typos in docstrings

dirstate-v2: update constant that wasn't kept in sync Despite the best efforts of the comment, this constant wasn't kept in sync when the flags were being rewritten. The fact that this doesn't actually break anything in the Rust implementation too much (which does use directories) relies on the fact that all nodes can have children and that dirstate traversal is not based on that flag, but for metadata in optimizations. However the bug could become more serious should we start encoding stronger guarantees using a combination of flags including this one.