closehead: migrate `opts` to native kwargs

churn: migrate `opts` to native kwargs

children: migrate `opts` to native kwargs

automv: migrate `opts` to native kwargs

absorb: migrate `opts` to native kwargs

transaction: fix __repr__() and make the default name bytes This likely was always wrong on py3, since going back to aff5996f3043, these were added as a r-strings. Callers seem to always be supplying bytes, which makes the `b'/'.join(...)` part OK, but then bytes can't be interpolated into str with "%s", so it wouldn't have worked in either case.

rhg: support rhg files [FILE] This comes mostly for free after the rhg status [FILE] implementation.

rhg: support "status FILE" This change adds a new [file] argument to rhg status, parses them as patterns, canonicalizes the paths, and constructs a new PatternMatcher to intersect with the existing matcher being passed to the status implementation. We also make filepatterns a public module so we can access the pattern-parsing functionality we need from commands/status.rs.

rust-matchers: add PatternMatcher This should match the behavior of the Python patternmatcher. We need this for status [FILES] support.

rust-status: only visit parts of the tree requested by the matcher This is an optimization that the matcher is designed to support, but we weren't doing it until now. This is primarily relevant for supporting "hg status [FILES]", where this optimization is crucial for getting good performance (without this optimization, that command will still scan the entire tree, and just filter it down after the fact). When this optimization fires we have to return false from traverse_fs_directory_and_dirstate, representing that that part of the tree *might* have new files which we didn't see because we skipped parts of it. This only affects the cached result of the status, and is necessary to make future status operations (which might use a different matcher) work properly.

rust-status: error on non-existent files in file_set file_set here consists of the files that were explicitly specified on the command line. Erroring on them matches the behavior of Python status.

rust-status: explicitly track bad file types Before this, we silently skipped bad file types. Now, we check to see if a path is an exact_match in our matcher, and if so, print an error for it, and only then do we skip it. A path will be an exact match when it's specified as an explicit command line argument, and this error-printing behavior is necessary for compatibility with Python status.

rust: de-hardcode glob_suffix We're adding patternmatcher in a subsequent commit, and this needs needs to be different for includematcher and patternmatcher.

rust: improve the type on DirsMultiset::from_manifest It could only return an HgPathError, but we didn't express this in the type, so we needed some unreachable!()s. Now that is expressed in the type.

rust: don't escape spaces in regex Spaces are not in fact a regex special character, and escaping them is not correct.

rust-filepatterns: also normalize RelPath These patterns should be normalized too; this at least does no harm, and is necessary for proper status support.

rust: simplify pattern file parsing Previously we kept the pattern syntax as a &[u8] until the last possible moment, which meant we had to handle potential errors when parsing that &[u8]. But such errors could never actually occur, given the structure of the code. Now we parse it eagerly (in two places) and pass it around as a PatternSyntax, so we can delete some error handling code. parse_one_pattern is in particular useful for parsing patterns passed on the command line, as we'll support later in this series.

rust-matchers: better support file_set in IntersectionMatcher Previously, if both of the input matchers had non-exact file sets, IntersectionMatcher would simply have an empty fileset. Now, we duplicate Python behavior: we *union* the filesets of the matchers in that case. This makes some sense, because without exact input file sets, we can't do an exact intersection and must over-approximate. Concretely, this is necessary because the file_set is the list of files that were explicitly listed by the user, and we want to provide explicit errors for all such files, from both matchers.

branching: merge stable into default

version: migrate `opts` to native kwargs

verify: migrate `opts` to native kwargs

tags: migrate `opts` to native kwargs

tag: migrate `opts` to native kwargs

summary: migrate `opts` to native kwargs This has the added benefit of unconfusing PyCharm- it thought `opts` in the nested functions were str based for some reason.

revert: migrate `opts` to native kwargs

rename: drop an unnecessary variable declaration at a wider scope

rename: migrate `opts` to native kwargs

remove: migrate `opts` to native kwargs

purge: migrate `opts` to native kwargs

pull: avoid referencing a variable before it is initialized If `urlutil.get_pull_paths()` somehow yields nothing, the reference after the loop appears to be a problem. I noticed this because PyCharm flagged (and still flags) the initialization of this inside the loop as somehow unused. I'm not sure why 1 isn't returned directly where it is set to the result of `postincoming()`, but maybe the code is structured this way in anticipation of future work.

pull: migrate `opts` to native kwargs

paths: migrate `opts` to native kwargs

merge: migrate `opts` to native kwargs

manifest: migrate `opts` to native kwargs

locate: migrate `opts` to native kwargs

import: migrate `opts` to native kwargs

heads: migrate `opts` to native kwargs

forget: migrate `opts` to native kwargs

copy: drop an unused variable

copy: migrate `opts` to native kwargs

config: use the writefile() utility to create the sample hgrc This function uses a context manager.

config: migrate `opts` to native kwargs

clone: migrate `opts` to native kwargs

bundle: migrate `opts` to native kwargs

branches: migrate `opts` to native kwargs

branch: migrate `opts` to native kwargs

bookmark: migrate `opts` to native kwargs There are a bunch of callers for `ui.formatter()`, so convert to bytes only for that call.

archive: migrate `opts` to native kwargs I'm not sure how to handle migrating core APIs like `scmutil.match`, so kick that can down the road. (Plus it's used all over, so the callers need to be migrated first.)

backout: migrate `opts` to native kwargs It will take a bit to unwind `cmdutil.commit`, so there's a conversion to byteskwargs there, without changing the type of `opts` in this function. That's also useful to flag it as needing to be changed.

cmdutil: migrate `opts` on commitstatus() to native kwargs

cat: avoid round tripping **opts -> byteskwargs -> strkwargs Some internal users still want byteskwargs, so they are serviced in place. Once this pattern of changing types is eliminated everywhere, the remaining internal uses can be cleaned up individually (hopefully).

gpg: fix an UnboundLocalError whenever using --force It looks like this has been broke for almost a decade, since 9c89ac99690e.

rhg: fix the bug where sparse config is interpreted as relglob instead of glob relglob apparently (in contrast with relpath) matches everywhere in the tree, whereas glob only matches at the root. The python version interprets these patterns as "glob" (see "normalize(include, b'glob', ...)" in match.py)

hgweb: encode WSGI environment using the ISO-8859-1 codec The WSGI specification (PEP 3333) specifies that on Python 3 all strings passed by the server must be of type str with code points encodable using the ISO 8859-1 codec. For some reason, I introduced a bug in 2632c1ed8f34 by applying the reverse change. Maybe I got confused because PEP 3333 says that arbitrary operating system environment variables may be contained in the WSGI environment and therefore we need to handle the WSGI environment variables like we would handle operating system environment variables. The bug mentioned in the previous paragraph and fixed by this changeset manifested e.g. in the path of the URL being encoded in the wrong way. Browsers encode non-ASCII bytes with the percent-encoding. WSGI servers will decode the percent-encoded bytes and pass them to the application as strings where each byte is mapped to the corresponding code point with the same ordinal (i.e. it is decoded using the ISO-8859-1 codec). Mercurial uses the bytes type for these strings (which makes much more sense), so we need to encode it again using the ISO-8859-1 codec. If we use another codec, it can result in nonsense.

branching: merge stable into default

Added signature for changeset 787af4e0e8b7

Added tag 6.5.1 for changeset 787af4e0e8b7

relnotes: add 6.5.1

perf: fix perf::tags That return is invalid and unnecessary.

revlog: fix the naming scheme use by split temporary file The `-s` is now added on the first piece only and the `.i` is added to the index. This match the initially intended naming scheme.

revlog: show that temporary split file use broken name scheme The `-s` is added on the wrong part and the `.i` is missing.

fncache: fix a bug that corrupts the fncache after transaction rollback

fncache: add a test demonstrating fncache corruption

transaction: actually delete file created during the transaction on rollback Transaction currently has two modes: - one where file created during the transaction are deleted on rollback, - one where file created during the transaction are truncated to 0 on rollback. Before this change, `hg rollback` and `hg recover` are using the "delete" mode and transaction abort is using the "truncate" option. This difference is never really explained. A long time ago, there was two code paths, with this divergence existing for unclear reasons. When the two code paths got merged into a single one, a boolean argument have been added to preserve this divergence, mostly probably as a cargo cult. The divergence is weird and induce bad surprises, and the truncate behavior is a bit odds, introducing other bad surprises (e.g. 08ecbdba186f) So solve this, we stop using the "truncate" behavior and unify on the "delete" behavior. Despite being currently more "common", the truncate behavior seems less natural, resulting in the transaction leaving empty file around. This is landed on default, early in the cycle, to help us catch problems that could emerge.

rhg: tweak rhg fallback code structure The goal here is to share the code that accesses the config setting, but also move it to the top-level, so that we can more easily change this code later. (in fact in Jane Street we want to make it choose chg in some cases)

journal: track bookmark deletion Bookmark deletion are now properly tracked by the journal extensions.

journal: show that we do not track deletion This was an oversight for the initial implementation.

contrib: upgrade merge-lists from clap v3 to clap v4 Mostly to avoid having clap v3 in our dependencies, due to https://osv.dev/vulnerability/GHSA-g98v-hv3f-hcfr (even though that seems pretty harmless).

compat: use cp -R in `test-revlog-delta-find.t` (issue6835) This follow POSIX and make things compatible with NetBSD.

compat: use cp -R in `test-bundle-phase-internal.t` (issue6835) This follow POSIX and make things compatible with NetBSD.

infinitepush: drop the extension The writing have been on the wall for a long time.

perf: display all timing by default This is much more useful and end up enable it everywhere. I don't think we have strong backward compatibility guarantee for perf.

wrapfunction: deprecated calling "wrapfunction" with bytes This will help us to detect and eradicates legacy calls. This is on the path to deprecating `util.safehasattr`.

wrapfunction: deprecates calling `wrappedfunction` with bytes This will help us to detect and eradicates legacy calls. This is on the path to deprecating `util.safehasattr`.

wrapfunction: use sysstr instead of bytes as argument in various tests This is as valid and simpler, it will help us to eventually get ride of `safehasattr`.

wrapfunction: use sysstr instead of bytes as argument in "exthelper" This is as valid and simpler, it will help us to eventually get ride of `safehasattr`.

wrapfunction: use sysstr instead of bytes as argument in "sqlitestore" This is as valid and simpler, it will help us to eventually get ride of `safehasattr`.

wrapfunction: use sysstr instead of bytes as argument in "share" This is as valid and simpler, it will help us to eventually get ride of `safehasattr`.

wrapfunction: use sysstr instead of bytes as argument in "perf" This is as valid and simpler, it will help us to eventually get ride of `safehasattr`.

wrapfunction: use sysstr instead of bytes as argument in "zeroconf" This is as valid and simpler, it will help us to eventually get ride of `safehasattr`.

wrapfunction: use sysstr instead of bytes as argument in "sqlitestore" This is as valid and simpler, it will help us to eventually get ride of `safehasattr`.

wrapfunction: use sysstr instead of bytes as argument in "sparse" This is as valid and simpler, it will help us to eventually get ride of `safehasattr`.

wrapfunction: use sysstr instead of bytes as argument in "schemes" This is as valid and simpler, it will help us to eventually get ride of `safehasattr`.

wrapfunction: use sysstr instead of bytes as argument in "remotenames" This is as valid and simpler, it will help us to eventually get ride of `safehasattr`.

wrapfunction: use sysstr instead of bytes as argument in "remotefilelog" This is as valid and simpler, it will help us to eventually get ride of `safehasattr`.

wrapfunction: use sysstr instead of bytes as argument in "pager" This is as valid and simpler, it will help us to eventually get ride of `safehasattr`.

wrapfunction: use sysstr instead of bytes as argument in "narrow" This is as valid and simpler, it will help us to eventually get ride of `safehasattr`.

wrapfunction: use sysstr instead of bytes as argument in "lfs" This is as valid and simpler, it will help us to eventually get ride of `safehasattr`.

wrapfunction: use sysstr instead of bytes as argument in "largefiles" This is as valid and simpler, it will help us to eventually get ride of `safehasattr`.

wrapfunction: use sysstr instead of bytes as argument in "keyword" This is as valid and simpler, it will help us to eventually get ride of `safehasattr`.

wrapfunction: use sysstr instead of bytes as argument in "journal" This is as valid and simpler, it will help us to eventually get ride of `safehasattr`.

wrapfunction: use sysstr instead of bytes as argument in "infinitepush" This is as valid and simpler, it will help us to eventually get ride of `safehasattr`.

wrapfunction: use sysstr instead of bytes as argument in "histedit" This is as valid and simpler, it will help us to eventually get ride of `safehasattr`.

wrapfunction: use sysstr instead of bytes as argument in "highlight" This is as valid and simpler, it will help us to eventually get ride of `safehasattr`.

wrapfunction: use sysstr instead of bytes as argument in the "git" extension This is as valid and simpler, it will help us to eventually get ride of `safehasattr`.

wrapfunction: use sysstr instead of bytes as argument in "fsmonitor" This is as valid and simpler, it will help us to eventually get ride of `safehasattr`.

wrapfunction: use sysstr instead of bytes as argument in "fastannotate" This is as valid and simpler, it will help us to eventually get ride of `safehasattr`.

wrapfunction: use sysstr instead of bytes as argument in "clonebundles" This is as valid and simpler, it will help us to eventually get ride of `safehasattr`.

wrapfunction: use sysstr instead of bytes as argument in "bookflow" This is as valid and simpler, it will help us to eventually get ride of `safehasattr`.

wrapfunction: use sysstr instead of bytes as argument in "beautifygraph" This is as valid and simpler, it will help us to eventually get ride of `safehasattr`.

byteify-strings: passe sysstr to attr function and wrapper These function now accepts string so we should not do byteify them anymore.

repoview: fix the filter created by `extrafilter` The `name` variable was updated in each interaction of the loop, making all new filters inheriting from the same base (the last one iterated), leading to terrible terrible misbehavior.

repoview: show `extrafilter` misbehave when other filtering is in play See new changeset for fix and details.

rust-config: add docstring to inner `get_parse` method The signature is quite heavy, a docstring to explain the idea doesn't hurt.

rust-config: add devel warning when using undeclared config items This mirrors the Python implementation now that we're done catching up.

rhg: use default configitem in `cat` Now that we've declared the items, we can get rid of the manual fallback.

rhg: add config defaults to configitems.toml This will allow us to use the same default config mechanic as the rest of the code.

rust-blackbox: use `is_extension_enabled` config helper It's there, may as well use it to make the code clearer and less bug-prone.

configitems: move blackbox's config items to the new configitems.toml In order for the Rust code to gain access to default values of in-core extensions that have a Rust implementation, we need to centralize them alongside the core items declarations. This is the first and so far only one of the extensions that have gained Rust support, I don't think it's worth the churn to move the rest of the extension's configitems yet.

rust-config: add config getters that don't fall back to defaults This is useful in cases where we access config items that are more... lenient with their types than a fresh new system would allow. For now there is only a single use of this, but we might get more later.

rust-config: add support for default config items Now that configitems.toml exists, we can read from it the default values for all core config items. We will add the devel-warning for use of undeclared config items in a later patch when we're done adding the missing entries for `rhg`.

configitems: declare items in a TOML file Mercurial ships with Rust code that also needs to read from the config. Having a way of presenting `configitems` to both Python and Rust is needed to prevent duplication, drift, and have the appropriate devel warnings. Abstracting away from Python means choosing a config format. No single format is perfect, and I have yet to come across a developer that doesn't hate all of them in some way. Since we have a strict no-dependencies policy for Mercurial, we either need to use whatever comes with Python, vendor a library, or implement a custom format ourselves. Python stdlib means using JSON, which doesn't support comments and isn't great for humans, or `configparser` which is an obscure, untyped format that nobody uses and doesn't have a commonplace Rust parser. Implementing a custom format is error-prone, tedious and subject to the same issues as picking an existing format. Vendoring opens us to the vast array of common config formats. The ones being picked for most modern software are YAML and TOML. YAML is older and common in the Python community, but TOML is much simpler and less error-prone. I would much rather be responsible for the <1000 lines of `tomli`, on top of TOML being the choice of the Rust community, with robust crates for reading it. The structure of `configitems.toml` is explained inline.

thirdparty: vendor tomli The next commit will introduce a .toml file to abstract configitems away from Python. Python 3.11 has a toml read-only library (`tomllib`), which gives us a way out of vendoring eventually. For now, we vendor the backport, specifically version 1.2.3 which is still compatible with Python 3.6.

configitems: add `documentation` field It may be useful to expose documentation information in the help in some form. This will be populated in a future changeset by using the current comments that are relevant for users.

configitems: use standard "dynamicdefault" approach in edge case This makes for fewer edge cases, which will help a future patch.

configitems: fix typo in devel warning about extension overrides

extensions: imp module is removed in Python 3.12 - use importlib to load files imp has been deprecated for a long time, and has finally been removed in Python 3.12 . imp was only used for loading extensions that has been specified with direct .py path or path to a package directory. The same use cases can be achieved quite simple with importlib, , possiby with small changes in corner cases with undefined behaviour, such as extensions without .py source. There might also be corner cases and undefined behaviour around use of sys.modules and reloading.

utils: imp module is removed in Python 3.12 - get is_frozen() from _imp imp has been deprecated for a long time, and has finally been removed in Python 3.12 . The successor importlib is using the same internal _imp module as imp, but doesn't expose it's is_frozen. Using the internal function directly seems like the cleanest solution. Another alternative to imp.is_frozen("__main__") is sys.modules['__main__'].__spec__.origin == 'frozen' but that seems even more internal and fragile.

extensions: address ast deprecations introduced in Python 3.12 Tests would fail with: .../mercurial/extensions.py:910: DeprecationWarning: ast.Str is deprecated and will be removed in Python 3.14; use ast.Constant instead if isinstance(a, ast.Str): .../mercurial/extensions.py:912: DeprecationWarning: ast.Bytes is deprecated and will be removed in Python 3.14; use ast.Constant instead elif isinstance(a, ast.Bytes): .../mercurial/extensions.py:913: DeprecationWarning: Attribute s is deprecated and will be removed in Python 3.14; use value instead name = a.s

vfs: handle shutil.rmtree deprecation of onerror in Python 3.12 Tests would fail with warnings: .../mercurial/vfs.py:289: DeprecationWarning: onerror argument is deprecated, use onexc instead The excinfo changed slightly, but we don't use it anyway.

tests: fix sortdict doctest with Python 3.12 The output of OrderedDict changed to use plain dict syntax: $ python3.11 -c "import collections;print(collections.OrderedDict([('a', 0), ('b', 1)]))" OrderedDict([('a', 0), ('b', 1)]) $ python3.12 -c "import collections;print(collections.OrderedDict([('a', 0), ('b', 1)]))" OrderedDict({'a': 0, 'b': 1})

utils: stop using datetime.utcfromtimestamp() deprecated in Python 3.12 Python3.12 made tests fail with warnings: DeprecationWarning: datetime.utcfromtimestamp() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.fromtimestamp(timestamp, datetime.UTC). Computing the diff while in timestamp seconds seems to preserve to the original intent from ae04af1ce78d. It would be nice to have some doctest coverage of this, with the problematic corner cases that has popped up over time...

branching: merge stable into default

Added signature for changeset 3ffc7209bbae

Added tag 6.5 for changeset 3ffc7209bbae

relnotes: add 6.5 final

rust-changelog: removed now useless early conditional for NULL_REVISION Now that the underlying `RevlogEntry` returned for `NULL_REVISION` is viable, it has become pointless to check NULL_REVISION several times, even for performance: the check will be far more frequent than the benefit of bailing earlier in the rare case where the requested revision is `NULL_REVISION`. This case is covered explicitly by the first assertion of `test_data_from_rev_null`.

rust-revlog: fix RevlogEntry.data() for NULL_REVISION Before this change, the pseudo-entry returned by `Revlog.get_entry` for `NULL_REVISION` would trigger errors in application code using it. For example, this fixes a crash spotted with changelog data while implementing RHGitaly: `Changelog.data_for_rev(-1)` was already returning the pseudo content as expected, e.g., for `hg log`, but `Changelog.entry_for_rev(-1).data()` would still crash with "corrupted revlog, hash check failed for revision -1". There is an added test for this scenario.

rust-revlog: using constant in test Always better

rust-revlog: fix incorrect results with NULL_NODE prefixes In case a short hash is a prefix of `NULL_NODE`, the correct revision number lookup is `NULL_REVISION` only if there is no match in the nodemap. Indeed, if there is a single nodemap match, then it is an ambiguity with the always matching `NULL_NODE`. Before this change, using the Mercurial development repository as a testbed (it has public changesets with node ID starting with `0005` and `0009`), this is what `rhg` did (plain `hg` provided for reference) ``` $ rust/target/debug/rhg cat -r 000 README README: no such file in rev 000000000000 $ hg cat -r 000 README abort: ambiguous revision identifier: 000 ``` Here is the expected output for `rhg` on ambiguous prefixes (again, before this change): ``` $ rust/target/debug/rhg cat -r 0001 README abort: ambiguous revision identifier: 0001 ``` The test provided by 8c29af0f6d6e in `test-rhg.t` could become flaky with this change, unless all hashes are fixed. We expect reviewers to be more sure about that than we are.

rust-revlog: split out method for `rev_from_node` without persistent nodemap This will make easier for the bug fix that is about to come.

win32mbcs: unbyteify some strings for py3 support A crash was reported on the TortoiseHg bug tracker for this[1]. [1] https://foss.heptapod.net/mercurial/tortoisehg/thg/-/issues/5905

heptapod-ci: turn off pipelines for merge request events These are always duplicated by the external (from the user) push, or internal (from Heptapod itself) push pipeline, so they're entirely redundant.

heptapod-ci: always make the default run condition explicit This has no effect on behavior, it's just to make everything obvious.

heptapod-ci: remove push exception for named branches The internal workings of Heptapod cause even fast-forward merges to look like pushes to Gitlab. The only way to run a pipeline on topic-less named branches is not through the web interface. I am probably the only person affected by this, so it's not really a problem. This was confirmed with Georges, the maintainer of Heptapod, though the only way to be *actually* sure, is to merge this and see.

heptapod-ci: (hopefully) fix the pipeline savings system There is no easy way for me to test this outside of the actual repo since it would require setting up `heptapod-runner` locally, making a Git repo with the right changes, etc. It's not worth my time at this point. If this fails, I'll try to ask for help from Georges, the maintainer of Heptapod.

heptapod-ci: don't run pipelines for topic-less branches See inline comment for more details.

clonebundles: add warning if auto-generate is enabled without formats

clonebundles: fix display of auto-generate.on-change lines

hgweb: drop references to deprecated cgitb cgitb is going away and gives warnings when importing, and that make tests fail: $TESTTMP/hgweb.cgi:5: DeprecationWarning: 'cgitb' is deprecated and slated for removal in Python 3.13 The lack of a "nice" high level error handler is not a huge problem, neither for users (where it is disabled anyway) or for tests (where we don't use a browser and the plain tracebacks often are more readable). It is inevitable that it is going away, and there is no obvious alternative. Remove it and move on.

setup: fall back to setuptools setup if distutils isn't available The setuptools comments around this seems slightly outdated. Setuptools is improving and distutils is being deprecated, so it should perhaps be the default. But at least, it is a fair fallback.

tests: use simple mock smtp server instead of deprecated asyncore smtpd test-patchbomb-tls.t would fail with: .../hg/tests/dummysmtpd.py:6: DeprecationWarning: The asyncore module is deprecated and will be removed in Python 3.12. The recommended replacement is asyncio import asyncore .../hg/tests/dummysmtpd.py:8: DeprecationWarning: The smtpd module is deprecated and unmaintained and will be removed in Python 3.12. Please see aiosmtpd (https://aiosmtpd.readthedocs.io/) for the recommended replacement. import smtpd The recommended migration path to the standalone asiosmtpd would be overkill. The tests do not need a full smtp server - we can just use a very simple mock hack to preserve the existing test coverage.

tests: fix dummysmtpd argument check

tests: improve test-patchbomb-tls.t by by logging errors and data The actual SSL error might be like: ::1 ssl error: [SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:1002) and will probably vary so much that it can't be checked in the test. It is however very useful when debugging failures.

tests: show test-patchbomb-tls.t smtp server log Improve test coverage by exposing what the smtp server actually receives. Make dummystmtpd redirect stderr to a log file.

demandimport: don't delay _distutils_hack import test-demandimport.py would fail on 'import distutils.msvc9compiler' because warnings: /usr/lib/python3.11/site-packages/_distutils_hack/__init__.py:18: UserWarning: Distutils was imported before Setuptools, but importing Setuptools also replaces the `distutils` module in `sys.modules`. This may lead to undesirable behaviors or errors. To avoid these issues, avoid using distutils directly, ensure that setuptools is installed in the traditional way (e.g. not an editable install), and/or make sure that setuptools is always imported before distutils. warnings.warn( /usr/lib/python3.11/site-packages/_distutils_hack/__init__.py:33: UserWarning: Setuptools is replacing distutils. warnings.warn("Setuptools is replacing distutils.") Telling demandimport to ignore this module will allow the hack to work as intended. Note: The test for distutils.msvc9compiler comes from 2205d00b6d2b. But since then, distutils is going away, and setuptools has moved forward and is replacing it. It is unclear exactly what is being tested here and how setuptools should depended on msvc9compiler. The test might no longer be relevant.

tests: update test-remotefilelog-gc.t for Python 3.11 The test output changed because test coverage changed because normpath changed: $ python3.10 -c 'import os; print(repr(os.path.normpath("asdas\0das")))' 'asdas\x00das' $ python3.11 -c 'import os; print(repr(os.path.normpath("asdas\0das")))' 'asdas'

tests: use grep -F instead of obsolescent fgrep Testing on Fedora 38 failed with: fgrep: warning: fgrep is obsolescent; using grep -F The warning comes from https://git.savannah.gnu.org/cgit/grep.git/commit/?id=a9515624709865d480e3142fd959bccd1c9372d1 . For further anecdotal evidence of the change, see https://www.phoronix.com/news/GNU-Grep-3.8-Stop-egrep-fgrep . grep -F is POSIX, but there is a risk that it doesn't work the same on all platforms - especially older Unix versions. It should however always be possible to put a GNU grep in $PATH before running the tests.

tests: use grep -E instead of obsolescent egrep Testing on Fedora 38 failed with: egrep: warning: egrep is obsolescent; using grep -E The warning comes from https://git.savannah.gnu.org/cgit/grep.git/commit/?id=a9515624709865d480e3142fd959bccd1c9372d1 . For further anecdotal evidence of the change, see https://www.phoronix.com/news/GNU-Grep-3.8-Stop-egrep-fgrep . This reverses the code check that goes back to e7d3b509af8b. grep -E is POSIX, but there is a risk that it doesn't work the same on all platforms - especially older Unix versions. It should however always be possible to put a GNU grep in $PATH before running the tests.

dirstate: avoid leaking disk space in `hg debugrebuilddirstate` Before this MR running `hg debugrebuilddirstate` simply grows the dirstate without bound, never shrinking it, because the unused bytes counter stays low, even though the entirety of the file becomes unused.

dirstate-v2: actually fix the dirstate-v2 upgrade race It looks like the previous fix for the dirstate-v2 upgrade race didn't work. The problem is that it only recovers in case the size of the v1 `dirstate` file is smaller than the `v2` one, whereas in real life it's always larger. This commit changes the test to be more realistic, which reveals the crash, and changes the code to fix the crash.

Added signature for changeset bb42988c7e15

Added tag 6.5rc0 for changeset bb42988c7e15

relnotes: add 6.5rc0

branching: merge default into stable for 6.5rc0

branching: merge stable into default

Added signature for changeset 271a4ab29605

Added tag 6.4.5 for changeset 271a4ab29605

relnotes: add 6.4.5

narrow: indicated the default of 'Yes' when confirming auto-remove-includes

clone-bundles: check we generate a valid specification The short bundle-spec are ambiguous and should be expanded to a full one before clone bundle can use it. We fix the issue and add a test actually using the generated manifest for a clone bundle operation.

bundle-spec: add an official method to format a spec into a string This will be useful for automatic-clonebundles.

clonebundles: adds a auto-generate.serve-inline option This new option disable the "external" serving of auto generated bundle in favor of using the new "inline bundle" feature that allow serving such bundle through the mercurial protocol. This is a less efficient, but very easy to setup way to start using clone bundle.

clone-bundle: only add the REQUIRESNI bit for http(s) we don't needs it for `peer-bundle-cache://` for example

stream-clone: add a stream-v3 variant of test-persistent-nodemap.t This will be useful to track and validate future changes to the stream-v3 protocol.

stream-clone: split the stream part out of test-persistent-nodemap.t We want to test multiple variants, so let's make it an independent test. Small adaptations and simplifications are needed in the test output as the tested repository is now shorter.

stream-clone: explicitly split debug test output for application per version Altering the format may introduce significant changes in output and in order. It is going to be simpler to deal with explicitly different blocks for the output.

stream-clone: add a v3 version of the protocol This new version is less rigid regarding the extract number of files and number of bytes to be actually transfered, it also lays the groundwork for other improvements. The format stays experimental, but this is an interesting base to build upon.

stream-clone: fix the help for an argument of perf::stream-locked-section The help for `--stream-version` had a typo and missed a value.

infinitepush: aggressively deprecated infinite push We add a large warning and an explicit abort. Both can be disable through config, but neither are explicitly documented. If any user exists I really want them to get in touch with the community.

clone-bundle: double check that inline clone bundle does not break local case Since there is a large NotImplementedError in the code, I prefer we have a double check here.

clone-bundle: drop the now unused `inlineclonebundles` capabilities We no longer us the capabilities as a side way to passe argument to the `clonebundle commands`

clonebundles: introduce a new write protocol command The old wire command does not take argument so we need a new command. We make that new command take argument to prepare it for the future (like narrow support).

clone-bundles: reorder some test section and add titles This make things clearer around the inline clone bundle tests.

clone-bundle: rename the methods and wireprotole command The current naming is very clone centric, since then, the url scheme was moved to a more generic wording to make room for other use-cases (e.g. pull bundles). We apply the same logic to the internal API and, very important prior to the release, to the wire protocol command.

delta-find: remove dead code intended to deal with forced delta reuse Since the case was dealt with sooner (see XXX), we no longer need to deal with it in this part of the code.

library: incorporate demandimport into runpy invocation

library: enable runpy invocation on mercurial package

match: add `filepath:` pattern to match an exact filepath relative to the root It's useful in certain automated workflows to make sure we recurse in directories whose name conflicts with files in other revisions. In addition it makes it possible to avoid building a potentially costly regex, improving performance when the set of files to match explicitly is large. The benchmark below are run in the following configuration : # data-env-vars.name = mozilla-central-2018-08-01-zstd-sparse-revlog # benchmark.name = files # benchmark.variants.rev = tip # benchmark.variants.files = all-list-filepath-sorted # bin-env-vars.hg.flavor = no-rust It also includes timings using the re2 engine (through the `google-re2` module) to show how much can be saved by just using a better regexp engine. Pattern time (seconds) time using re2 ----------------------------------------------------------- just "." 0.4 0.4 list of "filepath:…" 1.3 1.3 list of "path:…" 25.7 3.9 list of patterns 29.7 10.4 As you can see, Without re2, using "filepath:" instead of "path:" is a huge win. With re2, it is still about three times faster to not have to build the regex.

path: use the next `display` argument to deal with boolean We have a generic mechanism that we can now use for the special case now.

path: display proper user facing value for pulled-delta-reuse-policy The integer was not what the user expect nor what could be provided as a config value.

paths: add an argument to format the suboption display We will use it in the next function to the delta policy display. It could also be use to deal with the other special case in the command code, but that is unnecessary churn for stable so that part will go on default.

ui: fix crash in `hg paths` when pulled-delta-reuse-policy is in use The crash is a result of [value] being an int, but being used as a bytestring.

ui: add a test demonstrating a crash in `hg paths`

re2: fix reporting of availability in `hg debuginstall` We add and use an official API to check is re2 is available. This prevent the bug previously in place were debuginstall was considering confusing `None` (i.e. non-initialized) and `False` (i.e. unavailable).

re: make _checkre2 a static method There is no need to it to be a method, so we make it at least a static method to make it easier to call.

re2: exit `_checkre2` early if calling it is useless If we already know if re2 is supported there is no need to run the check again. Stricly speaking we don't call it twice, but this seems a simple enough safeguard to include in this series.

branching: merge with stable

backout: remove db9b6ce803a9 That changeset lost its topic for unknown reason and ended up being picked by some automation. So I backout it previous incarnation until it can get properly reviewed.

infinitepush: aggressively deprecated infinite push We add a large warning and an explicit abort. Both can be disable through config, but neither are explicitly documented. If any user exists I really want them to get in touch with the community.

rust-dependencies: switch from `users` to `whoami` `users` is unmaintained, with the maintainer apparently MIA. `whoami` is a popular crate that does simple wrapping of platform-specific calls. This makes the overhead from using `blackbox` lower. It used to be up 10ms on slower hardware for both calls, now down to <1ms which is always good.

rust-hg-core: move from `ouroboros` to `self_cell` `ouroboros` has a fundamental soundness problem that, while not applicable today, could become applicable given new compiler optimizations.¹ `self_cell` is a crate that accomplishes a lot of the same things that `ouroboros` did while remaining sound (that is, unless a new soundness issue is discovered) by not assuming as much about the memory layout of the program. `self_cell` has been scrutinized heavily in the past few months by very competent people, some from the compiler team and has shown no weaknesses for a while, with a 1.0 stable release coming out a couple months ago. Our internal API is exactly the same, this is just an implementation detail. To reiterate, no actual soundness issue was found with our use of `ouroboros`, but there might be evolutions of `rustc` (or even a future separate compiler) that could generate unsound code. [1] https://github.com/joshua-maros/ouroboros/issues/88

debug-revlog-stats: make it use the new store entry API This give use free tree manifest support in the process.

store: use the revlog type from revlog's constant instead of custom one We already have a revlog type that everybody use. So we use that one instead.

repair: use `is_manifestlog` to recognise manifest revlog This is simpler that comparing flag, and we want to remove these flags anyway.

perf: add a perf::stream-consume We know how long it take to generate, lets check how long it take to apply now.

perf: add a perf::stream-generate command This record the time we take to generate a bundle.

perf: add a new "context" argument to timer This allow to simple setup/teardown outside of the timed section. Especially using object that need context manager, like a temporary files.

perf: add support for stream-v3 during benchmark This is getting important as the v3 protocol will diverge from the v2 protocol.

perf: add a function to find a stream version generator The logic is clearer and can be reused for other commands in the future.

treemanifest: make `updatecaches` update the nodemaps for all directories Without this, if the cache for a nested directory is in a bad state, it's very hard to repair it.

stream-clone: avoid opening a revlog in case we do not need it Opening an revlog has a cost, especially if it is inline as we have to scan the file and construct an index. To prevent the associated slowdown, we just do a minimal scan to check that an inline file is still inline, and simply stream the file without creating a revlog when we can. This provides a big boost compared to the previous changeset, even if the full generation is still penalized by the initial gathering of information. All benchmarks are run on linux with Python 3.10.7. # benchmark.name = hg.exchange.stream.generate # benchmark.variants.version = v2 ### Compared to the previous changesets We get a large win all across the board! # mercurial-2018-08-01-zstd-sparse-revlog before: 0.250694 seconds after: 0.105986 seconds (-57.72%) # pypy-2018-08-01-zstd-sparse-revlog before: 3.885657 seconds after: 1.709748 seconds (-56.00%) # netbeans-2018-08-01-zstd-sparse-revlog before: 16.679371 seconds after: 7.687469 seconds (-53.91%) # mozilla-central-2018-08-01-zstd-sparse-revlog before: 38.575482 seconds after: 17.520316 seconds (-54.58%) # mozilla-try-2019-02-18-zstd-sparse-revlog before: 81.160994 seconds after: 37.073753 seconds (-54.32%) ### Compared to 6.4.3 We are still significantly slower than 6.4.3, the extra time is usually twice slower than the extra time we observe on the locked section, which is a quite interesting information. Except for mercurial-central that is much faster. That discrepancy is not really explained yet. # mercurial-2018-08-01-zstd-sparse-revlog 6.4.3: 0.072560 seconds after: 0.105986 seconds (+46.07%) (- 0.03 seconds) # pypy-2018-08-01-zstd-sparse-revlog 6.4.3: 1.211193 seconds after: 1.709748 seconds (+41.16%) (-0.45 seconds) # netbeans-2018-08-01-zstd-sparse-revlog 6.4.3: 4.932843 seconds after: 7.687469 seconds (+55.84%) (-2.75 seconds) # mozilla-central-2018-08-01-zstd-sparse-revlog 6.4.3: 34.012226 seconds after: 17.520316 seconds (-48.49%) (-16.49 seconds) # mozilla-try-2019-02-18-zstd-sparse-revlog 6.4.3: 23.850555 seconds after: 37.073753 seconds (+55.44%) (+13.22 seconds)

store: stop relying on a `revlog_type` property We want to know if a file is related to a revlog, but the rest is dealt with differently already, so we simplify things further. as a bonus, this cleanup This provides a small but noticeable speedup. The number below use `hg perf::stream-locked-section` to measure the time spend in the locked section of the streaming clone. Number are run on various repository and compare different steps.: 1) the effect of this patchs, 2) the effect of the cleanup series, 2) current state compared to because large refactoring. All benchmarks are run on linux with Python 3.10.7. ### Effect of this patch # mercurial-2018-08-01-zstd-sparse-revlog # benchmark.name = perf-stream-locked-section before: 0.030246 seconds after: 0.029274 seconds (-3.21%) # pypy-2018-08-01-zstd-sparse-revlog before: 0.545012 seconds after: 0.520872 seconds (-4.43%) # netbeans-2018-08-01-zstd-sparse-revlog before: 2.719939 seconds after: 2.626791 seconds (-3.42%) # mozilla-central-2018-08-01-zstd-sparse-revlog before: 6.304179 seconds after: 6.096700 seconds (-3.29%) # mozilla-try-2019-02-18-zstd-sparse-revlog before: 14.142687 seconds after: 13.640779 seconds (-3.55%) ### Effect of this series A small but sizeable speedup # mercurial-2018-08-01-zstd-sparse-revlog before: 0.031122 seconds after: 0.029274 seconds (-5.94%) # pypy-2018-08-01-zstd-sparse-revlog before: 0.589970 seconds after: 0.520872 seconds (-11.71%) # netbeans-2018-08-01-zstd-sparse-revlog before: 2.980300 seconds after: 2.626791 seconds (-11.86%) # mozilla-central-2018-08-01-zstd-sparse-revlog before: 6.863204 seconds after: 6.096700 seconds (-11.17%) # mozilla-try-2019-02-18-zstd-sparse-revlog before: 14.921393 seconds after: 13.640779 seconds (-8.58%) ### Current state compared to the pre-refactoring state The refactoring introduced multiple string manipulation and dictionary creation that seems to induce a signifiant slowdown Slowdown # mercurial-2018-08-01-zstd-sparse-revlog 6.4.3: 0.019459 seconds after: 0.029274 seconds (+50.44%) ## pypy-2018-08-01-zstd-sparse-revlog 6.4.3: 0.290715 seconds after: 0.520872 seconds (+79.17%) # netbeans-2018-08-01-zstd-sparse-revlog 6.4.3: 1.403447 seconds after: 2.626791 seconds (+87.17%) # mozilla-central-2018-08-01-zstd-sparse-revlog 6.4.3: 3.163549 seconds after: 6.096700 seconds (+92.72%) # mozilla-try-2019-02-18-zstd-sparse-revlog 6.4.3: 6.702184 seconds after: 13.640779 seconds (+103.53%)

store: directly pass the filesize in the `details` of revlog The dictionary only contains 1 (or 0) entries, we can directly store that information (or None). Moving to a simpler argument passing result in a noticable speedup (because Python) The number below use `hg perf::stream-locked-section` to measure the time spend in the locked section of the streaming clone. Number are run on various repository. ### mercurial-2018-08-01-zstd-sparse-revlog before: 0.031247 seconds after: 0.030246 seconds (-3.20%) ### mozilla-central-2018-08-01-zstd-sparse-revlog before: 6.718968 seconds after: 6.304179 seconds (-6.17%) ### mozilla-try-2019-02-18-zstd-sparse-revlog before: 14.631343 seconds after: 14.142687 seconds (-3.34%) ### netbeans-2018-08-01-zstd-sparse-revlog before: 2.895584 seconds after: 2.719939 seconds (-6.07%) ### pypy-2018-08-01-zstd-sparse-revlog before: 0.561843 seconds after: 0.543034 seconds (-3.35%)

store: explicitly pass file_size when creating StoreFile A small cleanup before large cleanup in the next patch.

store: have the revlog determine which files are volatile itself This is a first step toward simplifying the walk step.

test-dirstate-version-fallback: future-proof the test for a different default Dirstate-v2 will become the default at some point, which would cause this test to fail. Let's save someone else the headache later.

clonebundles: add support for inline (streaming) clonebundles The idea behind inline clonebundles is to send them through the ssh or https connection to the Mercurial server. We've been using this specifically for streaming clonebundles, although it works for 'regular' clonebundles as well (but is less relevant, since pullbundles exist). We've had this enabled for around 9 months for a part of our users. A few benefits are: - no need to secure an external system, since everything goes through the same Mercurial server - easier scaling (in our case: no risk of inconsistencies between multiple mercurial-server mirrors and nginx clonebundles hosts) Remaining topics/questions right now: - The inline clonebundles don't work for https yet. This is because httppeer doesn't seem to support sending client capabilities. I didn't focus on that as my main goal was to get this working for ssh.

Added signature for changeset da372c745e0f

Added tag 6.4.4 for changeset da372c745e0f

relnotes: add 6.4.4

revlog: avoid possible collision between directory and temporary index Since 6.4, we create a temporary index file to write the split data without overwriting the inline version too early. However, the store encoding does not prevent these new `.i.s` file to collide with a directory with the same name. While the odds for such a collision to happens are fairly low, the collision would prevent Mercurial from working. The store encoding have a mitigation solution in place to prevent such collisions from happening for `.i` and `.d` files, but not for other extensions. We cannot update this encoding scheme to solve the issue since it would diverge from older version of Mercurial. Instead, we create an alternative directory tree dedicated to such files. The use of the `.i` extension combined with store encoding will prevent collisions there.

revlog: move the computation of the split_index path in a property This is about to become more complex, so we gather the logic in a single place.

rust-dirstate: fall back to v1 if reading v2 failed This will help us not fail when a v1 dirstate is present on disk while a v2 was expected (which could happen with a racy/interrupted upgrade).

dirstate: add test showing dirstate version mismatch causes an error We should fall back to trying dirstate v1 when v2 fails to read.

rust-dirstate: rename `has_dirstate_v2` to `use_dirstate_v2` It is closer to the right semantics. I added a docstring to better explain the reasonning. In the next patch(es), I will address the underlying issue of finding the "wrong" version of the dirstate on disk.

delta-find: fix pulled-delta-reuse-policy=forced behavior The code that select delta still has too many oportunity to discard the delta is has been forcibly asked to reuse. However is is fairly easy to use a dedicated fastpath for this case. So we do so. Cleaning other code that tries to enforce that policy will be done on default.

delta-find: display more information about the search in some case This will be useful to access the effect of the delta reuse policy.

deltafind: issue debug information when we fast-path rivial case too More debug options never hurts.

delta-find: gather the condition to blindly use a full snapshot together We are about to make the `if` body bigger, so having only one of them is simpler/

delta-find: initialize the debug information much sooner (when possible) This help us to record debug information in alternative path.

delta-find: fix `parents` round detection We should compare integer with integer, instead of bytes (node).

delta-find: intrduce a `_one_dbg_data` method This helps with the initialisation of the expected debug information.

delta-find: move final debug processing in a `_dbg_process_data` method This will help us making sure we can call the necessary debugging code in alternative codepath.

delta-fine: use the `_debug_search` attribute directly By adjusting the value at creation time, we can avoid the need to use and intermediate variable.

delta-find: move the `gather_debug` logic in a property This will make it simpler to get be reused in multiple places.

tests: check `pulled-delta-reuse-policy=forced` without general delta If general delta is not available, the amount of delta we can reuse at all will be more limited. We check that application still work and does not corrupt the repository.

tree-manifest: allow `debugupgraderepo` to run on tree manifest repo There does not seems to be anything wrong with running the current logic on them. So we remove the limitation.

bundles: clarify streaming v2 bundle usage This seems useful to clarify through the documentation (we were still using debugcreatestreamclonebundle, even though 'hg bundle' is already an option...).

stream-clone: update debugcreatestreamclonebundle helps People to stop using streamv1, so we should point them to alternative in the place where people might find it.

rewrite: simplify the `retained_extras` extra logic First, we move the definition of value outside of the rebase extensions, as this apply to all rebase-like operation and some live in other place (like evolve). Second we make it a simple set, so that it is easy for an extension to add a new value in it. Third, we move the associated logic in core too. That make it easily available to other extensions. Fourth we simplify it usage, as the verbose version of the filtering is just a handful on line long, we are just going to test all the value for updates, so the Projection overlay is not bringing much here. Note that, we make it a module level set, is a key is worth preserving it is probably worth preserving in all cases. This was already the behavior prior to this change.

revlog: fix a bug in revlog splitting Specifically, if the .i.s files are not added to the transaction, then a follow-up modification of such a file records it into transaction as non-empty, which is incorrect.

stream-clone: smoothly detect and handle a case were a revlog is split This detect and handle the most common case for a race condition around stream and revlog splitting. The one were the revlog is split between the initial collection of data and the time were we start considering stream that data. In such case, we repatch an inlined version of that revlog together when this happens. This is necessary as stream-v2 promised a specific number of bytes and a specific number of files to the client. In stream-v3, we will have the opportunity to just send a split revlog instead. Getting a better version of the protocol for stream-v3 is still useful, but it is no longer a blocket to fix that race condition. Note that another, rarer race condition exist, were the revlog is split while we creating the revlog and extracing content from it. This can be dealt with later.

stream-clone: implement decidated `get_streams` method for revlog For revlog, we can do better using the maximum linkrev expected. This approach open the way to dealing with a much larger set of non-trivial changes, like splitting of inline revlogs. We will actually tackle this issue in the next changesets (thanks to this one).

stream-clone: make it the responsability of the store entry to stream content The store entry has more context, this will especially be true when it comes to revlogs. So we move the details of how to retrieve binary content to the StoreEntry. The stream clone code now focus on the protocol bits.

store: declare a `files` method on BaseStoreEntry This will help pytype to type check. We have to move `StoreFile` earlier in the file to use it in the type declaration.

revlog: add a `get_revlog` method This might seen weird, but I actually thing we have been needing this for a long time. There is multiple object that kind of pretend being revlogs while actually wrapping the actual revlog. Since multiple code needs to access the actuel revlog. See documentation for more details. Expect cleanup of various places one the current series is done.

stream-clone: drop the _emit_v2 function It has no user left.

stream-clone: directly use `_entries_walk` to generate stream-v2 This does not requires that much changes and will give us much more flexibility, like improving revlog handling to gracefully handle race situation.

stream-clone: pre-indent some code This make the next changeset clearer.