typing: fix directives mangled by black Differential Revision: https://phab.mercurial-scm.org/D10209

typing: disable module attribute warnings for properly conditionalized code Differential Revision: https://phab.mercurial-scm.org/D10208

typing: disable import error warnings that are already handled I'm assuming that the cffi ones are handled somewhere, but the others definitely are. Differential Revision: https://phab.mercurial-scm.org/D10207

typing: disable a few errors when accessing Windows specific attributes This is already guarded with conditionals. The entire win32 module is conditionally loaded. File "/mnt/c/Users/Matt/hg/mercurial/utils/procutil.py", line 162, in <module>: No attribute 'winstdout' on module 'mercurial.posix' [module-attr] File "/mnt/c/Users/Matt/hg/mercurial/utils/procutil.py", line 163, in <module>: No attribute 'winstdout' on module 'mercurial.posix' [module-attr] File "/mnt/c/Users/Matt/hg/mercurial/utils/procutil.py", line 174, in <module>: No attribute 'winstdout' on module 'mercurial.posix' [module-attr] File "/mnt/c/Users/Matt/hg/mercurial/utils/procutil.py", line 175, in <module>: No attribute 'winstdout' on module 'mercurial.posix' [module-attr] Differential Revision: https://phab.mercurial-scm.org/D10206

typing: disable a few errors calling py3.7+ functions in resourceutil.py There's a graceful fallback when these methods are unavailable. Differential Revision: https://phab.mercurial-scm.org/D10205

test-copies: add a test updating file content while merging a pure overwrite As the current test misbehavior show, this confuse various part of the code: - the committed file is different depending of the commit source, - the upgrade code compute the wrong information on this changeset. This will be fixed in upcoming patches. Differential Revision: https://phab.mercurial-scm.org/D10087

mergestate: don't pop stateextras when there are no conflicts on filemerge Even if `filemerge.filemerge()` didn't result in conflicts, we should not remove stateextras for a file since we now use that for more things than just merge time information. We use stateextras to store information which is required to be used by commit. I tracked this down while finding why a patch of mine which adds more commit related information does not work as expected and looses the extras in mergestate. Differential Revision: https://phab.mercurial-scm.org/D10148

node: introduce nodeconstants class In preparing for moving from SHA1 hashes to a modern hash function, place nullid and other constant magic vules in a class. Provide the active set of constants in the repository and push it down. Provide nullid directly in strategic places like the repository as it is accessed very often. This changeset introduces the API change, but not the mechanical replacement of the node.py attributes itself. Differential Revision: https://phab.mercurial-scm.org/D9750

revlog: guarantee that p1 != null if a non-null parent exists This change does not affect the hashing (which already did this transformation), but can change the log output in the rare case where this behavior was observed in repositories. The change can simplify iteration code where regular changesets and merges are distinct branches. Differential Revision: https://phab.mercurial-scm.org/D10150

push: allow to specify multiple destinations I end up needing that on a regular basis and it turn out to be very simple to implement. See documentation and test for details. Differential Revision: https://phab.mercurial-scm.org/D10161

push: indent the some part of the command That code will be put in a loop in the next changeset, pre-indenting make the next change clearer. Differential Revision: https://phab.mercurial-scm.org/D10160

bundle: optional multithreaded compression, ATM zstd-only Compression type can be a huge chunk of "hg bundle", especially when using the higher compression levels. With level=22 and threads=7, the NetBSD test repository took 28:39 wall time and 157:47 user time. Before, level=22 would take 129:20 wall time and 129:07 user time. Differential Revision: https://phab.mercurial-scm.org/D9283

rust: Preallocate the returned `Vec` in `utils::files::relativize_path` Profiling `rhg files > /dev/null` on an old snapshot of mozilla-central (with `perf` and the Firefox Profiler: https://github.com/firefox-devtools/profiler/blob/main/docs-user/guide-perf-profiling.md) showed non-trivial time spend in this function and in `realloc`. This change makes the wall-clock time for that process on my machine go from ~190 ms to ~150 ms. Differential Revision: https://phab.mercurial-scm.org/D10199

black: merge config into main pyproject.toml now that we have it This means that naive contributors who just run `black` on a source file will get reasonable behavior as long as they have a recent black. Yay! This was previously D9834 but was rolled back due to test failures. nbjoerg thinks it's time to try again, so let's give it a shot. Differential Revision: https://phab.mercurial-scm.org/D10185

pyproject: add config file This will tell pip et al to call our setup.py for the majority of packaging concerns, but also gives us a place to put standard config stuff like black. This was previously D9833, but was rolled back due to test breakage. nbjoerg thinks that breakage is now resolved, so we're trying again. Differential Revision: https://phab.mercurial-scm.org/D10184

perf-util: add an helper revset to use the same spec as the case search script This make result of the searcher script easier to use. Differential Revision: https://phab.mercurial-scm.org/D10228

perf-helpers: add a search-discovery-case script This a small script I built to look for interesting discovery case. It is fairly basic but could be useful in various situation so lets put it in the main repositories. Differential Revision: https://phab.mercurial-scm.org/D10225

debugdiscovery: add missing byte string marker to some help text This is causing crash when using --help. Differential Revision: https://phab.mercurial-scm.org/D10227

debugdiscovery: also integrate the discovery output in the json one We add a way for formatter to informs code that free output is unwanted, and we incorporate it in the json output. Differential Revision: https://phab.mercurial-scm.org/D10224

debugdiscovery: add support for Json output They are some pre-output that will be impractical, I'll take care of it in the next changeset. Differential Revision: https://phab.mercurial-scm.org/D10223

perf-helper: add a new sampling revset based on anti-chain See inline documentation for details. Differential Revision: https://phab.mercurial-scm.org/D10222

perf-helper: add a small extension with revsets to select repository subset Playing with discovery requires building interesting case. To do this we need revsets to try to generate them. We start with a quite simple one. See documentation for details. Differential Revision: https://phab.mercurial-scm.org/D10221

amend: mark commit obsolete after moving working copy We were doing it this way: 1. move working copy (repo.setparents) 2. add obsmarkers (scmutil.cleanupnodes) 3. fix dirstate (dirstate.normal/drop) Step 1 and 3 are closely related, so let's move them together. It seems safest to create the obsmarkers last. This patch thus makes the order 1, 3, 2. Differential Revision: https://phab.mercurial-scm.org/D10197

pull: allow to specify multiple sources I end up needing that on a regular basis and it turn out to be very simple to implement. See documentation and test for details. Differential Revision: https://phab.mercurial-scm.org/D10159

pull: pre-indent a part of the function We are about to run it in a loop, so lets pre-indent it to clarify the actual change in the next changesets. Differential Revision: https://phab.mercurial-scm.org/D10158

command: clarify `postincoming` return and that return handling The command should return None or a return code. The previous code was returning boolean directly relying on the fact that `True → 1` and `False → 0`. This is a good road to troubles, so lets be explicit about that return. Differential Revision: https://phab.mercurial-scm.org/D10157

hg: make `clean` return consistent with the `update` function The function return a boolean and is used as such. Lets be explicit about it. Differential Revision: https://phab.mercurial-scm.org/D10156

tests: rename `test-pull` to `test-pull-network.t` If is mostly about ssh and http interaction so lets avoid confusion with a generic `hg pull` test Differential Revision: https://phab.mercurial-scm.org/D10155

tests: move a test about update in test-pull-update.t Differential Revision: https://phab.mercurial-scm.org/D10154

split: close transaction in the unlikely event of a conflict while rebasing `hg split` *should* never result in conflicts, but in case there are bugs, we should at least commit the transaction so they can continue the rebase. One of our users ran into the regression fixed by D10120. They fixed the conflict and the tried to continue the rebase, but it failed with "abort: cannot continue inconsistent rebase" because the rebase state referred to commits written in a transaction that was never committed. Side note: `hg split` should probably turn off copy tracing to reduce the impact of such bugs, and to speed it up as well. Copies made in the rebased commits should still be respected because `hg rebase` calls `copies.graftcopies()`. Differential Revision: https://phab.mercurial-scm.org/D10164

rust-status: fix issue6456 for the Rust implementation also This implementation is being used by a few people now, so we need it fixed, even though a big rewrite will be coming quite soon. Differential Revision: https://phab.mercurial-scm.org/D10217

README: document requirement for builtin zstd Differential Revision: https://phab.mercurial-scm.org/D10226

ci: Add a job testing with rhg installed as `hg` This significantly increases test coverage of rhg, without duplicating many tests that already exist. The `cargo build` command being remove only compiled a `rhg` executable (as shown by the preceding `cd` command) but since a previous patch `run-tests.py --rhg` now takes care of that. Differential Revision: https://phab.mercurial-scm.org/D10196

run-test: install rhg if --rhg is passed Before this, --rhg was only working with --local. Differential Revision: https://phab.mercurial-scm.org/D10195

makefile: add a install option This gives and easy way to install rhg that we can use in `run-test.py` in the next changesets. Differential Revision: https://phab.mercurial-scm.org/D10194

makefile: add a build-rhg option This gives an easy action to build the rhg-binary. This will be useful for the `install-rhg` action in the next changeset. Differential Revision: https://phab.mercurial-scm.org/D10193

makefile: add a install-chg option This is done as a gratuitous improvement on the way to add makefile entry to build and install rhg. It seems saner to have equivalent entry for chg too. Differential Revision: https://phab.mercurial-scm.org/D10192

makefile: add a build-chg option This is done as a gratuitous improvement on the way to add makefile entry to build and install rhg. It seems saner to have equivalent entry for chg too. Differential Revision: https://phab.mercurial-scm.org/D10191

tests: Disable for rhg remaining tests that fail in that mode These cases are in features not yet implemented by rhg for which triggering a fallback is not practical. Disabling some tests allows us to reach passing CI and catch any future regression in the rest of the tests. Differential Revision: https://phab.mercurial-scm.org/D10190

rhg: Remove `rhg.fallback-executable=hg` default configuration When `rhg.on-unsupported` is configured to `fallback` and an unsupported feature is encountered, the previous default was to look for an `hg` executable in `$PATH`. This default was fragile since it was easy to end up accidentally using an older version of Mercurial installed system-wide, when a local (perhaps patched) installation was intended. Instead, it is now an error to have `rhg.on-unsupported=fallback` without also configuring an explicit path or the fallback executable. Differential Revision: https://phab.mercurial-scm.org/D10189

rhg: Return an error code for `rhg config Section.idontexist` This is what Python-based hg does. Differential Revision: https://phab.mercurial-scm.org/D10145

rhg: Fall back to Python for --version Clap has some built-in support for printing something on --version, but it looks different than what Pytho-based hg does. Also, at the moment we’re not giving version numbers to rhg separate from the Mercurial version. Differential Revision: https://phab.mercurial-scm.org/D10144

rhg: Exit with an error code if `files` finds nothing This matches the behavior of Python-based hg. Differential Revision: https://phab.mercurial-scm.org/D10143

rhg: `cat` command: print error messages for missing files And exit with an error code if no file was matched. This matches the behavior of Python-based hg. Differential Revision: https://phab.mercurial-scm.org/D10142

rhg: Silently ignore missing files in config %include … instead of aborting with an error message. This is what Python-based hg does. Differential Revision: https://phab.mercurial-scm.org/D10141

rhg: Add support for environment variables in config include paths Some tests rely on this. Differential Revision: https://phab.mercurial-scm.org/D10140

rhg: Add support for the HGRCSKIPREPO environment variable It’s easy enough and affects tests, well, that test that feature. Differential Revision: https://phab.mercurial-scm.org/D10139

rhg: Don’t make repository path absolute too early Some error messages want to include a relative path, which affects the output of some tests. Differential Revision: https://phab.mercurial-scm.org/D10138

rhg: Fall back to Python if ui.relative-paths is configured This feature is not supported yet, and affects the output of some tests. Differential Revision: https://phab.mercurial-scm.org/D10137

tests: clarify some missing output in test-merge-subrepos This makes the test behavior clearer, especially, why the output are inconsistent. Differential Revision: https://phab.mercurial-scm.org/D10136

tests: Adapt expected output for minor differences with rhg Differential Revision: https://phab.mercurial-scm.org/D10135

rhg: Add support for --cwd This affect the meaning of relative paths in `--repository`, which are resolved "early" by rhg in order to load config which is needed before fallback to Python is considered. An incorrect path could cause errors when loading a non-existent repo, leading to failing tests even when fallback is enabled. Differential Revision: https://phab.mercurial-scm.org/D10134

rhg: Align with Python on some more error messages Differences can cause some tests to fail Differential Revision: https://phab.mercurial-scm.org/D10133

rhg: Add an allow-list of ignored extensions Because rhg doesn’t know how a Python extension would affect behavior it implements in Rust, when an unsupported extension is enabled it conservatively falls back to Python-based hg. However many users will have unsupported extensions enabled in practice. Maybe they don’t actually affect rhg behavior, but we don’t know. This adds a `rhg.ignored-extensions` configuration that lets users list extensions that rhg can safely ignore and proceed even if they’re not supported in Rust. Differential Revision: https://phab.mercurial-scm.org/D10188

rhg: Fall back to Python if unsupported extensions are enabled Extensions might affect behavior in ways we can’t anticipate, so just ignoring them is not correct. Later we’ll add opt-in configuration to ignore specific extensions. Differential Revision: https://phab.mercurial-scm.org/D10112

rhg: Sort config files when adding a directory For example in `/etc/mercurial/hgrc.d/` or with `HGRCPATH=some-directory`. Previously files where parsed in the order returned by the filesystem, which is undefined. But order is significant when multiple files define the same configuration key: the "last" one wins. Differential Revision: https://phab.mercurial-scm.org/D10111

rhg: Align config file parse error formatting with Python Differences can cause tests to fail Differential Revision: https://phab.mercurial-scm.org/D10110

rhg: Fall back to Python for bundle repositories rhg does not support bundles at all, yet. Differential Revision: https://phab.mercurial-scm.org/D10102

rhg: Fall back to Python on --repository with an URL A low-hanging fruit to improve on this would be to properly parse and handle `file:` URLs. But other Python-based hg supports some other URL schemes for features that rhg does not support yet. Differential Revision: https://phab.mercurial-scm.org/D10101

rhg: Print non-absolutized path in "repository {} not found" errors … like Python does. Differences in ouput can cause tests to fail. Differential Revision: https://phab.mercurial-scm.org/D10100

rhg: Align "malformed --config" error message with Python Differences in error message formatting can cause tests to fail. Differential Revision: https://phab.mercurial-scm.org/D10099

rhg: Fall back to Python on unsupported `rhg config <section>` Printing an entire section (as opposed to a single value with `rhg config foo.bar`) is not supported yet in Rust only. Differential Revision: https://phab.mercurial-scm.org/D10098

rhg: Fall back to Python for unsupported revset syntax rhg only supports a small subset of the syntax. On parse error, this gives Python a chance instead of aborting immediately. Differential Revision: https://phab.mercurial-scm.org/D10097

tests: Add `rhg` and `no-rhg` for #require and #if in .t files With this we can allow some small differences in expected output between Rust-based and Python-based code paths. Differential Revision: https://phab.mercurial-scm.org/D10096

tests: Enable rhg fallback to Python by default in tests This premise of `run-tests.py --rhg`: fallback should make `rhg` behave the same as `hg`, except faster in some cases. To test run the whole test suite with installed `rhg` as `hg` and with fallback enabled. Differential Revision: https://phab.mercurial-scm.org/D10187

rhg: Add environment variables for fallback configuration For the `rust-tests.py --rhg` we want every `hg` command in tests to run `rhg` with fallback enabled, but other methods of setting configuration are limited or disruptive as discussed in code comment. Differential Revision: https://phab.mercurial-scm.org/D10186

paths: add a `*` special path to define default sub option Differential Revision: https://phab.mercurial-scm.org/D10163

ui: pass a `ui` object to `paths.getpath` I want to introduce more path's suboption and make it possible to use default value for them. Processing theses sub-options might result in warnings. We need a `ui` object to issue such warnings. To make things simpler, we add an helper method on the `ui` object. Differential Revision: https://phab.mercurial-scm.org/D10162

configitems: add TODOs blocking the move out of experimental for revlogv2 These are the todos so far, we probably will have more, but we might as well list them while they're fresh in our minds. Differential Revision: https://phab.mercurial-scm.org/D10216

sidedata-exchange: rewrite sidedata on-the-fly whenever possible When a A exchanges with B, the difference of their supported sidedata categories is made, and the responsibility is always with the client to generated it: - If A pushes to B and B requires category `foo` that A does not have, A will need to generate it when sending it to B. - If A pulls from B and A needs category `foo`, it will generate `foo` before the end of the transaction. - Any category that is not required is removed. If peers are not compatible, abort. It is forbidden to rewrite sidedata for a rev that already has sidedata, since that would introduce unreachable (garbage) data in the data file, something we're not prepared for yet. Differential Revision: https://phab.mercurial-scm.org/D10032

revlog-index: add `replace_sidedata_info` method During a `pull` operation where the server does not provide sidedata, the client that requires it should generate them on-the-fly. In the generic case, we need to wait for the changelog + manifests + filelogs to be added, since we don't know what the sidedata computers might need: this means rewriting the sidedata of index entries from within the pull transaction (and no further back) right after we've added them. Both Python and C implementations only allow for rewriting the sidedata offset and length for revs within the transaction where they were created. Differential Revision: https://phab.mercurial-scm.org/D10031

revlogv2: temporarily forbid inline revlogs See inline comments. I plan to fix the underlying issue before revlogv2 is stabilized. Differential Revision: https://phab.mercurial-scm.org/D10030

changegroupv4: add sidedata helpers These helpers carry the information and computers needed to rewrite sidedata when generating/applying patches. We will be making use of them soon. Differential Revision: https://phab.mercurial-scm.org/D10029

revlog: add attribute on revlogs that specifies its kind The sidedata logic needs to check whether the revlog it's working on is a changelog, a manifest or a filelog. Furthermore, future versions of the revlog format will most likely see a split between the three types (i.e. they will store different information), so having this will be useful for other future endeavors as well. Differential Revision: https://phab.mercurial-scm.org/D10151

sidedata-exchange: add `wanted_sidedata` and `sidedata_computers` to repos Each repo will advertise the sidedata categories it requires (categories being unique and canonical), and have a set of "computers", functions to generate sidedata from `(repo, revlog, rev, previous_sidedata)`, for a given category. The set of computers can be a superset of the set of the wanted categories, but not smaller: repos are expected to be coherent in their handling of sidedata. Differential Revision: https://phab.mercurial-scm.org/D10028

delta: add sidedata field to revision delta When emitting revision delta, we need to also emit the sidedata information just added in the revlogv2 format if appropriate. Differential Revision: https://phab.mercurial-scm.org/D10027

changegroup: add v4 changegroup for revlog v2 exchange This change only adds the required infrastructure for the new changegroup format and does not do any actual exchange. This will be done in the next patches. Differential Revision: https://phab.mercurial-scm.org/D10026

revlogv2: don't assume that the sidedata of the last rev is right after data We are going to be rewriting sidedata soon, it's going to be appended to the revlog data file, meaning that the data and the sidedata might not be contiguous. Differential Revision: https://phab.mercurial-scm.org/D10025

sidedata: move to new sidedata storage in revlogv2 The current (experimental) sidedata system uses flagprocessors to signify the presence and store/retrieve sidedata from the raw revlog data. This proved to be quite fragile from an exchange perspective and a lot more complex than simply having a dedicated space in the new revlog format. This change does not handle exchange (ironically), so the test for amend - that uses a bundle - is broken. This functionality is split into the next patches. Differential Revision: https://phab.mercurial-scm.org/D9993

cext: add support for revlogv2 This enables the C code to retrieve/create entries in the revlog v2 format. This is mainly a matter of taking into account the additional slots for sidedata, etc. Differential Revision: https://phab.mercurial-scm.org/D9846

bitmanipulation: add utils to read/write bigendian 64bit integers Differential Revision: https://phab.mercurial-scm.org/D9845

format: remove sidedata format variant This format variant can only exist when also using revlog v2, so it's useless now. Differential Revision: https://phab.mercurial-scm.org/D10113

revlogv2: allow upgrading to v2 Revlogv2 implies sidedata. Right now sidedata is not really used in production, and Revlogv2 will be used for the first production-ready version of sidedata support. Differential Revision: https://phab.mercurial-scm.org/D9844

revlog: introduce v2 format As documented in [1], this is still tentative and could be subject to change, but we need to lay down the foundations in order to work on the next abstraction layers. [1] https://www.mercurial-scm.org/wiki/RevlogV2Plan Differential Revision: https://phab.mercurial-scm.org/D9843

formatting: fix redundant parentheses These were introduced by 0d055849d5f9d682ef931d2566b760d5c6bf7e52. Differential Revision: https://phab.mercurial-scm.org/D10229

wsgicgi: switch the default PATH_INFO back to str It looks like everything here is meant to be str, and regressed in 687b865b95ad. Pytype didn't flag this, but is really confused about the range of possible types for the dictionary values, and flagged missing `lower()` on various types in this area. Differential Revision: https://phab.mercurial-scm.org/D10204

crecord: convert an instance of bytes to str Caught by pytype: File "/mnt/c/Users/Matt/hg/mercurial/crecord.py", line 1154, in printstring: Function _CursesWindow.addstr was called with the wrong arguments [wrong-arg-types] Expected: (self, str: str, ...) Actually passed: (self, str: bytes, ...) Differential Revision: https://phab.mercurial-scm.org/D10203

crecord: change str literals to byte literals Differential Revision: https://phab.mercurial-scm.org/D10202

bisect: replace a missing method on the list primitive Caught by pytype: File "/mnt/c/Users/Matt/hg/mercurial/commands.py", line 1095, in bisect: No attribute 'last' on list [attribute-error] In list Differential Revision: https://phab.mercurial-scm.org/D10201

typing: disable a module-attr warning in the worker module's py2 code Differential Revision: https://phab.mercurial-scm.org/D10183

wireprotoserver: convert ErrorResponse to bytes Caught by pytype: File "/mnt/c/Users/Matt/hg/mercurial/wireprotoserver.py", line 236, in handlewsgirequest: Function _bytestr.__init__ was called with the wrong arguments [wrong-arg-types] Expected: (self, ints: Iterable[int]) Actually passed: (self, ints: mercurial.hgweb.common.ErrorResponse) The following methods aren't implemented on mercurial.hgweb.common.ErrorResponse: __iter__ File "/mnt/c/Users/Matt/hg/mercurial/wireprotoserver.py", line 239, in handlewsgirequest: Function _bytestr.__init__ was called with the wrong arguments [wrong-arg-types] Expected: (self, ints: Iterable[int]) Actually passed: (self, ints: mercurial.hgweb.common.ErrorResponse) The following methods aren't implemented on mercurial.hgweb.common.ErrorResponse: __iter__ File "/mnt/c/Users/Matt/hg/mercurial/wireprotov2server.py", line 91, in handlehttpv2request: Function _bytestr.__init__ was called with the wrong arguments [wrong-arg-types] Expected: (self, ints: Iterable[int]) Actually passed: (self, ints: mercurial.hgweb.common.ErrorResponse) The following methods aren't implemented on mercurial.hgweb.common.ErrorResponse: __iter__ Differential Revision: https://phab.mercurial-scm.org/D10182

verify: convert an exception to bytes before logging I'm not entirely sure why this code appears to be trying to convert twice, but it was flagged by pytype: File "/mnt/c/Users/Matt/hg/mercurial/verify.py", line 84, in _exc: Function _bytestr.__init__ was called with the wrong arguments [wrong-arg-types] Expected: (self, ints: Iterable[int]) Actually passed: (self, ints: Exception) The following methods aren't implemented on Exception: __iter__ Differential Revision: https://phab.mercurial-scm.org/D10181

typing: add an assertion to the upgrade engine to help pytype Differential Revision: https://phab.mercurial-scm.org/D10180

subrepo: handle unexpected file types from git gracefully This was flagged by pytype because `tar.extractfile(...)` can return None if the entry is not a file or symlink. I don't think that git supports other types, but better safe than sorry. Differential Revision: https://phab.mercurial-scm.org/D10179

patch: convert a UI message to bytes when editing a patch Differential Revision: https://phab.mercurial-scm.org/D10178

merge: force an exception message to bytes before printing as a warning Caught by pytype: File "/mnt/c/Users/Matt/hg/mercurial/merge.py", line 1346, in batchremove: Function _bytestr.__init__ was called with the wrong arguments [wrong-arg-types] Expected: (self, ints: Iterable[int]) Actually passed: (self, ints: str) Differential Revision: https://phab.mercurial-scm.org/D10177