paths: add a `*` special path to define default sub option Differential Revision: https://phab.mercurial-scm.org/D10163

ui: pass a `ui` object to `paths.getpath` I want to introduce more path's suboption and make it possible to use default value for them. Processing theses sub-options might result in warnings. We need a `ui` object to issue such warnings. To make things simpler, we add an helper method on the `ui` object. Differential Revision: https://phab.mercurial-scm.org/D10162

configitems: add TODOs blocking the move out of experimental for revlogv2 These are the todos so far, we probably will have more, but we might as well list them while they're fresh in our minds. Differential Revision: https://phab.mercurial-scm.org/D10216

sidedata-exchange: rewrite sidedata on-the-fly whenever possible When a A exchanges with B, the difference of their supported sidedata categories is made, and the responsibility is always with the client to generated it: - If A pushes to B and B requires category `foo` that A does not have, A will need to generate it when sending it to B. - If A pulls from B and A needs category `foo`, it will generate `foo` before the end of the transaction. - Any category that is not required is removed. If peers are not compatible, abort. It is forbidden to rewrite sidedata for a rev that already has sidedata, since that would introduce unreachable (garbage) data in the data file, something we're not prepared for yet. Differential Revision: https://phab.mercurial-scm.org/D10032

revlog-index: add `replace_sidedata_info` method During a `pull` operation where the server does not provide sidedata, the client that requires it should generate them on-the-fly. In the generic case, we need to wait for the changelog + manifests + filelogs to be added, since we don't know what the sidedata computers might need: this means rewriting the sidedata of index entries from within the pull transaction (and no further back) right after we've added them. Both Python and C implementations only allow for rewriting the sidedata offset and length for revs within the transaction where they were created. Differential Revision: https://phab.mercurial-scm.org/D10031

revlogv2: temporarily forbid inline revlogs See inline comments. I plan to fix the underlying issue before revlogv2 is stabilized. Differential Revision: https://phab.mercurial-scm.org/D10030

changegroupv4: add sidedata helpers These helpers carry the information and computers needed to rewrite sidedata when generating/applying patches. We will be making use of them soon. Differential Revision: https://phab.mercurial-scm.org/D10029

revlog: add attribute on revlogs that specifies its kind The sidedata logic needs to check whether the revlog it's working on is a changelog, a manifest or a filelog. Furthermore, future versions of the revlog format will most likely see a split between the three types (i.e. they will store different information), so having this will be useful for other future endeavors as well. Differential Revision: https://phab.mercurial-scm.org/D10151

sidedata-exchange: add `wanted_sidedata` and `sidedata_computers` to repos Each repo will advertise the sidedata categories it requires (categories being unique and canonical), and have a set of "computers", functions to generate sidedata from `(repo, revlog, rev, previous_sidedata)`, for a given category. The set of computers can be a superset of the set of the wanted categories, but not smaller: repos are expected to be coherent in their handling of sidedata. Differential Revision: https://phab.mercurial-scm.org/D10028

delta: add sidedata field to revision delta When emitting revision delta, we need to also emit the sidedata information just added in the revlogv2 format if appropriate. Differential Revision: https://phab.mercurial-scm.org/D10027

changegroup: add v4 changegroup for revlog v2 exchange This change only adds the required infrastructure for the new changegroup format and does not do any actual exchange. This will be done in the next patches. Differential Revision: https://phab.mercurial-scm.org/D10026

revlogv2: don't assume that the sidedata of the last rev is right after data We are going to be rewriting sidedata soon, it's going to be appended to the revlog data file, meaning that the data and the sidedata might not be contiguous. Differential Revision: https://phab.mercurial-scm.org/D10025

sidedata: move to new sidedata storage in revlogv2 The current (experimental) sidedata system uses flagprocessors to signify the presence and store/retrieve sidedata from the raw revlog data. This proved to be quite fragile from an exchange perspective and a lot more complex than simply having a dedicated space in the new revlog format. This change does not handle exchange (ironically), so the test for amend - that uses a bundle - is broken. This functionality is split into the next patches. Differential Revision: https://phab.mercurial-scm.org/D9993

cext: add support for revlogv2 This enables the C code to retrieve/create entries in the revlog v2 format. This is mainly a matter of taking into account the additional slots for sidedata, etc. Differential Revision: https://phab.mercurial-scm.org/D9846

bitmanipulation: add utils to read/write bigendian 64bit integers Differential Revision: https://phab.mercurial-scm.org/D9845

format: remove sidedata format variant This format variant can only exist when also using revlog v2, so it's useless now. Differential Revision: https://phab.mercurial-scm.org/D10113

revlogv2: allow upgrading to v2 Revlogv2 implies sidedata. Right now sidedata is not really used in production, and Revlogv2 will be used for the first production-ready version of sidedata support. Differential Revision: https://phab.mercurial-scm.org/D9844

revlog: introduce v2 format As documented in [1], this is still tentative and could be subject to change, but we need to lay down the foundations in order to work on the next abstraction layers. [1] https://www.mercurial-scm.org/wiki/RevlogV2Plan Differential Revision: https://phab.mercurial-scm.org/D9843

rebase: inline simple function for finding obsolete subset of commits `_filterobsoleterevs()` is just one line long. It was introduced in 2d294dada4f8 (rebase: small refactoring to allow better extensibility from extensions, 2016-01-14), for use by the "inhibit" extension. That extension was removed from the evolve repo in 87e87881059d (compat: drop the inhibit hacky extension, 2017-10-24). Differential Revision: https://phab.mercurial-scm.org/D10198

ci: hook network-io tests into the pipeline This runs the "pip install" tests once for Python 2 and 3 each. Differential Revision: https://phab.mercurial-scm.org/D10075

merge: with stable

test: update expected output in test-http.t The output was introduced in a4c19a162615 and is wrong and unstable. So we glob it as other part of these tests already do. Differential Revision: https://phab.mercurial-scm.org/D10153

remotefilelog: remove unused import This is no longer used since a4c19a162615. Differential Revision: https://phab.mercurial-scm.org/D10152

wireprotov1peer: don't raise internal errors in some cases Specifically, when the peer is closed in the middle of a batch of rpcs. Differential Revision: https://phab.mercurial-scm.org/D10074

sshpeer: don't fail forwarding output from closed connections The test still shows an internal error, but one that happens further along. Differential Revision: https://phab.mercurial-scm.org/D10073

test: show internal exception with batchable rpcs over ssh Differential Revision: https://phab.mercurial-scm.org/D10072

rhg: Ignore trailing newlines in .hg/sharedpath Differential Revision: https://phab.mercurial-scm.org/D10132

tests: Add `--rhg` and `--with-rhg=<path>` options for `run-tests.py` They are mostly equivalent to the corresponding `chg` options. For now, many tests are still failing in this configuration. It is *not* run on CI. Differential Revision: https://phab.mercurial-scm.org/D10095

rhg: Add support for automatic fallback to Python `rhg` is a command-line application that can do a small subset of what `hg` can. It is written entirely in Rust, which avoids the cost of starting a Python interpreter and importing many Python modules. In a script that runs many `hg` commands, this cost can add up. However making users decide when to use `rhg` instead of `hg` is not practical as we want the subset of supported functionality to grow over time. Instead we introduce "fallback" behavior where, when `rhg` encounters something (a sub-command, a repository format, …) that is not implemented in Rust-only, it does nothing but silently start a subprocess of Python-based `hg` running the same command. That way `rhg` becomes a drop-in replacement for `hg` that sometimes goes faster. Whether Python is used should be an implementation detail not apparent to users (other than through speed). A new `fallback` value is added to the previously introduced `rhg.on-unsupported` configuration key. When in this mode, the new `rhg.fallback-executable` config is determine what command to use to run a Python-based `hg`. The previous `rhg.on-unsupported = abort-silent` configuration was designed to let a wrapper script call `rhg` and then fall back to `hg` based on the exit code. This is still available, but having fallback behavior built-in in rhg might be easier for users instead of leaving that script "as an exercise for the reader". Using a subprocess like this is not idea, especially when `rhg` is to be installed in `$PATH` as `hg`, since the other `hg.py` executable needs to still be available… somewhere. Eventually this could be replaced by using PyOxidizer to a have a single executable that embeds a Python interpreter, but only starts it when needed. Differential Revision: https://phab.mercurial-scm.org/D10093

rhg: Add a `rhg.on-unsupported` configuration key For now the two values are: * `abort-silent`: silently exit with code 252, the previous default behavior * `abort`: print an error message about what feature is not supported, then exit with code 252. Now the default. Differential Revision: https://phab.mercurial-scm.org/D10091

rhg: Make configuration available as early as possible in main() Differential Revision: https://phab.mercurial-scm.org/D10090

rust: Upgrade Cargo.lock to the newer format As discussed in https://phab.mercurial-scm.org/D10085#153099 See https://github.com/rust-lang/cargo/pull/7070 and https://blog.rust-lang.org/2020/01/30/Rust-1.41.0.html#less-conflict-prone-cargolock-format Differential Revision: https://phab.mercurial-scm.org/D10089

sshpeer: enable+fix warning about sshpeers not being closed explicitly I recommend looking at this with a diff that ignores indentation. The test changes are because localrepo.close() updates some cache, which appears happens earlier now on rollbacks or strips or something. The http changes are because httppeer.close() prints stats with --verbose. Differential Revision: https://phab.mercurial-scm.org/D9999

sshpeer: add a develwarning if an sshpeer is not closed explicitly The warning is disabled until the next commit, because fixing it results in a noisy diff due to indentation changes. Differential Revision: https://phab.mercurial-scm.org/D9998

remotefilelog: rework workaround for sshpeer deadlocks The wrapping of `sshpeer.cleanup` silently broke when `cleanup` was renamed to `_cleanup`, a couple of years ago. I don't know what `orig.im_self` is, but regardless, the intention of the wrapping seems pretty clear: close stderr before sshpeer._cleanuppipes blocks on it. So do that. Differential Revision: https://phab.mercurial-scm.org/D9997

sshpeer: add a method to check if a doublepipe is closed So we can tell in a next commit if we're trying to close an already closed connection or not (in which case, we may warn). Differential Revision: https://phab.mercurial-scm.org/D9996

sshpeer: make sshpeer.close() close the underlying connection So the connection can be closed eagerly in future commits, instead of relying on __del__. Differential Revision: https://phab.mercurial-scm.org/D9995

copies-rust: add a macro-based unit-testing framework `compare_values`, `merge_copies_dict`, and `CombineChangesetCopies` are APIs whose signatures involve non-trivial types. Calling them directly in unit tests would involve a lot of verbose setup code that obscures the meaningful parts of a given test case. This adds a macro-based test-harness with pseudo-syntax to tersely create arguments and expected return values in the correct types. For now there is only one (not particularly meaningful) test case per tested function, just to exercize the macros. Differential Revision: https://phab.mercurial-scm.org/D10071

copies-rust: rewrite ChangedFiles binary parsing by using the new from-bytes-safe crate and a custom struct that encodes the expected data structure. Differential Revision: https://phab.mercurial-scm.org/D10068

tags: validate nodes in _getfnodes() and update cache in case of unknown nodes `hgtagsfnodescache` can contain unknown nodes due to cache corruption and this lead to a traceback on operations like `hg tags` as we don't validate nodes. This patch validates that all filenodes returned after `hgtagsfnodescache` are known to the repository. If there exists any unknown filenode, we force recompute it and update the cache. The test change demonstrates the fix. Differential Revision: https://phab.mercurial-scm.org/D10083

debugtagscache: verify that filenode is correct Previous patch from Matt demonstrates that `debugtagscache` does not warn about filenode being unknown which can be caused by a corrupted cache. We start by showing that it's an unknown node. Differential Revision: https://phab.mercurial-scm.org/D10015

tests: demonstrate a case where a corrupt tag cache causes an abort I happened to hit this trying to cover other cases around valid vs missing entries. I have no idea if this is something that could occur more naturally (similar to how a missing file node in `hgtagsfnodes1` can occur after a strip). There is a test just above this added in f5a7cf0adb12 mentioning it "overwrites the junk", though that tests truncation instead of actual garbage. But since this is just a cache, it probably shouldn't abort with a cryptic message like this. The two options I see both have downsides- either rebuild the cache (and potentially take a long time), or hint to the user to run a debug command. Differential Revision: https://phab.mercurial-scm.org/D9812

debugcommands: prevent using `is False` I was touching this code in a future patch and marmoute warned about usage of `is False` here. Quoting marmoute: ``` "is False" is going to check if the object you have the very same object in memory than the one Python allocated for False (in practice 0) This will "mostly work" on cpython because of implementation details, but is semantically wrong and can start breaking unexpectedly ``` Differential Revision: https://phab.mercurial-scm.org/D10014

hgtagsfnodes: refactor code to compute fnode into separate fn I plan to use this code at one more place while fixing a bug caused by an invalid fnode present in cache. Differential Revision: https://phab.mercurial-scm.org/D10013

error: remove shortening of node in error message This strips the complete 20 bytes node which was not found. Having the the full node in error message is important as it makes debugging easier. If a short node is to be displayed, that should be done by callers. Differential Revision: https://phab.mercurial-scm.org/D9994

copies: filter out copies grafted from another branch Consider this simple history: ``` @ 3 modify y | o 2 copy x to y, modify x | | o 1 copy x to y, modify x |/ o 0 add x ``` If we now rebase commit 3 onto 1, Mercurial will look for copies between commit 2 and commit 1. It does that by going backwards from 2 to 0 and then forwards from 0 to 1. It will find that x was copied to y, since that was what happened on the path between them (namely in commit 1). That leads Mercurial to do a 3-way merge between y@3 and y@1 with x@2 as base. We want to use y@2 as base instead. That's also what happened until commit 1d6d1a15. This patch fixes the regression by adding another filtering step when chaining copies via a diffbase. The new filtering step removes copies that were the same between the two branches (same source and destination, but not necessarily the same contents). Differential Revision: https://phab.mercurial-scm.org/D10120

copies: inline _backwardrenames() in pathcopies() I'll add another filtering step in `patchcopies()` next. I need access to the forward copies for that. Differential Revision: https://phab.mercurial-scm.org/D10119

copies: extract function _backwardcopies() for reversing renames I'll add another callers in the next patch. Differential Revision: https://phab.mercurial-scm.org/D10118

tests: demonstrate how grafted copies are counted when tracing across branches This test demonstrates a regression from 1d6d1a15. Differential Revision: https://phab.mercurial-scm.org/D10117

typing: add some type annotations to mercurial/pathutil.py Differential Revision: https://phab.mercurial-scm.org/D10128

typing: add some type annotations to mercurial/util.py Differential Revision: https://phab.mercurial-scm.org/D10127

typing: add some type annotations to mercurial/phases.py Some of these were helpful in typing other modules, and then I typed the easy-ish ones. Black forces the long `Phasedefaults` definition to be wrapped, which pytype seems OK with (as shown with `reveal_type()`), but it does seem to confuse PyCharm a bit. Differential Revision: https://phab.mercurial-scm.org/D10126

typing: add type annotations to the public methods of mercurial/subrepoutil.py Differential Revision: https://phab.mercurial-scm.org/D10125

typing: add type annotations to mercurial/i18n.py I'm a little unsure of this because `gettext()` clearly allows for passing unicode. But the comments seem to indicate that this is related to tests, and this was useful for catching unicode being passed to `_()` in the keyring extension. I'm also not sure why `_(None)` would make any sense, so maybe the argument shouldn't be optional? I didn't add it to the lambda in plain mode because that spilled beyond 80 characters and so black mangled it. Black and pytype disagree on where the comment to disable a check needs to go, so this has to disable and then enable the checking. Differential Revision: https://phab.mercurial-scm.org/D10124

typing: add type annotations to mercurial/utils/dateutil.py For now, I'm just typing around the edges to help find issues with TortoiseHg. If the custom `hgdate` type is useful elsewhere as I go, I'll move it to a file dedicated to custom types. I'm not loving the ban on camelcase type names here that test-check-code.t flagged, but I'm not sure how to disable that even if everyone agreed that it's a bad idea to go against the normal convention for types. While here, fix an issue that pytype found in `parsedate` when an invalid date tuple is passed by raising a ProgrammingError instead of crashing. (Tuple doesn't have a `strip` attribute.) Differential Revision: https://phab.mercurial-scm.org/D10123

shelve: fix conversion of exceptions to strings flagged by pytype I've seen this done several ways and don't know what's correct. But pytype was unhappy about the previous way: FAILED: /mnt/c/Users/Matt/hg/tests/.pytype/pyi/mercurial/shelve.pyi /usr/bin/python3.6 -m pytype.single --imports_info /mnt/c/Users/Matt/hg/tests/.pytype/imports/mercurial.shelve.imports --module-name mercurial.shelve -V 3.6 -o /mnt/c/Users/Matt/hg/tests/.pytype/pyi/mercurial/shelve.pyi --analyze-annotated --nofail --quick /mnt/c/Users/Matt/hg/mercurial/shelve.py File "/mnt/c/Users/Matt/hg/mercurial/shelve.py", line 244, in _verifyandtransform: Function bytestr.__init__ was called with the wrong arguments [wrong-arg-types] Expected: (self, ints: Iterable[int]) Actually passed: (self, ints: Union[KeyError, TypeError, ValueError]) File "/mnt/c/Users/Matt/hg/mercurial/shelve.py", line 253, in _getversion: Function bytestr.__init__ was called with the wrong arguments [wrong-arg-types] Expected: (self, ints: Iterable[int]) Actually passed: (self, ints: ValueError) The following methods aren't implemented on ValueError: __iter__ Differential Revision: https://phab.mercurial-scm.org/D10122

releasenotes: use the right API to access the 'sections' Preventing direct access to the underlying dict fix a breakage introduced by the refactoring in d3df397e7a59. This changeset is similar to 271dfcb98544, 5272542196cc and f7621fa14b84. The breackage of `releasenotes.py` stayed under my radar as the CI did not have fuzzywuzzy installed. (Something that is about to be fixed). Differential Revision: https://phab.mercurial-scm.org/D10121

requirements: also add a fncache constant Continue the cleanup to the remaining requirements Differential Revision: https://phab.mercurial-scm.org/D10109

requirements: also add a store constant Continue the cleanup to the remaining requirements Differential Revision: https://phab.mercurial-scm.org/D10108

requirements: also add a dotencode constant Continue the cleanup to the remaining requirements Differential Revision: https://phab.mercurial-scm.org/D10107