revlog: recommit 49fd21f32695 with a fix for issue6528 `filelog.size` currently special cases two forms of metadata encoding: - copy data via the parent order as flag bit - censor data by peaking into the raw delta All other forms of metadata encoding including the empty metadata block are mishandled. In `basefilectx.cmp` the empty metadata block is explicitly checked to compensate for this. Restore 49fd21f32695, but disable it for filelog, so that the original flag bit use contines to work. Document all this mess for now in preparation of a proper rework. Differential Revision: https://phab.mercurial-scm.org/D11203

merge-lists: make it possible to specify pattern to match The `merge-lists` tool doesn't know anything about Python other than its regex that attempts to match import lines. Let's make it possible to pass in a custom regex so it's easy to use the tool for e.g. C/C++ `#include` lines or Rust `use` lines (given the limited). Differential Revision: https://phab.mercurial-scm.org/D12392

contrib: add a partial-merge tool for sorted lists (such as Python imports) This is a pretty naive tool that uses a regular expression for matching lines. It is based on a Google-internal tool that worked in a similar way. For now, the regular expression is hard-coded to attempt to match single-line Python imports. The only commit I've found in the hg core repo where the tool helped was commit 9cd6292abfdf. I think that's because we often use multiple imports per import statement. I think this tool is still a decent first step (especially once the regex is made configurable in the next patch). The merging should ideally use a proper Python parser and do the merge at the AST (or CST?) level, but that's significantly harder, especially if you want to preserve comments and whitespace. It's also less generic. Differential Revision: https://phab.mercurial-scm.org/D12380

branching: merge stable into default

Added signature for changeset 5bd6bcd31dd1

Added tag 6.1.1 for changeset 5bd6bcd31dd1

relnotes: add notes for 6.1.1 This also fixes the header for 6.1 from 6.1rc0

merge: stable into default

rust-hgpath: add `repr(transparent)` to `HgPath` It's been stabilized a long time ago, so let's not rely on an implementation detail now. Differential Revision: https://phab.mercurial-scm.org/D12433

rust-dirstatemap: correctly decrement the copies counter This was caught when writing unit tests for the `DirstateMap`. We were always setting `had_copy_source` to `false` since we erased the value just before. Differential Revision: https://phab.mercurial-scm.org/D12432

rust-dirstatemap: properly decrement counter for tracked descendants I found this bug when writing unit tests after the fact for the `DirstateMap`. We never decremented the tracked descendants counter since we were always resetting the node data before reading it. This also drops the use of `state`, in favor of the new API to get that information. Differential Revision: https://phab.mercurial-scm.org/D12431

rust-dirstate: panic if the DirstateMap counters go below 0 When modifying the API I hit some... interesting errors (trying to allocate 178GB of RAM, for example) because I failed to keep the counters correctly updated. This counter underflow is likely to happen when code is changed around and can have up to eat-your-dirstate level of consequences, which is not nice. The very small runtime cost of checking these counters should really not be an issue and will help us uncover bugs when/if they do appear in the future. Differential Revision: https://phab.mercurial-scm.org/D12430

rust: fix unsound `OwningDirstateMap` As per the previous patch, `OwningDirstateMap` is unsound. Self-referential structs are difficult to implement correctly in Rust since the compiler is free to move structs around as much as it wants to. They are also very rarely needed in practice, so the state-of-the-art on how they should be done within the Rust rules is still a bit new. The crate `ouroboros` is an attempt at providing a safe way (in the Rust sense) of declaring self-referential structs. It is getting a lot attention and was improved very quickly when soundness issues were found in the past: rather than relying on our own (limited) review circle, we might as well use the de-facto common crate to fix this problem. This will give us a much better chance of finding issues should any new ones be discovered as well as the benefit of fewer `unsafe` APIs of our own. I was starting to think about how I would present a safe API to the old struct but soon realized that the callback-based approach was already done in `ouroboros`, along with a lot more care towards refusing incorrect structs. In short: we don't return a mutable reference to the `DirstateMap` anymore, we expect users of its API to pass a `FnOnce` that takes the map as an argument. This allows our `OwningDirstateMap` to control the input and output lifetimes of the code that modifies it to prevent such issues. Changing to `ouroboros` meant changing every API with it, but it is relatively low churn in the end. It correctly identified the example buggy modification of `copy_map_insert` outlined in the previous patch as violating the borrow rules. Differential Revision: https://phab.mercurial-scm.org/D12429

rust: explain why the current `OwningDirstateMap` is unsound See inline comments. Differential Revision: https://phab.mercurial-scm.org/D12428

dispatch: fix silly blackbox entries when hg is interrupted When hg is interrupted, it creates ui.log like this: 1970/01/01 00:00:00 user @0000000000000000000000000000000000000000 (62488)> killed! exited 255 after 1.78 seconds This is due to a scoping problem: two different uses of the name "msg" collide. So rename one of them. Differential Revision: https://phab.mercurial-scm.org/D12427

tests: stop excluding the pycompat module from pyflakes I assume this was skipped because of all of the py2 stuff causing a lot of spew. The "unused" imports are left in place in case any 3rd party stuff is using it. I don't care about most of it, but TortoiseHg uses `io` and `queue`, so minimally I'd like to keep those. Differential Revision: https://phab.mercurial-scm.org/D12423

tests: drop some py2 specific pyflake failures Differential Revision: https://phab.mercurial-scm.org/D12422

util: drop a duplicate import This was already imported several lines above. Differential Revision: https://phab.mercurial-scm.org/D12421

pycompat: drop the pickle import I suspect this is what df56e6bd37f6 meant to eliminate. Differential Revision: https://phab.mercurial-scm.org/D12420

util: restore the util.pickle symbol This was accidently dropped in df56e6bd37f6, which started importing pickle directly. That commit explicitly says it will retain it for compatibility with external stuff though. The unused import in pycompat isn't flagged because that module is skipped. Just importing with a comment seemed cleaner than `import X as Y` and then assigning to a `pickle` variable, just to avoid the pyflakes warning. Differential Revision: https://phab.mercurial-scm.org/D12419

merge: stable into default

precheck: fix false warning about content-divergence creation Before this patch, if we try to `hg prune` (without any successors) an already obsoleted cset which has at least one successor, it would false warn about new content-divergence. As we know, pruning cset without any successors can not create any divergence. Differential Revision: https://phab.mercurial-scm.org/D12002

streamclone: avoid some obscure error in a corner case I don't really know how, but I ran into this error: $ hg clone --stream ssh://user@dummy/empty-repo local-empty-repo streaming all changes abort: unable to apply stream clone: unsupported format: [255] I think you need an empty list of requirements for this to happen, which is weird, but an obscure error like this is not exactly helpful either. Since this is the result of an encoding bug anyway, just fix it. Differential Revision: https://phab.mercurial-scm.org/D12402

tags: fix typo in fast path detection of fnode resolution (issue6673) If I understand it, mctx.readfast() is unreliable here if p1/p2 .hgtags nodes differ, and tags on that branch would be randomly discarded depending on which parent were picked. The test case added by this patch would fail only on zstd-compressed repository. I didn't try hard to stabilize the failure case.

completion: install completers to conventional locations Installs the bash and zsh completers to the convential locations so they will automatically be picked up without user intervention. The zsh completer on Debian is still installed to vendor-completions to match their policy. bash: https://github.com/scop/bash-completion#faq zsh: https://github.com/zsh-users/zsh/blob/57305cf245853b8b30895b41a90142dffab97e38/INSTALL#L254 Debian zsh: https://salsa.debian.org/debian/zsh/-/blob/5086b5356abcef8849dc8a09902b7c55f01db3c0/debian/README.Debian#L73

dirstate-cext: properly invalidate mtime and data in `set_untracked` This was forgotten about in the initial implementation and was revealed while adding the `dirstate-v2` variant of `test-issue660.t`. Neither the existing Python implementation nor the upcoming Rust implementation suffer from this bug since they respectively have `None` and `Option<T>` to represent the lack of information. Differential Revision: https://phab.mercurial-scm.org/D12414

pytype: disable a few errors about Windows specific module attributes These were flagged by pytype 2022.03.21. Differential Revision: https://phab.mercurial-scm.org/D12401

rhg: sort unsupported extensions in error message This caused some flakiness in test output, and is also just better for users. Differential Revision: https://phab.mercurial-scm.org/D12389

hgignore: ignore .testtimes in more location See the inline comment. Differential Revision: https://phab.mercurial-scm.org/D12393

merge: with stable

partial-merge: add support for `.args` config (`$local` etc.) It will be useful to be able to define custom command-line arguments per partial merge tool just like we have for regular merge tools. In particular, I expect the same binary to handle multiple languages, so it will be useful to be able to pass some argument indicating the language, or perhaps simply an argument defining a regex that's used for finding lines to merge as a sorted set. Differential Revision: https://phab.mercurial-scm.org/D12383

filemerge: add support for partial conflict resolution by external tool A common class of merge conflicts is in imports/#includes/etc. It's relatively easy to write a tool that can resolve these conflicts, perhaps by naively just unioning the statements and leaving any cleanup to other tools to do later [1]. Such specialized tools cannot generally resolve all conflicts in a file, of course. Let's therefore call them "partial merge tools". Note that the internal simplemerge algorithm is such a partial merge tool - one that only resolves trivial "conflicts" where one side is unchanged or both sides change in the same way. One can also imagine having smarter language-aware partial tools that merge the AST. It may be useful for such tools to interactively let the user resolve any conflicts it can't resolve itself. However, having the option of implementing it as a partial merge tool means that the developer doesn't *need* to create a UI for it. Instead, the user can resolve any remaining conflicts with their regular merge tool (e.g. `:merge3` or `meld). We don't currently have a way to let the user define such partial merge tools. That's what this patch addresses. It lets the user configure partial merge tools to run. Each tool can be configured to run only on files matching certain patterns (e.g. "*.py"). The tool takes three inputs (local, base, other) and resolves conflicts by updating these in place. For example, let's say the inputs are these: base: ``` import sys def main(): print('Hello') ``` local: ``` import os import sys def main(): print('Hi') ``` other: ``` import re import sys def main(): print('Howdy') ``` A partial merge tool could now resolve the conflicting imports by replacing the import statements in *all* files by the following snippet, while leaving the remainder of the files unchanged. ``` import os import re import sys ``` As a result, simplemerge and any regular merge tool that runs after the partial merge tool(s) will consider the imports to be non-conflicting and will only present the conflict in `main()` to the user. Differential Revision: https://phab.mercurial-scm.org/D12356

pullbundle: fix file name in the help text It is pullbundles.manifest and not pullbundle.manifest. Differential Revision: https://phab.mercurial-scm.org/D12391

unamend: abort if commit was not created by `hg [un]amend` `hg unamend` can currently undo any kind of rewrite, as long as it has an obsmarker. However, that has quite unexpected results if you run it after e.g. `hg rebase` (expecting it to behave like a generic `hg undo` command), because it updates to the predecessor and leaves the old changes in the working copy. I think it's better to allow `hg unamend` only after `hg amend` (and after `hg unamend` because that's documented as being supported). Differential Revision: https://phab.mercurial-scm.org/D12390

branching: merge stable into default

test: use `wait-on-file` in `test-racy-mutations.t` The official utility scale its timeout with the run-tests.py one. So lets use it. Differential Revision: https://phab.mercurial-scm.org/D12382

amend: fix amend with copies in extras If copy information is stored only in the commit extras and not in filelogs, then they get lost on amend if the file wasn't also modified in the working copy. That's because we create `filectx` object from the old commit in those cases, and the `.copysource()` of such objects read only from the filelog. This patch fixes it by always creating a new `memfilectx` in these cases, passing the calculated copy information to it. Differential Revision: https://phab.mercurial-scm.org/D12387

tests: demonstrate that copy info in changeset gets lost on amend When copy information is stored in changesets, it gets lost on amend. We didn't notice that until now because our users at Google have the config set to `compatibility`, which means copy information is stored in both changeset and filelogs. Differential Revision: https://phab.mercurial-scm.org/D12386

ci: use the `v1.0` flavor of the docker images in the CI This new versioning will help us to maintain backward compatibility in the docker image. This will be useful to deal with mismatch between default/stable in version and the re-run CI on older changesets in the future. Once this changeset land on stable, we will have to merge it in default. Then we can start make backward incompatible changes in a new image version. Differential Revision: https://phab.mercurial-scm.org/D12388

rust-status: cap the number of concurrent threads to 16 During benchmarking it was determined that the use of more threads is very advantageous... until we use more than 16. This is most likely due to some resource contention (thrashing, etc.). Until we have time to figure out and fix the underlying cause, let's just cap at 16 threads. Differential Revision: https://phab.mercurial-scm.org/D12384

rhg: add support for ignoring all extensions Some workflows just want what `rhg` does and don't care about any extensions, this makes it easier. Differential Revision: https://phab.mercurial-scm.org/D12385

branching: merge stable into default

revlog: fix index_fast_rank (wip) As far as I can tell, rank is stored as a 32-bit big endian value, I'm not sure how grabbing the first byte can possibly work. I assume there's no test coverage here? cc @pacien Fixes: https://www.mercurial-scm.org/repo/hg/rev/e633e660158f Differential Revision: https://phab.mercurial-scm.org/D12376

tests: fix glob pattern for dynamic timer alignment The number of space characters varies depending on the number of digits of the timer, making some tests fail on slow machines in an unintended way: ```diff --- /build/mercurial-6.1/tests/test-merge-halt.t +++ /build/mercurial-6.1/tests/test-merge-halt.t.err @@ -210,6 +210,6 @@ merge halted after failed merge (see hg resolve) [240] $ hg shelve --list - default (* ago) changes to: foo (glob) + default (11s ago) changes to: foo $ hg unshelve --abort unshelve of 'default' aborted ERROR: test-merge-halt.t output changed ``` Differential Revision: https://phab.mercurial-scm.org/D12381

test: update test-clone-stream.t to pass on bigendian Fixes: a3cf460a6b1b ("stream-clone: also filter the requirement we put in the bundle 2") Differential Revision: https://phab.mercurial-scm.org/D12377

filemerge: when merge tool uses $output, don't leave markers in $local As explained in the previous patch, we incorrectly leave conflict markers in both `$local` and `$output` since D12190. I don't understand why it broke but the fix is simple and clear after all the recent refactoring. Differential Revision: https://phab.mercurial-scm.org/D12379

tests: demonstrate how conflict markers end up $local *and* $output When a merge tool is configured to keep conflict markers, they are supposed to be written to `$local` if `$output` is not mentioned in the tool's `merge-tools.<tool>.args` config, and in `$output` if it is mentioned. However, I broke the latter case in D12190. Differential Revision: https://phab.mercurial-scm.org/D12378

branching: merge stable into default

revlog: fix wrong type of rank_unknown variable We treat "rank" as an int everywhere, but declare rank_unknown as a char. On architectures where char is signed, that works out ok, but when char is unsigned, rank_unknown is 255 instead of -1. Differential Revision: https://phab.mercurial-scm.org/D12374

tests: fix formatting issue in run-tests.py after c194e93d1ebc Differential Revision: https://phab.mercurial-scm.org/D12375

rust-hg-core: use correct type for libc hostname buffer The type of libc::c_char is u8 on aarch64 rather than i8, which causes the use of a specifically-typed constant to fail. Differential Revision: https://phab.mercurial-scm.org/D12373

import-checker: allow symbol imports from typing module As we add typing annotations, we'll want to use a lot of symbols from the `typing` module. Typing `typing` all the time will be annoying. Let's allow symbol imports from this module. While I was here, I changed some comments from "whitelist" to "allow list" as the former is non-inclusive terminology. Differential Revision: https://phab.mercurial-scm.org/D12365

pycompat: remove json.loads polyfill for Python 3.5 We no longer support Python 3.5 so this can be deleted. Differential Revision: https://phab.mercurial-scm.org/D12364

pycompat: remove check for Python >= 3.6 We dropped support for Python 3.5 so this is always true. Differential Revision: https://phab.mercurial-scm.org/D12363

hgdemandimport: delete check for Python 3.5 We dropped support for Python 3.5. So we no longer need to do this. Differential Revision: https://phab.mercurial-scm.org/D12362

hg: always import hgdemandimport The deleted if condition is always true now that we dropped Python 2 and 3.5. Differential Revision: https://phab.mercurial-scm.org/D12361

tests: support another error case when detecting ipv4/ipv6 support I encountered this on Linux in a VM environment with a rather strange networking setup (both on the host and in the VM). Differential Revision: https://phab.mercurial-scm.org/D12371

debugdiscovery: fix a typo in the help Differential Revision: https://phab.mercurial-scm.org/D12372

rust-hg-cypython: upgrade dependencies This upgrades all dependencies to their latest version. This is routinely done to keep-up. Differential Revision: https://phab.mercurial-scm.org/D12359

rust-hg-core: upgrade dependencies This upgrades all dependencies to their latest version, except `clap` and `zstd` whose latest versions do not support our minimum supported Rust version 1.48.0. Same as for `rhg`, it contains security fix for `regex` which does not affect us too much, but doesn't hurt, and the rest of the upgrades are there simply to keep up. Differential Revision: https://phab.mercurial-scm.org/D12358