urlutil: add a new `get_unique_push_path` This function is dedicated to call that needs a single destination. Currently most caller actually need that since few actually support multiple destinations (the most importants `hg push` and `hg outgoing` do). So having a clear API point for that will be important when the time comes to have a single `[paths]` alias resolving to multiple urls. Differential Revision: https://phab.mercurial-scm.org/D10407

fetch: use `get_unique_pull_path` to retrieve the path The fetch command does not support multiple destination, so we use the new dedicated API for that. Differential Revision: https://phab.mercurial-scm.org/D10406

patchbomb: use `get_unique_pull_path` The patchbomb code does not support multiple destinations yet. It would not be too hard to implemented since `hg outgoing` does. However that is a bit too much of a detour right now. In addition that kind of "select outgoing changesets for processing" code exist for various commands so the best would be to factor it out later. Let us use the dedicated API until this is solved. This comes with a small output change in the test that I find more accurate. So I kept it. Differential Revision: https://phab.mercurial-scm.org/D10405

narrow: use `get_unique_pull_path` The narrow's `tracked` command does not support multiple destination, lets use the dedicated API then. Differential Revision: https://phab.mercurial-scm.org/D10404

urlutil: add a new `get_unique_pull_path` This function is dedicated to call that needs a single destination. Currently most caller actually need that since few actually support multiple destinations (the most importants `hg push` and `hg outgoing` do). So having a clear API point for that will be important when the time comes to have a single `[paths]` alias resolving to multiple urls. Differential Revision: https://phab.mercurial-scm.org/D10403

mq: use the new `get_clone_path` to get the remote url We stop using `ui.expandpath` to use a function with a more defined semantic. This will help us to make it an option to point [paths] entry to multiple destination. Differential Revision: https://phab.mercurial-scm.org/D10402

urlutil: add a `get_clone_path` function We add a new function with a semantic focussed on `clone` operation (so without an existing repository). I am not certain the return type is the best, but this is what we need for now. Once all caller are migrated we might start thinking about that the API should be. For now that will do. Differential Revision: https://phab.mercurial-scm.org/D10401

outgoing: use `get_push_paths` in the revset too The revsets now use the same code as everyone and is ready for entry in `[paths]` pointing to multiple entries. Differential Revision: https://phab.mercurial-scm.org/D10393

incoming: use `urlutil.get_pull_paths` Lets use the new method to increase code reuse. However I did not implement support for multiple source yet. It would be possible create multiple temporary bundle, but that is a bit too much work outside of my current route to make the detour. Differential Revision: https://phab.mercurial-scm.org/D10392

outgoing: accept multiple destinations This align the behavior of `hg outgoing` with the one of `hg incoming`. In addition this prepare the introduction of having simple `path` resolve to multiple destination in practice (eg: `default`) Differential Revision: https://phab.mercurial-scm.org/D10391

outgoing: pass subrepo path using function argument instead of abssource hack This is clearer, remove the needs for the `repo._subtoppath` hack and will make our live easier when making `outgoing` accept multiple destinations. Differential Revision: https://phab.mercurial-scm.org/D10390

subrepo: introduce a `repo_rel_or_abs_source` function The `subrepoutil` module has various function to compute the path of a sub-repository compared to the root of the top repository. However, they was no function available to compute the relative path of the repository "source". And we need this information for exchange operation (in our case, `hg outgoing`). The information is currently build using the `repo._subtoppath` hack. We reuse the same logic but in a dedicated function independent of the `repo._subtoppath` hack. Differential Revision: https://phab.mercurial-scm.org/D10434

urlutil: use bytes for Abort messages Caught by pytype. Also fix a minor grammar issue in the second message. Differential Revision: https://phab.mercurial-scm.org/D10433

changegroup: update comment Gratuitous fix, I stumbled upon this comment while reading code. Differential Revision: https://phab.mercurial-scm.org/D10339

re2: adjust local variable assignment scope Pytype flagged the exception handler where these are used as `[name-error]`, and PyCharm complained they may be used before assignment. Differential Revision: https://phab.mercurial-scm.org/D10376

rhg: Make `files` work on repo-relative paths when possible When the current directory is outside of the repository we need to turn everything into absolute filesystem paths in order to compute correct relative paths. This was previously done unconditionally, but is not necessary when the current directory is inside the repository. With this change `rhg files > /dev/null` at the root of a mozilla-central snapshot goes from ~150 ms to ~70 ms. My repository is located at a somewhat long path though (93 bytes). The effect may not be as pronounced at a shorter path. Differential Revision: https://phab.mercurial-scm.org/D10200

split: avoid strip if split is a no-op (identical to original) Differential Revision: https://phab.mercurial-scm.org/D10389

split: fix issue with empty splits adjusting phases Differential Revision: https://phab.mercurial-scm.org/D10388

split: add test demonstrating issue with empty splits adjusting phases Differential Revision: https://phab.mercurial-scm.org/D10387

test-lfs: fix expected output This is not covered by the CI and was broken. Differential Revision: https://phab.mercurial-scm.org/D10397

test-lfs: avoid a bashism when spawning the server For zsh, this &> call is read as "& >", this spin the process without redirection. As a result the server grab stdout, which does not get closed at the end of the test. As a result `run-tests.py` hang there forever, waiting for the the stream for be close. (which is probably as issue on its own). Differential Revision: https://phab.mercurial-scm.org/D10396

test-lfs: avoid pagination Otherwise, the pagination can trigger in when running the test with --debug, blocking the test execution for no good reason. Differential Revision: https://phab.mercurial-scm.org/D10395

run-test: make it clearer why we terminating process The --verbose log wer only talking about process termination. Now we have more information about why. Differential Revision: https://phab.mercurial-scm.org/D10394

push-dests: move the code around missing default dest inside `get_push_paths` This function has a clear semantic and moving the code dealing with this inside it will help reduce duplication. Differential Revision: https://phab.mercurial-scm.org/D10385

push-dests: rework the handling of default value This new core is more straightforward and doing this early will make the next changeset simpler. Differential Revision: https://phab.mercurial-scm.org/D10384

outgoing: merge the code handling --graph with the main one The --graph code had its own copy of the logic. With the previous reorganisation of the code, we can now merge it with the main code, reducing fragile complication. As a side effect, `hg out --graph` now use the right return code when they are nothing outgoing. This explain the change to output in `tests/test-largefiles-misc.t`. Differential Revision: https://phab.mercurial-scm.org/D10383

outgoing: move filtering logic in its own function This move code dedicated to a single purpose together and make the main code simpler. Right when we are getting ready to make it more complex :-D Differential Revision: https://phab.mercurial-scm.org/D10382

outgoing: make `recurse` a real function If we want to use this in a loop, we need to be able to pass argument. Differential Revision: https://phab.mercurial-scm.org/D10381

outgoing: remove some early return Since 066b8d8f75b8, the push command accept multiple destination. However `hg outgoing` does not. On the way to fix this, we need to clean up the outgoing code. We start with removing some early return to make the code ready to house more changes. Differential Revision: https://phab.mercurial-scm.org/D10380

infinitepush: use the new function to determine push destination Since 066b8d8f75b8, the push command accept multiple destination. `infinitepush` was not aware of that. We now use the new `urlutil.get_push_paths` function to determine the push destination, fixing the issue. This will also make future evolution of that logic transparent for infinitepush We still disallow push to multiple destinations if infinite push is enabled because I don't know what this means for infinite push. However user will now get a clear error message instead of a crash. Differential Revision: https://phab.mercurial-scm.org/D10379

urlutil: add a `get_pull_paths` to perform the pull destination logic As is this changeset does not change anything. However having an official empty point will help unifying the logic and encapsulate the details and update the logic to support path definition pointing to multiple other path. Differential Revision: https://phab.mercurial-scm.org/D10378

urlutil: add a `get_push_paths` to perform the push destination logic As is this changeset does not change anything. However having an official empty point will help unifying the logic and encapsulate the details and update the logic to support path definition pointing to multiple other path. Differential Revision: https://phab.mercurial-scm.org/D10377

urlutil: extract `parseurl` from `hg` into the new module The new module is well fitting for this new code. And this will be useful to make the gathered code collaborate more later. Differential Revision: https://phab.mercurial-scm.org/D10375

urlutil: extract `url` related code from `util` into the new module The new module is well fitting for this new code. And this will be useful to make the gathered code collaborate more later. Differential Revision: https://phab.mercurial-scm.org/D10374

urlutil: extract `path` related code into a new module They are a lot of code related to url and path handling scattering into various large module. To consolidate the code before doing more change (for defining "multi-path"), we gather it together. Differential Revision: https://phab.mercurial-scm.org/D10373

help: document the `path://` url scheme If we want people to use it, we need to document it. Differential Revision: https://phab.mercurial-scm.org/D10372

help: point to `hg help urls` in `hg help config.paths` This seems useful to point at what people can put as value for these config. Differential Revision: https://phab.mercurial-scm.org/D10371

merge with stable

narrow: add capabilities for local repos, not just remote peers This fixes the bug where running `hg clone --narrow ./local-repo` fails with abort: server does not support narrow clones even when the server has narrow enabled. Differential Revision: https://phab.mercurial-scm.org/D10357

typing: make minor adjustments to mercurial/util.py to pass pytype checking I'm assuming the wrong-arg-count is a pytype bug, because this code is used by the config object. Avoiding initializing `_lrucachenode` node points to None eliminates a few `is not None` assertions, but apparently not all of them. I can't figure out why it gets confused over the state where these new assertions are. Differential Revision: https://phab.mercurial-scm.org/D10276

util: fix the signature for the pypy override of sortdict.update() PyCharm flagged this as not matching the base class signature. Not sure if there was anything supplying these extra arguments though. Differential Revision: https://phab.mercurial-scm.org/D10275

typing: add type hints to mercurial/error.py The only slightly unusual things here are that `location` is passed to `ParseError` and both bytes and an int (so this accepts both), and the message passed `ProgrammingError` is immediately converted to str. Therefore it is typed as `AnyStr`, because there are a couple of instances that are already passed as str. There are a couple of places where bytes are being passed to builtin exceptions that might need to be converted to str. Differential Revision: https://phab.mercurial-scm.org/D10274

tests: add a (very slow) test that executes pytype This is an updated form of D7295, and completes successfully with pytype 2021.03.22. The 5 or so crashes that were mostly in the hgweb files seems to have been fixed in 2021.03.10. Differential Revision: https://phab.mercurial-scm.org/D10237

upgrade: do not hardcore file extension of revlogs This logic already lives inside the `store` module. So lets reuse it instead. Differential Revision: https://phab.mercurial-scm.org/D10317

upgrade: take advantage of the new information returned by `store.walk` Before this change the upgrade code had to analyse filename to process them directly. Lets keep that logic private to the store and more to a more robust explicit approach. Differential Revision: https://phab.mercurial-scm.org/D10316

store: also return some information about the type of file `walk` found We start returning of 4th information in the `store.walk` return tuple: the type of the file. This will make it easier for caller to determine which kind of file they are looking at. This should especically help with the `upgrade-repo` code that has to do a lot of fragile index's file name comparison. Differential Revision: https://phab.mercurial-scm.org/D10315

tests: skip test-git-interop.t on Windows Casefolding isn't handled in dirstate yet, triggering a bunch of assertions. But while this is more correctly `no-icasefs`, it's more likely to get attention if someone sees it. I'd just rather not have it adding to the noise on Windows for now. Differential Revision: https://phab.mercurial-scm.org/D10312

contrib: restore the `hg fix` configuration in the examples After decc3bd3f20d, running `black` will DTRT, but running `hg fix` did nothing (unless the example config file was %included, in which case it truncated the file instead of formatting it). I'm not sure why that was happening, but let's not leave a code shredder laying around. Differential Revision: https://phab.mercurial-scm.org/D10311

blackbox: fix type error on log rotation on read-only filesystem Grepping around, the code uses either encoding.strtolocal or stringutil.forcebytestr in this situation. No idea which is best. Differential Revision: https://phab.mercurial-scm.org/D10293

rust: Remove use of `py.eval()` The previous Rust code allocated an intermediate `Vec`, converted that to a Python list, then used `eval` to run Python code that converts that list to a Python set. rust-cpython exposes Rust bindings for Python sets, let’s use that instead to construct a set directly. Differential Revision: https://phab.mercurial-scm.org/D10328

rust: Remove the compile-time 'dirstate-tree' feature flag This code has compiler errors since it is not built on CI and nobody has been working on it for some time. We (Octobus) are still pursuing status optimizations based on a tree data structure for the dirstate, but upcoming patches will use a run-time opt-in instead of compile-time, so that at least corresponding Rust code keeps compiling when other changes are made. Differential Revision: https://phab.mercurial-scm.org/D10329

procutil: avoid using os.fork() to implement runbgcommand We ran into the following deadlock: - some command creates an ssh peer, then raises without explicitly closing the peer (hg id + extension in our case) - dispatch catches the exception, calls ui.log('commandfinish', ..) (the sshpeer is still not closed), which calls logtoprocess, which calls procutil.runbgcommand. - in the child of runbgcommand's fork(), between the fork and the exec, the opening of file descriptors triggers a gc which runs the destructor for sshpeer, which waits on ssh's stderr being closed, which never happens since ssh's stderr is held open by the parent of the fork where said destructor hasn't run Remotefilelog appears to have a hack around this deadlock as well. I don't know if there's more subtlety to it, because even though the problem is determistic, it is very fragile, so I didn't manage to reduce it. I can imagine three ways of tackling this problem: 1. don't run any python between fork and exec in runbgcommand 2. make the finalizer harmless after the fork 3. close the peer without relying on gc behavior This commit goes with 1, as forking without exec'ing is tricky in general in a language with gc finalizers. And maybe it's better in the presence of rust threads. A future commit will try 2 or 3. Performance wise: at low memory usage, it's an improvement. At higher memory usage, it's about 2x faster than before when ensurestart=True, but 2x slower when ensurestart=False. Not sure if that matters. The reason for that last bit is that the subprocess.Popen always waits for the execve to finish, and at high memory usage, execve is slow because it deallocates the large page table. Numbers and script: before after mem=1.0GB, ensurestart=True 52.1ms 26.0ms mem=1.0GB, ensurestart=False 14.7ms 26.0ms mem=0.5GB, ensurestart=True 23.2ms 11.2ms mem=0.5GB, ensurestart=False 6.2ms 11.3ms mem=0.2GB, ensurestart=True 15.7ms 7.4ms mem=0.2GB, ensurestart=False 4.3ms 8.1ms mem=0.0GB, ensurestart=True 2.3ms 0.7ms mem=0.0GB, ensurestart=False 0.8ms 0.8ms import time for memsize in [1_000_000_000, 500_000_000, 250_000_000, 0]: mem = 'a' * memsize for ensurestart in [True, False]: now = time.time() n = 100 for i in range(n): procutil.runbgcommand([b'true'], {}, ensurestart=ensurestart) after = time.time() ms = (after - now) / float(n) * 1000 print(f'mem={memsize / 1e9:.1f}GB, ensurestart={ensurestart} -> {ms:.1f}ms') Differential Revision: https://phab.mercurial-scm.org/D9019

share: store relative share paths with '/' separators I created a relative share in Windows and tried to use it in WSL, and it failed: abort: .hg/sharedpath points to nonexistent directory /mnt/c/Users/Matt/hg-review/.hg/..\..\hg\.hg Use `normpath` on the read side so that the code has the usual Windows style paths it always had (I don't think that matters much), but it also eliminates the directory escaping path components in the case where the path is printed. This will not fix repositories that have already been created, but it's trivial enough to hand edit the file to correct it. Differential Revision: https://phab.mercurial-scm.org/D10330

unit-tests: Fix `cargo test` on 32-bit platforms Fixes https://bz.mercurial-scm.org/show_bug.cgi?id=6506 This makes `IndexEntryBuilder::build`, which is only used in unit tests, use `u32` or `u64` instead of platform-dependent `usize` when packing binary data to be used at test input. To run Rust unit tests in 32-bit mode in a x86-64 environment, use: rustup target add i686-unknown-linux-gnu # Once (cd rust && cargo test --target i686-unknown-linux-gnu) Differential Revision: https://phab.mercurial-scm.org/D10351

rename: add --forget option and stop suggesting `hg revert` for undoing Differential Revision: https://phab.mercurial-scm.org/D10355

win32: enable legacy I/O mode to fix missing pager output on Windows with py3 The equivalent interpreter option is set by wrapper.exe, but this *.bat file is what gets installed in a venv. Without this mode, any command that spins up a pager has no output, unless the pager is explicitly disabled. The variable is set inside the `setlocal` scope to keep it from leaking into the environment after the bat file exits. We should probably still figure out how to ship a compiled hg.exe when installing with `pip`, because the binary does other things like enable long filename support. But this avoids the dangerous and confusing lack of output in the meantime. Differential Revision: https://phab.mercurial-scm.org/D10354

persistent-nodemap: enable the feature by default when using Rust As discussed at the 5.6 sprint, we can make it enabled by default, but only for Rust installation. Differential Revision: https://phab.mercurial-scm.org/D9765

persistent-nodemap: disable it for test-wireproto-content-redirects.t This is not relevant for the test and this will avoid a lot of variations. Differential Revision: https://phab.mercurial-scm.org/D10338

persistent-nodemap: disable it for test-wireproto-command-capabilities.t This is not relevant for the test and this will avoid a lot of variations. Differential Revision: https://phab.mercurial-scm.org/D10337

persistent-nodemap: disable it unconditionally for test-wireproto-caching.t This is not relevant for the test and this will avoid a lot of variations. Differential Revision: https://phab.mercurial-scm.org/D10336