merge: default into stable for 5.8 rc

mail: force SMTPException to bytes before wrapping in error.Abort Pytype wasn't complaining about this for some reason, but PyCharm was and we already handle another instance of this exception the same way in this function. Differential Revision: https://phab.mercurial-scm.org/D10473

mail: split out the SMTP login to allow the keyring extension to wrap it The keyring extension only needs to tweak this tiny section of the larger function. But without any place to intercept the username/password fetching, it copy/pasted the entire function, and has grown a bunch of compatibility hacks to support older versions of Mercurial as well. Differential Revision: https://phab.mercurial-scm.org/D10471

store: exclude `undo.` nodemap's file from `walk` There are "temporary" local file that we should not be transfered by `walk` user like local clone and stream clone. This fix the small issue that the new tests highlighted. Differential Revision: https://phab.mercurial-scm.org/D10482

streamclone: treat volatile file as "fullfile" The nodemap related file might change (or get deleted) during the stream clone in a way incompatible with the streaming process. So we introduce a new flag for this type of file and integrate it with the existing `revlog_type` field recently added to `store.walk` returns. We use that new flat to dispatch such file to the existing mechanism for "atomic replacement" file for the nodemap docket and datafile. This fix the bugs we have been adding tests for. Strictly speaking, the nodemap datafile is happened only a could maybe be used in a slightly more efficient way, however this is good enough for now. Differential Revision: https://phab.mercurial-scm.org/D10481

nodemap: add a test about nodemap "vacuum" during stream clone When the nodemap "vacuum" it generate a new datafile, with a new unique name and delete the old one. This confuse the stream clone code and create the same kind of crash we were seeing in the previous patch. We build a test case where this happens. The next changeset will contains a fix We can also notice that we are, wrongfully exchanging `undo.*` files that the client will not be able to use. This will be dealt with soon. Differential Revision: https://phab.mercurial-scm.org/D10480

nodemap: add a test about racy commit during stream clone That test show that the resulting client nodemap is different from the server one. This happens because the server one transferred a corrupted node map. The data file match the pre-commit content while the docket has post commit content. As the result the nodemap was detected invalid and recomputed. When running without the rust implementation, the code is also generating a new datafile unconditionally, This mean the older file is no longer there are transfer time, resulting in a crash. We will fix this issue later, but we start with writing tests highlighting the issue. Differential Revision: https://phab.mercurial-scm.org/D10479

streamclone: remove sleep based "synchronisation" in tests Sleep based test synchronisation does not work. Variation in machine performance and load can make the two process miss their windows. Instead we migrate to explicit signaling through the file system as other tests file are using. Differential Revision: https://phab.mercurial-scm.org/D10478

testing: add a `write_file` function The function is kinda trivial, but having a simple function avoid silly mistake. Differential Revision: https://phab.mercurial-scm.org/D10477

testing: add a utility function to wait for file create This is similar to `tests/testlib/wait-on-file`, but for the python code Differential Revision: https://phab.mercurial-scm.org/D10476

streamclone: check the errors log in tests Code can be buggy, in that case having error message tend to help. Differential Revision: https://phab.mercurial-scm.org/D10475

mail: use a context manager when writing to mbox Differential Revision: https://phab.mercurial-scm.org/D10484

errors: remove unnecessary varargs handling from OutOfBandError In my recent D10465, I moved some code over from scmutil into `OutOfBandError.__init__`. The code was written to deal with an arbitrary number of `message` arguments to the constructor. It turns out that we only ever pass 0 or 1. Given that, let's simplify it. Differential Revision: https://phab.mercurial-scm.org/D10483

phab-refresh: use a special comment on the stable branch Phabricator make it hard to spot the branch information. As a result patch intended for stable are consistently queued for default. We add a special command and flashy GIF to try to attract reviewer attention on the fact patch are intended for stable. Differential Revision: https://phab.mercurial-scm.org/D10464

phab-refresh: extract the comment in a variable This make the script more readable and it is about to become more complex. Differential Revision: https://phab.mercurial-scm.org/D10463

tests: make test-archive.t pass on py3.9 (issue6504) Something got stricter at parsing URL query parameters and now the parameters need to be separated by "&"; ";" is no longer allowed. See issue6504 for details. Differential Revision: https://phab.mercurial-scm.org/D10472

errors: raise RemoteError in some places in exchange.py This is part of https://www.mercurial-scm.org/wiki/ErrorCategoriesPlan Differential Revision: https://phab.mercurial-scm.org/D10467

errors: introduce a class for remote errors Having an exception for remote errors makes it much easier to exit with the right detailed exit code. Differential Revision: https://phab.mercurial-scm.org/D10466

errors: make OutOfBandError extend Abort I'm about to create a new `RemoteError` exception and make `OutOfBandError` extend it. This patch prepares for that. Differential Revision: https://phab.mercurial-scm.org/D10465

revlog: replace revlog._io.size with a new revlog.index.entry_size The `revlogio` class is mostly a relic from the past. Once in charge of the full revlog related Input/Output code, that class gradually lost responsibilities to the point where more IO are now done by `revlog.index` objects or revlog objects themself. I would like to ultimately remove the `revlogio` class, to do so I start simple with move the "entry size" information on the index. (The index is already responsible of the binary unpacking, so it knows the size. Differential Revision: https://phab.mercurial-scm.org/D10309

perf-discovery: use `get_unique_pull_path` This performance command now use the new API, unless the benchmarked Mercurial is older. Differential Revision: https://phab.mercurial-scm.org/D10423

help: clarify when `$HG_URL` and `$HG_SOURCE` are added in hooks Differential Revision: https://phab.mercurial-scm.org/D10400

nodemap: deal with data mmap error If the file is too small, the mmapread call would raise a ValueError. We catch that and ignore nodemap content (as we do without mmap). This make the repository slightly slower (until the next write) but usable. Unlike the current crash. Differential Revision: https://phab.mercurial-scm.org/D10458

nodemap: test various corruption scenario for the persistent nodemap Corruption can happens in the wild, either because some of our code is buggy or because repository were shared/transfered in a strange manners. Currently one of the corruption case (not enough data) lead to a crash. We add explicit tests for theses cases. Differential Revision: https://phab.mercurial-scm.org/D10457

cmdutil: fix an uninitialize variable usage in clearunfinished() I happened to notice that PyCharm flagged this while looking for something else. I'm pretty sure it meant to check the current state in the iteration, and not keep reusing the last one in the previous iteration. Differential Revision: https://phab.mercurial-scm.org/D10459

help: document `$HG_NEW_OBSMARKERS` in pretxnclose hook Differential Revision: https://phab.mercurial-scm.org/D10399

help: prepend environment variables used in hooks with "$" Most of them have "$" prefix but some don't, so adding it for consistency. Differential Revision: https://phab.mercurial-scm.org/D10398

revset: drop an outdated comment There is not `ui.expandpath` call there anymore. Differential Revision: https://phab.mercurial-scm.org/D10435

ui: deprecated `ui.expandpath` This function return a single path, something incompatible with the coming change to `[paths]` definition to allow multiple associated urls. Now that all user got migrated, we deprecate the function. Differential Revision: https://phab.mercurial-scm.org/D10432

urlutil: remove usage of `ui.expandpath` in `get_clone_path` We want to deprecate `ui.expandpath` and simplify the code before adding more complexity in the form of `[paths]` entry pointing to multiple url. So we inline the relevant bits. Differential Revision: https://phab.mercurial-scm.org/D10431

urlutil: remove usage of `ui.expandpath` in `get_unique_pull_path` We want to deprecate `ui.expandpath` and simplify the code before adding more complexity in the form of `[paths]` entry pointing to multiple url. So we inline the relevant bits. Differential Revision: https://phab.mercurial-scm.org/D10430

urlutil: remove usage of `ui.expandpath` in `get_pull_paths` We want to deprecate `ui.expandpath` and simplify the code before adding more complexity in the form of `[paths]` entry pointing to multiple url. So we inline the relevant bits. Differential Revision: https://phab.mercurial-scm.org/D10429

debugbackupbundle: use `get_unique_pull_path` This is the last known user of `ui.expandpath` outside of `urlutil`. Hooray. Differential Revision: https://phab.mercurial-scm.org/D10428

debugssl: use `get_unique_pull_path` Let's move this code to the new API. Differential Revision: https://phab.mercurial-scm.org/D10427

debugdiscovery: use `get_unique_pull_path` Lets move this code to the new API. Differential Revision: https://phab.mercurial-scm.org/D10426

profiling: use `util.expandpath` instead of `ui.expandpath` for output Given we are talking about the path to an output file, I am fairly certain that the initial author meant to expand `~` and the like and not to resolve entry from `[paths]`. Differential Revision: https://phab.mercurial-scm.org/D10425

dispatch: use the new API to resolve --repository An even weirder feature of Mercurial is the ability to use `[paths]` alias as value of `--repository`. The weird feature now use the new APIs. Differential Revision: https://phab.mercurial-scm.org/D10424

revset: use `get_unique_pull_path` for `remote(…)` This one is also resolving path. For now we restrict the feature to a single destination. This might change in the future. Differential Revision: https://phab.mercurial-scm.org/D10422

remotefilelog: use `get_unique_pull_path` in `getflogheads` Another command moved to the new API, hooray. Differential Revision: https://phab.mercurial-scm.org/D10421

summary: use the new APIs Summary can perform some incoming/outgoing queries (that should be common to the other command with the same needs, but that is another story). We now use the new APIs to do so. The current code behavior is a bit fishy, relying to the fact "default" will be picked as the destination in last resort. I did not altered that, but left various comment to highlight the issue. Differential Revision: https://phab.mercurial-scm.org/D10420

share: use `get_clone_path` Since `hg share` is mostly like clone, do the same changes there. Differential Revision: https://phab.mercurial-scm.org/D10419

init: use `get_clone_path` when suitable `hg init` has this weird feature were you can refer to `[paths]` entry select the path to initialize. We move that code to the new APIs. Differential Revision: https://phab.mercurial-scm.org/D10418

clone: use `get_clone_path` "Surprisingly", the new API is well suited for `hg clone` too. Differential Revision: https://phab.mercurial-scm.org/D10417

incoming: kill the `repo._subtoppath =` hack We do the same as for `hg outgoing`, instead of relying on implicit passing value by monkey punching them onto the repo object, we pass equivalent information by argument to the proper function. This is way cleaner. Differential Revision: https://phab.mercurial-scm.org/D10416

identify: use `get_unique_pull_path` The command only support a single destination. We use the associated API. Differential Revision: https://phab.mercurial-scm.org/D10415

bundle: support multiple destinations `hg outgoing` and `hg push` now support multiple destination. We do the same for `hg bundle`. Various other commands needs this kind of behavior and it would be create to factor this code out for their usage. However this is an adventure for another time. They are some minor change to the tests (in addition to the new test for the feature): * one because I updated a message to plurals, * another one because the error changed, and I actually find it clearer. Differential Revision: https://phab.mercurial-scm.org/D10414

lfs: use the new APIs The part of the lfs store that requires a destination cannot operate on multiple destination (yet). So we move them to the dedicated APIs. Differential Revision: https://phab.mercurial-scm.org/D10413

fastannotate: use `get_unique_pull_path` The code does not support multiple destination yet, so lets move it to the dedicated API. Differential Revision: https://phab.mercurial-scm.org/D10412

remotefilelog: use the right expandpath in to expand `~` Given the context where path is most likely a file system path, I suspect that the initial author confused ui.expandpath and util.expandpath (a proxy for os.path.expanduser and co) Differential Revision: https://phab.mercurial-scm.org/D10411

transplant: use `get_unique_pull_path` The command does not support multiple destination (yet). Differential Revision: https://phab.mercurial-scm.org/D10410

infinitepush: use `get_unique_pull_path` The extension does not support multiple destination (yet). Differential Revision: https://phab.mercurial-scm.org/D10456

relink: use `get_unique_pull_path` The code does not support multiple destination and will probably never do so. So lets use the dedicated API for this case. Differential Revision: https://phab.mercurial-scm.org/D10409

histedit: use `get_unique_push_path` Same as for `pathbomb`, this code does not support multiple destinations yet. It might in the future, probably with code put in common with patchwork, but for now lets use the dedicated API. Differential Revision: https://phab.mercurial-scm.org/D10408

urlutil: add a new `get_unique_push_path` This function is dedicated to call that needs a single destination. Currently most caller actually need that since few actually support multiple destinations (the most importants `hg push` and `hg outgoing` do). So having a clear API point for that will be important when the time comes to have a single `[paths]` alias resolving to multiple urls. Differential Revision: https://phab.mercurial-scm.org/D10407

fetch: use `get_unique_pull_path` to retrieve the path The fetch command does not support multiple destination, so we use the new dedicated API for that. Differential Revision: https://phab.mercurial-scm.org/D10406

patchbomb: use `get_unique_pull_path` The patchbomb code does not support multiple destinations yet. It would not be too hard to implemented since `hg outgoing` does. However that is a bit too much of a detour right now. In addition that kind of "select outgoing changesets for processing" code exist for various commands so the best would be to factor it out later. Let us use the dedicated API until this is solved. This comes with a small output change in the test that I find more accurate. So I kept it. Differential Revision: https://phab.mercurial-scm.org/D10405

narrow: use `get_unique_pull_path` The narrow's `tracked` command does not support multiple destination, lets use the dedicated API then. Differential Revision: https://phab.mercurial-scm.org/D10404

urlutil: add a new `get_unique_pull_path` This function is dedicated to call that needs a single destination. Currently most caller actually need that since few actually support multiple destinations (the most importants `hg push` and `hg outgoing` do). So having a clear API point for that will be important when the time comes to have a single `[paths]` alias resolving to multiple urls. Differential Revision: https://phab.mercurial-scm.org/D10403

mq: use the new `get_clone_path` to get the remote url We stop using `ui.expandpath` to use a function with a more defined semantic. This will help us to make it an option to point [paths] entry to multiple destination. Differential Revision: https://phab.mercurial-scm.org/D10402

urlutil: add a `get_clone_path` function We add a new function with a semantic focussed on `clone` operation (so without an existing repository). I am not certain the return type is the best, but this is what we need for now. Once all caller are migrated we might start thinking about that the API should be. For now that will do. Differential Revision: https://phab.mercurial-scm.org/D10401