bundle2: remove restriction around sidedata We are now capable of generating the missing sidedata on-the-fly. Differential Revision: https://phab.mercurial-scm.org/D10345

heptapod-ci: enable pytype checking The trigger is manual for now, until we get a better idea about stability, resource usage, etc. Differential Revision: https://phab.mercurial-scm.org/D10462

hghave: fix the definition of `python3` to work on Windows Both py2 and py3 executables are named `python.exe`, and may or may not be on PATH. So use the dispatcher executable that comes with py3 to fetch the version of the latest py3 executable. This allows at least one relnotes test to run on Windows. Differential Revision: https://phab.mercurial-scm.org/D10694

util: avoid echoing the password to the console on Windows py3 (issue6446) The `getpass.getpass()` implementation on Windows first checks if `sys.stdin` and `sys.__stdin__` are the same object. It's not on py3 because the former is replaced in dispatch.py with something that doesn't normalize '\n' to '\r\n'. When they aren't the same object, it simply calls `sys.stdin.readline()` instead of the mscvrt functions that read the input characters before they are echoed. This simply copies the `getpass.win_getpass()` implementation without the stdin check, and byteifies around the edges. I'm not sure if there's a reasonable replacement for the check that we could implement. When echoing input into the hg command, the `ui.interactive()` check causes `ui.getpass()` to bail before getting here. If the proper config switches are used to bypass that and call this, the process stalls until '\n' is input into the console. So there could be a deadlock here when run by another command if the wrong config settings are applied. Differential Revision: https://phab.mercurial-scm.org/D10708

sidedata: enable sidedata computers to optionally rewrite flags Sidedata computers may want to influence the flags of the revision they touch. For example, the computer for changelog-based copytracing can add a flag to signify that this revision might affect copytracing, inversely removing said flag if the information is no longer applicable. See inline documentation in `storageutil` for more details. Differential Revision: https://phab.mercurial-scm.org/D10344

cg4: introduce protocol flag to signify the presence of sidedata We need a way of signaling whether the current revision has sidedata or not, and re-using the revision flags would waste potential revlog flags and mix two normally independent layers. In this change, we add a single byte at the start of the ch4 delta header to set potential protocol flags. We also reclaim the revlog flag for sidedata, since it is no longer used, in its place now lives the (also experimental) copytracing flag. When generating deltas, apply the `CG_FLAG_SIDEDATA` flag if there is sidedata. When applying the deltas, if said flag is present, the next chunk contains the sidedata. Differential Revision: https://phab.mercurial-scm.org/D10343

changegroup: don't limit cgv4 to revlogv2 repos To help the transition from revlogv1 to revlogv2, we need to be able to enable cgv4 for revlogv1 repos, so that revlogv2 clients can handle adding/removing sidedata over the wire. Differential Revision: https://phab.mercurial-scm.org/D10342

sidedata: gate sidedata functionality to revlogv2 in more places Since revlogv1 is not capable of storing sidedata, we prevent sidedata mechanisms around the revlog layer from doing anything. We however keep the ones that allow a revlogv1 repo to generate sidedata on-the-fly on a push, the pull case simply does not add the sidedata to the revlog. Differential Revision: https://phab.mercurial-scm.org/D10341

sidedata: register copies sidedata computer regardless of the revlog version Repositories should not gate their sidedata computers based on any requirement, only their wanted sidedata. A repository might need to generate sidedata wanted by the peer that it itself does not want. Differential Revision: https://phab.mercurial-scm.org/D10340

revlog: replace the old `revlog_kind` approach with the new `target` one The new `target` attribute supersedes the previous one. Differential Revision: https://phab.mercurial-scm.org/D10353

revlog: introduce an explicit tracking of what the revlog is about Since the dawn of time, people have been forced to rely to lossy introspection of the index filename to determine what the purpose and role of the revlog they encounter is. This is hacky, error prone, inflexible, abstraction-leaky, <insert-your-own-complaints-here>. In f63299ee7e4d Raphaël introduced a new attribute to track this information: `revlog_kind`. However it is initialized in an odd place and various instances end up not having it set. In addition is only tracking some of the information we end up having to introspect in various pieces of code. So we add a new attribute that holds more data and is more strictly enforced. This work is done in collaboration with Raphaël. The `revlog_kind` one will be removed/adapted in the next changeset. We expect to be able to clean up various existing piece of code and to simplify coming work around the newer revlog format. Differential Revision: https://phab.mercurial-scm.org/D10352

config: add --source option to include source of value Showing the source of each config option is quite useful and not something the user should have to reach for the `--debug` flag for. I updates documentation and tests, except for one place in `test-hgrc.t` where I thought the test might have been intended to also test that `--debug` results in `ui.quiet` etc being test. Differential Revision: https://phab.mercurial-scm.org/D10668

rewriteutil: say how many commits would become orphan if commit is rewritten This copies the message from the evolve extension, but modifies it a bit to work with the grammar (in particular with the use of "change branch of" as `action`). I don't know why it doesn't use the same `_formatrevs()` as for public commmits. Differential Revision: https://phab.mercurial-scm.org/D10671

rewriteutil: give examples of public changesets that can't be rewritten This patch copies the feature from the evolve extension. Differential Revision: https://phab.mercurial-scm.org/D10670

convert: update p4 for Python 3 Differential Revision: https://phab.mercurial-scm.org/D10703

tests: change the fixer commands to use the buffer attribute on stdio objects Otherwise `\r` was getting injected into the fixed lines and throwing off the commit hashes on Windows when the fixer is invoked with py3. Differential Revision: https://phab.mercurial-scm.org/D10637

tests: stabilize test-persistent-nodemap.t on Windows Several issues here: - Hooks can't invoke shell scripts on Windows, so use `sh` to launch - `dd` in MSYS only recognizes `status=noxfer` - The `PATH` updating triggered a massive slowdown, but is no longer needed I have no idea why, but removing the `PATH` update substantially increased the speed of the test. It was running finishing at ~4:30 with `--debug` and ~14:50 without it, but now completes in ~2:20 on my Windows laptop. Differential Revision: https://phab.mercurial-scm.org/D10636

tests: invoke some shell scripts through the shell interpreter for Windows Otherwise, Windows was prompting what program to use to open the file (or just opening it if there was a file association configured). Differential Revision: https://phab.mercurial-scm.org/D10635

tests: run python script through quoted interpreter instead of directly This helps Windows when python is installed to %PROGRAMFILES%. Differential Revision: https://phab.mercurial-scm.org/D10634

tests: ensure `$PYTHON` is quoted for Windows Global installs of python3 go into "Program Files", and tons of tests fail with mysterious errors if this isn't quoted. Most of this is a followup to 0826d684a1b5, but a some of these were existing issues. Shebang lines are ignored because quoting breaks direct execution- these will need to be launched indirectly with the quoted `$PYTHON` command. Differential Revision: https://phab.mercurial-scm.org/D10633

black: make codebase compatible with black v21.4b2 and v20.8b1 I don't know what exact version of black made it care about these whitespace differences, but this is the version I got when I just installed it with `pip3 install black`. I'm intentionally not increasing the version of black required, as I don't want to force everyone to upgrade their version of black, and these fixes are backwards compatible with black v20.8b1. If there are more issues in the future and this becomes a maintenance burden I may do so in a future change. Tested with both versions of black (I got the older version via `pip3 install black==20.8b1`) Differential Revision: https://phab.mercurial-scm.org/D10539

debugcommands: fix some plural-agreements I noticed Differential Revision: https://phab.mercurial-scm.org/D10669

merge: with stable

Added signature for changeset 067f2c53fb24

Added tag 5.8 for changeset 067f2c53fb24

branching: merge stable into default

phabricator: adapt to the new `urlutil.url()` API This avoids a bunch of deprecation warnings in the tests. Differential Revision: https://phab.mercurial-scm.org/D10541

extensions: ignore exceptions from an extension's `getversion()` method This method is usually called when there's a stacktrace being generated, or with `hg version -v`. Raising another exception risks mangling the bug report info. I hit this issue when trying to add the method to the keyring extension to report the version of the extension and the underlying module, and ran into demandimport issues prior to py3.8. It seems like a wise thing to do anyway, though unfortunately there's no convenient `ui` object around to issue a warning. Use 'unknown' to signal that it tried to report a version and failed, unlike the default case of printing nothing. Differential Revision: https://phab.mercurial-scm.org/D10540

git: ensure all dirstate state values are bytes I'm not sure how this particular git status occurs, but after the fallout of issue 6510 and getting into the issue 6511 state where `git status` shows the files as modified in both the "to be committed" and "not staged" lists, `hg diff` was crashing in `workingctx.__contains__()`. Differential Revision: https://phab.mercurial-scm.org/D10532

tests: synchronize the git and Mercurial username The problem with the default name of "test" set by the test runner is the stringutil methods are unable to split out separate user and email addresses that git wants. This means the username is recorded in git as "test <test>". Amending a commit with that user ends up trying to use "<test>" as the person field for the new commit, and the git library complains about the angle brackets. We should probably abort with a clearer message any time this bad form is used with the git extension. One of the commit dates is tweaked to recreate the ambiguous hash prefix from before. Differential Revision: https://phab.mercurial-scm.org/D10531

git: initialize `extra` to have at least the branch name for nullid Otherwise, this crashes trying to convert to local encoding: ... File "/mnt/c/Users/Matt/hg/mercurial/logcmdutil.py", line 333, in _show branch = ctx.branch() File "/mnt/c/Users/Matt/hg/mercurial/context.py", line 675, in branch return encoding.tolocal(self._changeset.extra.get(b"branch")) File "/mnt/c/Users/Matt/hg/mercurial/encoding.py", line 181, in tolocal if isasciistr(s): TypeError: a bytes-like object is required, not 'NoneType' This was originally reported to the thg bug tracker. https://foss.heptapod.net/mercurial/tortoisehg/thg/-/issues/5629 Differential Revision: https://phab.mercurial-scm.org/D10528

git: consistently use str for parents when rebuilding the index database The tests show no changes, but when these values are overwritten shortly after when a git commit object is available, that uses str. It seems better to use that for consistency. It does materially affect the database though, because the old value stored was `X'3030..3030'` and is now '00..00' when the changelog table is dumped in sqlite3. There is one query that specifies the parents, but it passes the non null hashes as str, so it worked as expected. That likely explains the lack of test changes. Differential Revision: https://phab.mercurial-scm.org/D10527

git: pass the correct type to the sqlite3 query for baselog.hasnode() It looks like this function is mostly used in exchange (which isn't supported now) and histedit (which crashes). But I did verify in a REPL that passing bytes to `db.execute()` fails to find the row, whereas passing str works. Differential Revision: https://phab.mercurial-scm.org/D10526

git: use the correct type for stopping changelog.revs() The `tip` function returns a binary node, but the database is expecting an int. Differential Revision: https://phab.mercurial-scm.org/D10525

git: use the proper filenode for deleted files in changelog.changelogrevision Inline printing shows it still doesn't find the removed file in the test, but it is at least a str instead of bytes like the immediate query before it that does find files. Looking at the database, it doesn't look like the remove was recorded. Additionally, `hg log -r 'removes("re:.*")'` stacktraces. Differential Revision: https://phab.mercurial-scm.org/D10524

tests: add coverage for git.changelog.headrevs(...) Differential Revision: https://phab.mercurial-scm.org/D10523

git: make changelog.tiprev() return int instead of tuple (issue6510) Differential Revision: https://phab.mercurial-scm.org/D10522

git: fix partial node matching Looks like everything is meant to be a str here. Differential Revision: https://phab.mercurial-scm.org/D10521

tests: stabilize test-git-interop.t for some versions of git Not sure why this differs on my system, but it's likely due to running 2.17.1 on Ubuntu 18.04. Differential Revision: https://phab.mercurial-scm.org/D10520

manifests: push down expected node length into the parser This strictly enforces the node length in the manifest lines according to what the repository expects. One test case moves large hash testing into the non-treemanifest part as treemanifests don't provide an interface for overriding just the node length for now. Differential Revision: https://phab.mercurial-scm.org/D10533

core: don't hard-code node length Differential Revision: https://phab.mercurial-scm.org/D10536

core: don't hard-code hex node lengths Differential Revision: https://phab.mercurial-scm.org/D10535

tests: bump default timeout to 360s A number of tests hit or almost hit the default limit even on modern hardware. While the tests are ideally split into smaller pieces, that's non-trivial work. HyperThreading and similar technologies can trigger this often, even without any other load on the machine. Differential Revision: https://phab.mercurial-scm.org/D10534

enforcesinglehead-test: add the expected node output next to the error this make it simpler to validate that the test is correct. Differential Revision: https://phab.mercurial-scm.org/D10545

revlog: code for `revlogv0` in its own module This code is mostly unused compatiblity code. Yet it take a prohiminent place in the `revlog.py` module. That module is already quite big, so we move all that code in a dedicated module. Differential Revision: https://phab.mercurial-scm.org/D10511

revlog: have an explicit "pack_header" method Having to pass the version header when retrieving the binary version of every single entry is a bit silly. So we extract that special logic in its own method. This also prepare the move to newer revlog format, not storing the header within an actual entry… Differential Revision: https://phab.mercurial-scm.org/D10510

revlog: remove the revlogio class The class only contains a single `parseindex` method. Lets just make it a function and remove the `revlogio` class. It served us well but became thinner and thinner overtime. Differential Revision: https://phab.mercurial-scm.org/D10509

revlog: fix some comment style They displease check-code. Differential Revision: https://phab.mercurial-scm.org/D10542

revlog: add a `entry_binary` method on index The revlog index is already responsible for unpacking the binary entry, it would be simpler to make it responsible for packing them. In practice the C version of the index is already doing this internally. We introduce a "entry_binary" method that return the binary version of an existing revision. The method currently need to also take the revlog header to deal with the "first revision" special case. We will introduce further refactor in a later changeset to split that logic out. Differential Revision: https://phab.mercurial-scm.org/D10508

template: make an explicit closure for formatting entry in peerurls This is about to be become significantly more complicated as `ui.path[x]` will become a list. Differential Revision: https://phab.mercurial-scm.org/D10443

template: use `list_paths` in `peerurls` Using common code will make it simpler to update the logic behind the path definition and storage. Differential Revision: https://phab.mercurial-scm.org/D10442

paths: use `list_paths` in `hg paths` Using common code will make it simpler to update the logic behind the path definition and storage. Differential Revision: https://phab.mercurial-scm.org/D10441

urlutil: introduce a new `list_paths` function This function will be useful for command and template that wants to display path related information. Differential Revision: https://phab.mercurial-scm.org/D10440

urlutil: deprecate `getpath` There as no remaining user of that function. Differential Revision: https://phab.mercurial-scm.org/D10439

urlutil: inline the relevant part of `getpath` in `get_push_paths` The part that `get_push_paths` needs is quite simple, inclining will help us to deprecated `getpath`. Differential Revision: https://phab.mercurial-scm.org/D10438

url_util: introduce a `try_path` function That function… try a build a path, returning None on failure. This helps us to simplify various part of the existing code. Differential Revision: https://phab.mercurial-scm.org/D10437

narrow: add more status messages when narrowing Each of the steps I added status messages for in this patch frequently take minutes or tens of minutes for our internal users. It would be nice to also have a progress bar but that will have to come later. Differential Revision: https://phab.mercurial-scm.org/D10503

narrow: add progress-reporting when looking for local changes in `hg tracked` Looking for local changes (changes not on the given remote) can take a long time, so we should have progress-reporting for it. Differential Revision: https://phab.mercurial-scm.org/D10501

chg: pass --no-profile to disable profiling when starting hg serve If profiling is enabled via global/user config (as far as I can tell, this doesn't affect use of the --profile flag, but it probably does affect --config profiling.enabled=1), then the profiling data can be *cumulative* for the lifetime of the chg process. This leads to some "interesting" results where hg claims the walltime is something like 200s on a command that took only a second or two to run. Worse, however, is that with at least some profilers (such as the default "stat" profiler), this can cause a large slowdown while generating the profiler output. Differential Revision: https://phab.mercurial-scm.org/D10470

profiling: add --no-profile to disable profiling enabled via config Differential Revision: https://phab.mercurial-scm.org/D10469

tests: fix test-chg to ignore a warning about being unable to set locale This is apparently coming from bash when bash is providing the sh that we're using to execute the .sh file generated by run-tests for this test. Bash on my machine: ``` $ sh --version GNU bash, version 5.1.4(1)-release (x86_64-pc-linux-gnu) Copyright (C) 2020 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software; you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. $ sh sh-5.1$ LC_CTYPE=unsupported_value echo hi sh: warning: setlocale: LC_CTYPE: cannot change locale (unsupported_value): No such file or directory hi ``` Differential Revision: https://phab.mercurial-scm.org/D10468

branch: delete obsolete message about changing branch of obsolete commit We now rely on `rewriteutil.precheck()` to check for divergence, so we don't need the extra check in `cmdutil.changebranch()`. The former check is a little less strict in that it allows you to rewrite a commit without non-obsolete successors. Differential Revision: https://phab.mercurial-scm.org/D10519

rewriteutil: check for divergence This code is adapted from the code in the evolve extension. It seems to be equivalent as far as the evolve extension's test suite can tell (the only impact when making their `precheck()` delegate to our version is that error messages are less detailed). I had to change the error message to work with "change branch of" being inserted as the action. Differential Revision: https://phab.mercurial-scm.org/D10518

rewriteutil: adapt "cannot %s while merging" to work with "change branch of" `rewriteutil.precheck()` creates error messages by inserting a given verb into a sentence. The `hg branch -r` command passes in "change branch of" as the verb. That doesn't work well with "cannot %s while merging" (making it "cannot change branch of while merging"). Let's insert a "changeset" there to make it work better. Building sentences like this seems obviously bad for i18n, but fixing that is out of scope for this series, IMO. Differential Revision: https://phab.mercurial-scm.org/D10530

rewriteutil: point to help about instability when rewriting creates orphan This replicates the message from the evolve extension. Differential Revision: https://phab.mercurial-scm.org/D10517

help: add topic about evolution, based on text from evolve extension I've taken the text produced by `hg help evolution` when the evolve extension is enabled and made that available by the same command with just hg core. Changes I've made: * Added "(EXPERIMENTAL)" to the title. (That doesn't hide the topic from `hg help`, though.) * Replaced old-style `experimental.evolution=<names>` config by new-style `experimental.evolution.<name>=true`. * Replaces a "obsolete markers" by "obsolescence markers". * Removed most content from "Current feature status". When the evolve extension is enabled, its help text takes precedence. Differential Revision: https://phab.mercurial-scm.org/D10516

rewriteutil: add devel warning if precheck is called with contexts I'm trying to upstream parts from the evolve extension. This check exists there. Differential Revision: https://phab.mercurial-scm.org/D10515

rewriteutil: replace "null changeset" by "the null revision" in error message The evolve extension uses "the null revision" and we seem to use that term much more frequently in core too. Differential Revision: https://phab.mercurial-scm.org/D10514

relnotes: copy contents of "next" to "5.8" and clear "next" The same procedure as every year^Wcycle. Unlike previous cycles, I haven't actually recorded the copy because we don't want to propagate further changes in "next" (from the stable branch) to "5.8". Differential Revision: https://phab.mercurial-scm.org/D10500

dirstateguard: use mktemp-like functionality to generate the backup filenames Previously these were generated with names like: `dirstate.backup.commit.<memory address of dirstateguard>` This could cause problems if two hg commands ran at the same time that used the same memory address, (which is apparently not uncommon if chg is involved), as memory addresses are not unique across processes. This issue was reported in the post-review comments on http://phab.mercurial-scm.org/D9952. Differential Revision: https://phab.mercurial-scm.org/D10504

node: replace nullid and friends with nodeconstants class The introduction of 256bit hashes require changes to nullid and other constant magic values. Start pushing them down from repository and revlog where sensible. Differential Revision: https://phab.mercurial-scm.org/D9465

repoview: separate concerns in _filteredrepotypes comment The cited issue in Python bugtracker is closed, but hasn't been fixed. We've been able to use the attached example and reproduce it with Python 3.9. The point where it turns from needless stress on the GC to the an actual leak is when one factors in the fact that the GC was before Python 3.4 unable to collect some types (see PEP 442). Note that even with Python 2.7, the simple example of cycles due to __mro__ are collectable. This was seen again with the example attached on the CPython issue.

repoview: fix memory leak of filtered repo classes The leak occurs in long-running server processes with extensions, and is measured at 110kB per request. Before this change, the contents of the `_filteredrepotypes` cache are not properly garbage collected, despite it begin a `WeakKeyDictionary`. Extensions have a tendency to generate a new repository class for each `localrepo` instantiation. Server processes based on `hgwebdir_mod` will instantiate a new `localrepo` for each HTTP request that involves a repository. As a result, with a testing process that repeatedly opens a repository with several extensions activated (`topic` notably among them), we see a steady increase in resident memory of 110kB per repository instantiation before this change. This is also true, if we call `gc.collect()` at each instantiation, like `hgwebdir_mod` does, or not. The cause of the leak is that the *values* aren't weak references. This change uses `weakref.ref` for the values, and this makes in our measurements the resident size increase drop to 5kB per repository instantiation, with no explicit call of `gc.collect()` at all. There is currently no reason to believe that this remaining leak of 5kB is related to or even due to Mercurial core. We've also seen evidence that `ui.ui` instances weren't properly garbage collected before the change (with the change, they are). This could explain why the figures are relatively high. In theory, the collection of weak references could lead to much more misses in the cache, so we measured the impact on the original case that was motivation for introducing that cache in 7e89bd0cfb86 (see also issue5043): `hg convert` of the mozilla-central repository. The bad news here is that there is a major memory leak there, both with and without the present changeset. There were no more cache misses, and we could see no more memory leak with this change: the resident size after importing roughly 100000 changesets was at 12.4GB before, and 12.5GB after. The small increase is mentioned for completeness only, and we believe that it should be ignored, at least as long as the main leak isn't fixed. At less than 1% of the main leak, even finding out whether it is merely noise would be wasteful. Original context where this was spotted and first mitigated: https://foss.heptapod.net/heptapod/heptapod/-/issues/466 The leak reduction was also obtained in Heptapod inner HTTP server, which amounts to the same as `hgwebdir_mod` for these questions. The measurements done with Python 3.9, similar figures seen with 3.8. More work on our side would be needed to give measurements with 2.7, because of testing server process does not support it.

repoview: style change in newtype() cache handling This way of writing it does not change the logic at all, but is more fit for the change we want to make in the next changeset. If anything, that's one dict lookup less in the hot path, but that should be non measurable.

tests: don't hard-code /bin/bash From Thomas Klausner in pkgsrc. Differential Revision: https://phab.mercurial-scm.org/D10507

lint: fix failing code check 8d2b62d716b0 introduced code that `test-check-code.t` wasn't happy about, this fixes it. Differential Revision: https://phab.mercurial-scm.org/D10506

typing: enable pytype processing on mercurial/upgrade_utils/actions.py This was the suggested workaround for the pytype bug that caused this file to be processed forever. Differential Revision: https://phab.mercurial-scm.org/D10460

incoming: use bytes for an error message Caught by pytype. Differential Revision: https://phab.mercurial-scm.org/D10461

merge: stable heads I forgot to pull before rolling rc1, so we just have a couple of patches that missed the rc1 train. Mea culpa.

Added signature for changeset 8d2b62d716b0

Added tag 5.8rc1 for changeset 8d2b62d716b0

relnote: mention a nodemap bug fixes The fix was in a3720569a43f. Differential Revision: https://phab.mercurial-scm.org/D10499

relnote: document the use of persistent-nodemap by default for Rust build Differential Revision: https://phab.mercurial-scm.org/D10498

relnode: document the move to `zstd` as default compression Differential Revision: https://phab.mercurial-scm.org/D10497

relnote: clarify the backward compatibility entry about p1/p2 swap The previous message might read a bit scary. So we clarify that the nodeid are not affected. Differential Revision: https://phab.mercurial-scm.org/D10496

mail: add a TODO about proper mbox locking Per @nbjoerg in IRC, the mbox isn't locked correctly. Differential Revision: https://phab.mercurial-scm.org/D10505

packaging: extract pre-computed version when running from plain tarball The tarballs created by setup.py are not including the usual archive markers that `hg archive` leaves, so the rewrite of the version number computation actually left the empty version string around. This meant that installations from PyPI would use 0.0.0 as version string. Differential Revision: https://phab.mercurial-scm.org/D10502

Added signature for changeset f67b8946bb1b

Added tag 5.8rc0 for changeset f67b8946bb1b

merge: default into stable for 5.8 rc

mail: force SMTPException to bytes before wrapping in error.Abort Pytype wasn't complaining about this for some reason, but PyCharm was and we already handle another instance of this exception the same way in this function. Differential Revision: https://phab.mercurial-scm.org/D10473

mail: split out the SMTP login to allow the keyring extension to wrap it The keyring extension only needs to tweak this tiny section of the larger function. But without any place to intercept the username/password fetching, it copy/pasted the entire function, and has grown a bunch of compatibility hacks to support older versions of Mercurial as well. Differential Revision: https://phab.mercurial-scm.org/D10471

store: exclude `undo.` nodemap's file from `walk` There are "temporary" local file that we should not be transfered by `walk` user like local clone and stream clone. This fix the small issue that the new tests highlighted. Differential Revision: https://phab.mercurial-scm.org/D10482

streamclone: treat volatile file as "fullfile" The nodemap related file might change (or get deleted) during the stream clone in a way incompatible with the streaming process. So we introduce a new flag for this type of file and integrate it with the existing `revlog_type` field recently added to `store.walk` returns. We use that new flat to dispatch such file to the existing mechanism for "atomic replacement" file for the nodemap docket and datafile. This fix the bugs we have been adding tests for. Strictly speaking, the nodemap datafile is happened only a could maybe be used in a slightly more efficient way, however this is good enough for now. Differential Revision: https://phab.mercurial-scm.org/D10481

nodemap: add a test about nodemap "vacuum" during stream clone When the nodemap "vacuum" it generate a new datafile, with a new unique name and delete the old one. This confuse the stream clone code and create the same kind of crash we were seeing in the previous patch. We build a test case where this happens. The next changeset will contains a fix We can also notice that we are, wrongfully exchanging `undo.*` files that the client will not be able to use. This will be dealt with soon. Differential Revision: https://phab.mercurial-scm.org/D10480

nodemap: add a test about racy commit during stream clone That test show that the resulting client nodemap is different from the server one. This happens because the server one transferred a corrupted node map. The data file match the pre-commit content while the docket has post commit content. As the result the nodemap was detected invalid and recomputed. When running without the rust implementation, the code is also generating a new datafile unconditionally, This mean the older file is no longer there are transfer time, resulting in a crash. We will fix this issue later, but we start with writing tests highlighting the issue. Differential Revision: https://phab.mercurial-scm.org/D10479

streamclone: remove sleep based "synchronisation" in tests Sleep based test synchronisation does not work. Variation in machine performance and load can make the two process miss their windows. Instead we migrate to explicit signaling through the file system as other tests file are using. Differential Revision: https://phab.mercurial-scm.org/D10478

testing: add a `write_file` function The function is kinda trivial, but having a simple function avoid silly mistake. Differential Revision: https://phab.mercurial-scm.org/D10477

testing: add a utility function to wait for file create This is similar to `tests/testlib/wait-on-file`, but for the python code Differential Revision: https://phab.mercurial-scm.org/D10476

streamclone: check the errors log in tests Code can be buggy, in that case having error message tend to help. Differential Revision: https://phab.mercurial-scm.org/D10475

mail: use a context manager when writing to mbox Differential Revision: https://phab.mercurial-scm.org/D10484

errors: remove unnecessary varargs handling from OutOfBandError In my recent D10465, I moved some code over from scmutil into `OutOfBandError.__init__`. The code was written to deal with an arbitrary number of `message` arguments to the constructor. It turns out that we only ever pass 0 or 1. Given that, let's simplify it. Differential Revision: https://phab.mercurial-scm.org/D10483

phab-refresh: use a special comment on the stable branch Phabricator make it hard to spot the branch information. As a result patch intended for stable are consistently queued for default. We add a special command and flashy GIF to try to attract reviewer attention on the fact patch are intended for stable. Differential Revision: https://phab.mercurial-scm.org/D10464

phab-refresh: extract the comment in a variable This make the script more readable and it is about to become more complex. Differential Revision: https://phab.mercurial-scm.org/D10463

tests: make test-archive.t pass on py3.9 (issue6504) Something got stricter at parsing URL query parameters and now the parameters need to be separated by "&"; ";" is no longer allowed. See issue6504 for details. Differential Revision: https://phab.mercurial-scm.org/D10472

errors: raise RemoteError in some places in exchange.py This is part of https://www.mercurial-scm.org/wiki/ErrorCategoriesPlan Differential Revision: https://phab.mercurial-scm.org/D10467

errors: introduce a class for remote errors Having an exception for remote errors makes it much easier to exit with the right detailed exit code. Differential Revision: https://phab.mercurial-scm.org/D10466

errors: make OutOfBandError extend Abort I'm about to create a new `RemoteError` exception and make `OutOfBandError` extend it. This patch prepares for that. Differential Revision: https://phab.mercurial-scm.org/D10465

revlog: replace revlog._io.size with a new revlog.index.entry_size The `revlogio` class is mostly a relic from the past. Once in charge of the full revlog related Input/Output code, that class gradually lost responsibilities to the point where more IO are now done by `revlog.index` objects or revlog objects themself. I would like to ultimately remove the `revlogio` class, to do so I start simple with move the "entry size" information on the index. (The index is already responsible of the binary unpacking, so it knows the size. Differential Revision: https://phab.mercurial-scm.org/D10309

perf-discovery: use `get_unique_pull_path` This performance command now use the new API, unless the benchmarked Mercurial is older. Differential Revision: https://phab.mercurial-scm.org/D10423

help: clarify when `$HG_URL` and `$HG_SOURCE` are added in hooks Differential Revision: https://phab.mercurial-scm.org/D10400

nodemap: deal with data mmap error If the file is too small, the mmapread call would raise a ValueError. We catch that and ignore nodemap content (as we do without mmap). This make the repository slightly slower (until the next write) but usable. Unlike the current crash. Differential Revision: https://phab.mercurial-scm.org/D10458

nodemap: test various corruption scenario for the persistent nodemap Corruption can happens in the wild, either because some of our code is buggy or because repository were shared/transfered in a strange manners. Currently one of the corruption case (not enough data) lead to a crash. We add explicit tests for theses cases. Differential Revision: https://phab.mercurial-scm.org/D10457

cmdutil: fix an uninitialize variable usage in clearunfinished() I happened to notice that PyCharm flagged this while looking for something else. I'm pretty sure it meant to check the current state in the iteration, and not keep reusing the last one in the previous iteration. Differential Revision: https://phab.mercurial-scm.org/D10459

help: document `$HG_NEW_OBSMARKERS` in pretxnclose hook Differential Revision: https://phab.mercurial-scm.org/D10399

help: prepend environment variables used in hooks with "$" Most of them have "$" prefix but some don't, so adding it for consistency. Differential Revision: https://phab.mercurial-scm.org/D10398

revset: drop an outdated comment There is not `ui.expandpath` call there anymore. Differential Revision: https://phab.mercurial-scm.org/D10435

ui: deprecated `ui.expandpath` This function return a single path, something incompatible with the coming change to `[paths]` definition to allow multiple associated urls. Now that all user got migrated, we deprecate the function. Differential Revision: https://phab.mercurial-scm.org/D10432

urlutil: remove usage of `ui.expandpath` in `get_clone_path` We want to deprecate `ui.expandpath` and simplify the code before adding more complexity in the form of `[paths]` entry pointing to multiple url. So we inline the relevant bits. Differential Revision: https://phab.mercurial-scm.org/D10431

urlutil: remove usage of `ui.expandpath` in `get_unique_pull_path` We want to deprecate `ui.expandpath` and simplify the code before adding more complexity in the form of `[paths]` entry pointing to multiple url. So we inline the relevant bits. Differential Revision: https://phab.mercurial-scm.org/D10430