node: replace nullid and friends with nodeconstants class The introduction of 256bit hashes require changes to nullid and other constant magic values. Start pushing them down from repository and revlog where sensible. Differential Revision: https://phab.mercurial-scm.org/D9465

repoview: separate concerns in _filteredrepotypes comment The cited issue in Python bugtracker is closed, but hasn't been fixed. We've been able to use the attached example and reproduce it with Python 3.9. The point where it turns from needless stress on the GC to the an actual leak is when one factors in the fact that the GC was before Python 3.4 unable to collect some types (see PEP 442). Note that even with Python 2.7, the simple example of cycles due to __mro__ are collectable. This was seen again with the example attached on the CPython issue.

repoview: fix memory leak of filtered repo classes The leak occurs in long-running server processes with extensions, and is measured at 110kB per request. Before this change, the contents of the `_filteredrepotypes` cache are not properly garbage collected, despite it begin a `WeakKeyDictionary`. Extensions have a tendency to generate a new repository class for each `localrepo` instantiation. Server processes based on `hgwebdir_mod` will instantiate a new `localrepo` for each HTTP request that involves a repository. As a result, with a testing process that repeatedly opens a repository with several extensions activated (`topic` notably among them), we see a steady increase in resident memory of 110kB per repository instantiation before this change. This is also true, if we call `gc.collect()` at each instantiation, like `hgwebdir_mod` does, or not. The cause of the leak is that the *values* aren't weak references. This change uses `weakref.ref` for the values, and this makes in our measurements the resident size increase drop to 5kB per repository instantiation, with no explicit call of `gc.collect()` at all. There is currently no reason to believe that this remaining leak of 5kB is related to or even due to Mercurial core. We've also seen evidence that `ui.ui` instances weren't properly garbage collected before the change (with the change, they are). This could explain why the figures are relatively high. In theory, the collection of weak references could lead to much more misses in the cache, so we measured the impact on the original case that was motivation for introducing that cache in 7e89bd0cfb86 (see also issue5043): `hg convert` of the mozilla-central repository. The bad news here is that there is a major memory leak there, both with and without the present changeset. There were no more cache misses, and we could see no more memory leak with this change: the resident size after importing roughly 100000 changesets was at 12.4GB before, and 12.5GB after. The small increase is mentioned for completeness only, and we believe that it should be ignored, at least as long as the main leak isn't fixed. At less than 1% of the main leak, even finding out whether it is merely noise would be wasteful. Original context where this was spotted and first mitigated: https://foss.heptapod.net/heptapod/heptapod/-/issues/466 The leak reduction was also obtained in Heptapod inner HTTP server, which amounts to the same as `hgwebdir_mod` for these questions. The measurements done with Python 3.9, similar figures seen with 3.8. More work on our side would be needed to give measurements with 2.7, because of testing server process does not support it.

repoview: style change in newtype() cache handling This way of writing it does not change the logic at all, but is more fit for the change we want to make in the next changeset. If anything, that's one dict lookup less in the hot path, but that should be non measurable.

tests: don't hard-code /bin/bash From Thomas Klausner in pkgsrc. Differential Revision: https://phab.mercurial-scm.org/D10507

lint: fix failing code check 8d2b62d716b0 introduced code that `test-check-code.t` wasn't happy about, this fixes it. Differential Revision: https://phab.mercurial-scm.org/D10506

typing: enable pytype processing on mercurial/upgrade_utils/actions.py This was the suggested workaround for the pytype bug that caused this file to be processed forever. Differential Revision: https://phab.mercurial-scm.org/D10460

incoming: use bytes for an error message Caught by pytype. Differential Revision: https://phab.mercurial-scm.org/D10461

merge: stable heads I forgot to pull before rolling rc1, so we just have a couple of patches that missed the rc1 train. Mea culpa.

Added signature for changeset 8d2b62d716b0

Added tag 5.8rc1 for changeset 8d2b62d716b0

relnote: mention a nodemap bug fixes The fix was in a3720569a43f. Differential Revision: https://phab.mercurial-scm.org/D10499

relnote: document the use of persistent-nodemap by default for Rust build Differential Revision: https://phab.mercurial-scm.org/D10498

relnode: document the move to `zstd` as default compression Differential Revision: https://phab.mercurial-scm.org/D10497

relnote: clarify the backward compatibility entry about p1/p2 swap The previous message might read a bit scary. So we clarify that the nodeid are not affected. Differential Revision: https://phab.mercurial-scm.org/D10496

mail: add a TODO about proper mbox locking Per @nbjoerg in IRC, the mbox isn't locked correctly. Differential Revision: https://phab.mercurial-scm.org/D10505

packaging: extract pre-computed version when running from plain tarball The tarballs created by setup.py are not including the usual archive markers that `hg archive` leaves, so the rewrite of the version number computation actually left the empty version string around. This meant that installations from PyPI would use 0.0.0 as version string. Differential Revision: https://phab.mercurial-scm.org/D10502

Added signature for changeset f67b8946bb1b

Added tag 5.8rc0 for changeset f67b8946bb1b

merge: default into stable for 5.8 rc

mail: force SMTPException to bytes before wrapping in error.Abort Pytype wasn't complaining about this for some reason, but PyCharm was and we already handle another instance of this exception the same way in this function. Differential Revision: https://phab.mercurial-scm.org/D10473

mail: split out the SMTP login to allow the keyring extension to wrap it The keyring extension only needs to tweak this tiny section of the larger function. But without any place to intercept the username/password fetching, it copy/pasted the entire function, and has grown a bunch of compatibility hacks to support older versions of Mercurial as well. Differential Revision: https://phab.mercurial-scm.org/D10471

store: exclude `undo.` nodemap's file from `walk` There are "temporary" local file that we should not be transfered by `walk` user like local clone and stream clone. This fix the small issue that the new tests highlighted. Differential Revision: https://phab.mercurial-scm.org/D10482

streamclone: treat volatile file as "fullfile" The nodemap related file might change (or get deleted) during the stream clone in a way incompatible with the streaming process. So we introduce a new flag for this type of file and integrate it with the existing `revlog_type` field recently added to `store.walk` returns. We use that new flat to dispatch such file to the existing mechanism for "atomic replacement" file for the nodemap docket and datafile. This fix the bugs we have been adding tests for. Strictly speaking, the nodemap datafile is happened only a could maybe be used in a slightly more efficient way, however this is good enough for now. Differential Revision: https://phab.mercurial-scm.org/D10481

nodemap: add a test about nodemap "vacuum" during stream clone When the nodemap "vacuum" it generate a new datafile, with a new unique name and delete the old one. This confuse the stream clone code and create the same kind of crash we were seeing in the previous patch. We build a test case where this happens. The next changeset will contains a fix We can also notice that we are, wrongfully exchanging `undo.*` files that the client will not be able to use. This will be dealt with soon. Differential Revision: https://phab.mercurial-scm.org/D10480

nodemap: add a test about racy commit during stream clone That test show that the resulting client nodemap is different from the server one. This happens because the server one transferred a corrupted node map. The data file match the pre-commit content while the docket has post commit content. As the result the nodemap was detected invalid and recomputed. When running without the rust implementation, the code is also generating a new datafile unconditionally, This mean the older file is no longer there are transfer time, resulting in a crash. We will fix this issue later, but we start with writing tests highlighting the issue. Differential Revision: https://phab.mercurial-scm.org/D10479

streamclone: remove sleep based "synchronisation" in tests Sleep based test synchronisation does not work. Variation in machine performance and load can make the two process miss their windows. Instead we migrate to explicit signaling through the file system as other tests file are using. Differential Revision: https://phab.mercurial-scm.org/D10478

testing: add a `write_file` function The function is kinda trivial, but having a simple function avoid silly mistake. Differential Revision: https://phab.mercurial-scm.org/D10477

testing: add a utility function to wait for file create This is similar to `tests/testlib/wait-on-file`, but for the python code Differential Revision: https://phab.mercurial-scm.org/D10476

streamclone: check the errors log in tests Code can be buggy, in that case having error message tend to help. Differential Revision: https://phab.mercurial-scm.org/D10475

mail: use a context manager when writing to mbox Differential Revision: https://phab.mercurial-scm.org/D10484

errors: remove unnecessary varargs handling from OutOfBandError In my recent D10465, I moved some code over from scmutil into `OutOfBandError.__init__`. The code was written to deal with an arbitrary number of `message` arguments to the constructor. It turns out that we only ever pass 0 or 1. Given that, let's simplify it. Differential Revision: https://phab.mercurial-scm.org/D10483