ci: add a "all" template to easily control "when" test run When debugging some job (usually windows one) it is handy to have a way to pass all other jobs to manual execution. We add a "root" template to control just that. Differential Revision: https://phab.mercurial-scm.org/D11276

Added signature for changeset 2813d406b036

Added tag 5.9rc1 for changeset 2813d406b036

issue6528: add a config option to control the fixing on the fly This will allow people who know to be safe to avoid any performance overhead (and other potential issue). Differential Revision: https://phab.mercurial-scm.org/D11271

issue6528: also filter delta on the fly when applying a changegroup This ensure that corrupted clone does not spread corruption to "fixed" version. This might come at a performance cost, we will had a config option to control this behavior in the next changesets. Differential Revision: https://phab.mercurial-scm.org/D11270

filelog: open the writing context a bit earlier in `addgroup` This is a small change made ahead of the next patch for clarification. Differential Revision: https://phab.mercurial-scm.org/D11269

issue6528: implement _is_revision_affected_fast using callback The delta comming from a bundle/stream does not exists in the revlog yet, so we will need other way to retrieve the same information. To prepare for this we split the function to use callbacks in the core logic. Differential Revision: https://phab.mercurial-scm.org/D11268

issue6528: implement _is_revision_affected using callback The delta comming from a bundle/stream does not exists in the revlog yet, so we will need other way to retrieve the same information. To prepare for this we split the function to use callbacks in the core logic. Differential Revision: https://phab.mercurial-scm.org/D11267

debugcommands: add a `--paranoid` option to `debug-repair-issue-6528` See justification inline. Differential Revision: https://phab.mercurial-scm.org/D11263

repair: improve performance of detection of revisions affected by issue6528 Explanations inside the patch. I've tested this on Mozilla-Central and it's 5 times faster than the naive approach on my laptop. Differential Revision: https://phab.mercurial-scm.org/D11262

debugcommands: introduce a debug command to repair repos affected by issue6528 This command is quite basic and slow, it will loop over the entirety of the filelogs in the repository and check each revision for corruption, then fixes the affected filelogs. It takes under 25 minutes for Mozilla-Central on my not-top-of-the-line laptop, using the `--to-report` and `--from-report` options will make this pretty tolerable to use, I think. This change also introduces a test for the fix. Differential Revision: https://phab.mercurial-scm.org/D11239

contrib: switch the Windows bootstrap environment to py3.9 Use the built in `venv` module instead of `virtualenv` for simplicity, and upgrade to a modern Mercurial that supports py3. One issue here is that `venv` doesn't copy `python3{,Y}.dll` into the `Scripts` subdirectory, so running the `hg.exe` that gets installed immediately fails on a clean system because Python isn't in `PATH`. There is code in `python.exe` to detect when it is in a venv and add the original python install to the DLL lookup path, which we don't do in `hg.exe` yet. The simple workaround for now is to run the `hg` script with `python.exe`. Typically `PYTHONLEGACYWINDOWSSTDIO` must be set in the environment on Windows, but the clone process works without it. Differential Revision: https://phab.mercurial-scm.org/D11275

contrib: comment out the 64-bit py38 dependency installation on Windows Not sure what is going on here, but it appears to not install py3.8 x64 in the usual `C:\hgdev` directory. The x32 installer works fine. I'm assuming this is a bug in this version of the installer, but didn't look into it too much. Differential Revision: https://phab.mercurial-scm.org/D11274

contrib: drop python2.7 from the Windows dependency install script While we aren't quite ready to drop py27 yet, the MS compiler is no longer available from MS (there is a copy on github if it's really needed), and that causes the script to fail. Differential Revision: https://phab.mercurial-scm.org/D11273

contrib: log the command and args for every process installing windows deps This is a little noisier, but makes it simple to debug when things fail. Differential Revision: https://phab.mercurial-scm.org/D11272

windows-ci: run Windows CI automatically alongside the others This will enable us to make Windows Python 3 a first-class citizen for the next 6.0 cycle. We will probably get some flaky tests and we're missing others that are skipped, but we'll turn them on it future patches. Differential Revision: https://phab.mercurial-scm.org/D11256

vfs: always use / as file separator (issue6546) Various part of vfs already enforce `/` usage and using `\` confuse the encoded vfs. So we simply use `/` all the time. Differential Revision: https://phab.mercurial-scm.org/D11260

subrepo: compare normalised vfs path Otherwise the realpath call can turn `/` into `\` on windows confusing the check. (We probably needs this in more location) Differential Revision: https://phab.mercurial-scm.org/D11259

pager: account for flakiness in Windows output This test case is cursed and probably not worth losing more time over. This makes apparent what the intended behavior is while still removing the flakiness from the CI. Differential Revision: https://phab.mercurial-scm.org/D11257

windows-ci: clean up the Heptapod CI file now that the baseline is solid Enough work has been done one the CI side, this now works with little effort on our side. The next patch will remove the manual switch. Differential Revision: https://phab.mercurial-scm.org/D11254

test-nointerrupt: make "sure" the handler "might" trigger (issue6558) We are sure that the signal got sent in the right time frame, however, we still have race, so either the code is actually buggy or we need some security to make sure the signal get processed. We might be affected by https://bugs.python.org/issue43406 ? Differential Revision: https://phab.mercurial-scm.org/D11251

testing: make sure write_file is "atomic" This make sure viewer cannot see the new file with partial content. This was likely the cause of some flakiness in `test-nointerrupt.t` Differential Revision: https://phab.mercurial-scm.org/D11250

test: disable test-subrepo-git.t in python2 + chg I am a couple of days in try to debug that at it seems minor enough with enough other priority to simply disable it for now. Differential Revision: https://phab.mercurial-scm.org/D11253

hgwebdir: avoid systematic full garbage collection Forcing a systematic full garbage collection upon each request can serioulsy harm performance. This is reported as https://bz.mercurial-scm.org/show_bug.cgi?id=6075 With this change we're performing the full collection according to a new setting, `experimental.web.full-garbage-collection-rate`. The default value is 1, which doesn't change the behavior and will allow us to test on real use cases. If the value is 0, no full garbage collection occurs. Regardless of the value of the setting, a partial garbage collection still occurs upon each request (not attempting to collect objects from the oldest generation). This should be enough to take care of reference cycles that have been created by the last request (assessment of this requires changing the setting, not to be 1). In my experience chasing memory leaks in Mercurial servers, the full collection never reclaimed any memory, but this is with Python 3 and biased towards small repositories. On the other hand, as explained in the Python developer docs [1], frequent full collections are very harmful in terms of performance if lots of objects survive the collection, and hence stay in the oldest generation. Note that `gc.collect()` is indeed trying to collect the oldest generation [2]. This happens usually in two cases: - unwanted lingering objects (i.e., an actual memory leak that the GC cannot do anything about). Sadly, we have lots of those these days. - desireable long-term objects, typically in caches (not inner caches carried by repositories, which should be collected with them). This is a subject of interest for the Heptapod project. In short, the flat rate that this change still permits is probably a bad idea in most cases, and the default value can be tweaked later on (or even be set to 0) according to experiments in the wild. The test is inspired from test-hgwebdir-paths.py [1] https://devguide.python.org/garbage_collector/#collecting-the-oldest-generation [2] https://docs.python.org/3/library/gc.html#gc.collect Differential Revision: https://phab.mercurial-scm.org/D11204

check-module-imports: ignore non-stdlib module installed by distribution Previously, the check script would detect breezy as part of the stdlib if installed using the debian package manager. This silence the following complains: tests/test-convert-bzr.t:117: imports not lexically sorted: breezy.bzr.bzrdir < sys tests/test-convert-bzr.t:117: stdlib import "breezy.bzr.bzrdir" follows local import: breezy tests/test-convert-bzr-ghosts.t:7: imports not lexically sorted: breezy.bzr.bzrdir < sys tests/test-convert-bzr-ghosts.t:7: stdlib import "breezy.bzr.bzrdir" follows local import: breezy tests/test-convert-bzr-treeroot.t:7: imports not lexically sorted: breezy.bzr.bzrdir < sys tests/test-convert-bzr-treeroot.t:7: stdlib import "breezy.bzr.bzrdir" follows local import: breezy Differential Revision: https://phab.mercurial-scm.org/D11249

remotefilelog: fix what looks like a wrong refactoring when various store functions started returning a revlog type as the first element of the tuple. Differential Revision: https://phab.mercurial-scm.org/D11243

rust-nodemap: falling back to C impl as mitigation This is a mitigation for https://bz.mercurial-scm.org/show_bug.cgi?id=6554 We see sometimes almost all data except the most recent revisions removed from the persistent nodemap, but we don't know how to reproduce yet. This has sadly repercussions beyond just needing to reconstruct the persistent nodemap: for instance, this automatically filters out all bookmarks pointing to revisions that the nodemap cannot resolve. If such filtering happens in a transaction, the update of the bookmarks file that happens at the end of transaction loses all bookmarks that have been affected. There may be similar consequences for other data. So this is a data loss, something that we have to prevent as soon as possible. As a mitigation measure, we will now fallback to the C implementation in case nodemap lookups failed. This will add some latency, e.g., in discovery, yet less than disabling the persistent nodemap entirely. We considered implementing the fallback directly on the Python side, but `revlog.get_rev()` is not systematically used, there are also several direct calls to the index method (`self.index.rev()` for a `revlog` instance). It is therefore more direct to implement the mitigation in the rust-cpython wrapper. Differential Revision: https://phab.mercurial-scm.org/D11238

obsolete: disable other evolution config options if createmarkers is off We used to raise an abort in this case, but recent changes to local clone command (377d8fc20e34) resulted in destrepo both caring about experimental.evolution config options and not initializing extensions. So imagine if you had evolve and allowdivergence enabled in your ~/.hgrc. Local clone stopped working after 377d8fc20e34 because evolve sets experimental.evolution=all, but only on srcrepo, for destrepo the extension is not initialized. It's possible to make local cloning work by initializing extensions for destrepo in some cases, but in other cases (e.g. allowdivergence in ~/.hgrc, evolve extension in original-repo/.hg/hgrc) it would still fail. In a discussion with Pierre-Yves David it was decided to simply force other evolution options to be false if createmarkers is not enabled. Differential Revision: https://phab.mercurial-scm.org/D11223

fix: use obsolete.isenabled() to check for experimental.allowdivergence Now that obsolete.isenabled() can also check if divergence is allowed, let's use it for consistency. Other experimental.evolution options are already checked via this function. Differential Revision: https://phab.mercurial-scm.org/D11222

rebase: use obsolete.isenabled() to check for experimental.allowdivergence Now that obsolete.isenabled() can also check if divergence is allowed, let's use it for consistency. Other experimental.evolution options are already checked via this function. Differential Revision: https://phab.mercurial-scm.org/D11221

obsolete: disable other evolution config options if createmarkers is off We used to raise an abort in this case, but recent changes to local clone command (377d8fc20e34) resulted in destrepo both caring about experimental.evolution config options and not initializing extensions. So imagine if you had evolve and allowdivergence enabled in your ~/.hgrc. Local clone stopped working after 377d8fc20e34 because evolve sets experimental.evolution=all, but only on srcrepo, for destrepo the extension is not initialized. It's possible to make local cloning work by initializing extensions for destrepo in some cases, but in other cases (e.g. allowdivergence in ~/.hgrc, evolve extension in original-repo/.hg/hgrc) it would still fail. In a discussion with Pierre-Yves David it was decided to simply force other evolution options to be false if createmarkers is not enabled. Differential Revision: https://phab.mercurial-scm.org/D11223

fix: use obsolete.isenabled() to check for experimental.allowdivergence Now that obsolete.isenabled() can also check if divergence is allowed, let's use it for consistency. Other experimental.evolution options are already checked via this function. Differential Revision: https://phab.mercurial-scm.org/D11222

rebase: use obsolete.isenabled() to check for experimental.allowdivergence Now that obsolete.isenabled() can also check if divergence is allowed, let's use it for consistency. Other experimental.evolution options are already checked via this function. Differential Revision: https://phab.mercurial-scm.org/D11221

heptapod-ci: enable pytype testing for all pipelines Now that pytype agrees with the codebase, let's run it by default to catch regressions. Differential Revision: https://phab.mercurial-scm.org/D11242

pytype: add assertions to explain revlogv2 invariants to Pytype Differential Revision: https://phab.mercurial-scm.org/D11241

typing: add several assertions to dirstatemap to appease pytype (grafted from default to stable) I think it's been mentioned in IRC that these can't be None in this case. This fixes: File "/mnt/c/Users/Matt/hg/mercurial/dirstatemap.py", line 213, in addfile: unsupported operand type(s) for &: 'None' and 'int' [unsupported-operands] No attribute '__and__' on None or '__rand__' on int Called from (traceback): line 290, in reset_state File "/mnt/c/Users/Matt/hg/mercurial/dirstatemap.py", line 214, in addfile: unsupported operand type(s) for &: 'None' and 'int' [unsupported-operands] No attribute '__and__' on None or '__rand__' on int Called from (traceback): line 290, in reset_state Differential Revision: https://phab.mercurial-scm.org/D11235

windows: avoid a bytes vs unicode crash reading passwords on py2 This broke in 5b3513177f2b. Specifically, after typing in the password on py2, it would crash with: TypeError: putwch() argument 1 must be cannot convert raw buffers, not str

streamclone: ensure the server sends the right amount of data Otherwise, the client would fail with some confusing error. I have seen an error which I think is this, perhaps due to a concurrent revlog split, which streamclones do not handle correctly and would result in a short read of the index of the revlog being split. Differential Revision: https://phab.mercurial-scm.org/D11236

rewriteutil: fix crash when a rewritten message references f{6,64} Without this, the rewriteutil logic thinks it's found a reference to the wdir pseudo-revision, and then tries to look it up and rewrite it. Stop it from doing that. Amusingly, I had trouble working with this changeset when I didn't describe the defect above using a regular expression, because it would trigger the bug in my installed version of hg. Differential Revision: https://phab.mercurial-scm.org/D11232

tests: add explicit coverage for update_hash_refs from rewriteutil I couldn't find any evidence this is covered by tests in core (but there's a good chance I missed it). We've seen a cute bug in that code, but first let's just cover the cases that work correctly. Differential Revision: https://phab.mercurial-scm.org/D11231

store: document the decoding discrepancy in store.py This will help future people that might be looking into this. Differential Revision: https://phab.mercurial-scm.org/D11220

clone: add a file with special character while testing uncompressed This will make sure we currently do not have any actual impact from issue6548. Differential Revision: https://phab.mercurial-scm.org/D11219

clone: test local clone in `test-clone-uncompressed.t` too This is not an uncompressed test but needs to be tested in the same kind of constraints regarding special file name. Differential Revision: https://phab.mercurial-scm.org/D11218

test-transaction-safety: relax some of the synchronisation schedule We can have `internal` and `external` runs at the same time, so we unlock `external` a bit sooner. Differential Revision: https://phab.mercurial-scm.org/D11216

test-transaction-safety: document the test schedule This helps reader to understand how the test work and actually helped me to detect a missing synchronisation step. Differential Revision: https://phab.mercurial-scm.org/D11215

rhg: Propagate permission errors when finding a repository The Rust standard library has a `Path::is_dir` method that returns false for any I/O error (such as a permission error), not just "No such file or directory". Instead add an `is_dir` function that returns false for non-directories and for "No such file or directory" errors, but propagates other I/O errors. Differential Revision: https://phab.mercurial-scm.org/D11230

rhg: Add build and config instructions to the README file This adds documentation explaining how to compile, configure, and use rhg as well as how the fallback mechanism works. Differential Revision: https://phab.mercurial-scm.org/D11229

dirstate: use `add` in the deprecation message about `add` The previous message was talking about `remove`. Differential Revision: https://phab.mercurial-scm.org/D11224

help: correct config.profiling.freq name (frequency->freq) Differential Revision: https://phab.mercurial-scm.org/D11217

run-tests: use a small timeout for chg instance There is case where the test runner fails to clean up the temporary files in that case, spawned chg instance can stay around for 1 hours. Getting them to shut down sooner cannot hurt. (We should also use a more robust approach for this cleanup) Differential Revision: https://phab.mercurial-scm.org/D11227

run-tests: do not inherit file descriptor when running a command This is one of the difference between python2 and python3 and could have been a reason why test hang with python2 + chg. This does not seems to help the hanging issue at all… However, now that this is written lets reduce the difference between python2 and python3. Differential Revision: https://phab.mercurial-scm.org/D11226

dummyssh: make sure we don't inherit files descriptor to the children This is one of the difference between python2 and python3 and could have been a reason why test hang with python2 + chg. This does not seems to help the hanging issue at all… However, now that this is written lets reduce the difference between python2 and python3. Differential Revision: https://phab.mercurial-scm.org/D11225

tests: make test-serve works on system that allow user to bind low port It is possible to encounter linux system configured in a way that allow port < 1024 to be bound by any users. So we update a test to focus more on the actual test: "resolving `daytime` to an actual port number. The new auto-scaling CI runner provided by clever cloud is doing so. This is the initial motivation for this change, but it seems useful anyway. Differential Revision: https://phab.mercurial-scm.org/D11228

git: restore basic functionality (issue6545) It looks like a big refactor happened on dirstate, and the git extension was just ignored. Differential Revision: https://phab.mercurial-scm.org/D11234

dirstate: fix typo in docstring Spotted while repairing git extension tests. Differential Revision: https://phab.mercurial-scm.org/D11233

typing: add several assertions to dirstatemap to appease pytype I think it's been mentioned in IRC that these can't be None in this case. This fixes: File "/mnt/c/Users/Matt/hg/mercurial/dirstatemap.py", line 213, in addfile: unsupported operand type(s) for &: 'None' and 'int' [unsupported-operands] No attribute '__and__' on None or '__rand__' on int Called from (traceback): line 290, in reset_state File "/mnt/c/Users/Matt/hg/mercurial/dirstatemap.py", line 214, in addfile: unsupported operand type(s) for &: 'None' and 'int' [unsupported-operands] No attribute '__and__' on None or '__rand__' on int Called from (traceback): line 290, in reset_state Differential Revision: https://phab.mercurial-scm.org/D11235