dirstate: introduce a symbolic constant for the AMBIGUOUS_TIME marker This is going to be clearer and easier to track than -1. Ultimately I would like to get ride of this special value everywhere but in the lower level, however we need to clarify the API first. This changeset is part of such clarification. Differential Revision: https://phab.mercurial-scm.org/D10928

dirstate: introduce a symbolic constant for the NONNORMAL marker This is going to be clearer and easier to track than -1. Ultimately I would like to get ride of this special value everywhere but in the lower level, however we need to clarify the API first. This changeset is part of such clarification. Differential Revision: https://phab.mercurial-scm.org/D10927

dirstate: introduce a symbolic constant for the FROM_P2 marker This is going to be clearer and easier to track than -2. Ultimately I would like to get ride of this special value everywhere but in the lower level, however we need to clarify the API first. This changeset is part of such clarification. Differential Revision: https://phab.mercurial-scm.org/D10926

dirstate: split a not-so-one-liner This is shorter and simpler to read. Differential Revision: https://phab.mercurial-scm.org/D10925

dirstate: split a not-so-one-liner This is simpler to read. Differential Revision: https://phab.mercurial-scm.org/D10924

dirstate: split a not-so-one-liner This is shorter and simpler to read. Differential Revision: https://phab.mercurial-scm.org/D10923

dirstate-tree: Keep a counter of descendant nodes that have an entry … and change the `DirstateMap::has_dir` method to be based on this counter being non-zero instead of the mere presence of a node. A node with zero descendent with an entry currently should be removed from the tree, but soon we’ll make the dirstate track additional nodes. (Specifically, for non-ignored directories in order to keep track of their mtime and optimize status by doing fewer `read_dir` calls.) Differential Revision: https://phab.mercurial-scm.org/D10922

dirstate: Removed unused instances of `DirsMultiset` … in Rust-backed dirstatemap. The Python class `dirstatemap` had cached properties `_dirs` and `_alldirs` that were not used for `hastrackeddir` and `hasdir` since they were redundant with corresponding fields for the Rust `DirstateMap` struct. `dirfoldmap` is modified to reuse instead the directory iterator introduced in 3b9914b28133c0918186b6e8b9e4f1916e21338d. Differential Revision: https://phab.mercurial-scm.org/D10921

dirstate-v2: Use 32-bit integers instead of 64-bit for offsets This saves 12 bytes per node. (Nodes representing files or directories.) These are offsets to other parts of the file. This would only be a limitation for a `.hg/dirstate` file larger than 4 GiB, which would only happen for a repository with dozens of millions of files and directories. Differential Revision: https://phab.mercurial-scm.org/D10920

status: Extend read_dir caching to directories with ignored files See code comments Differential Revision: https://phab.mercurial-scm.org/D10909

dirstate-v2: Drop cached read_dir results after .hgignore changes Soon we’ll want the status algorithm to be able to skip `std::fs::read_dir` in more cases, notabling when listing unknown files but not ignored files. When ignore patterns change (which we detect by their hash, added to the dirstate-v2 format in a previous changeset), a formerly-ignored file could become unknown without changing its parent directory’s modification time. Therefore we remove any directory mtime from the dirstate, effictively invalidating the existing caches. Differential Revision: https://phab.mercurial-scm.org/D10907

censor: split the core of the logic into its own function We now have a "generic" rewrite function (only able to do censoring for now) and a thin wrapper that implement the `censor` API with it. We are now ready to start incorporating strip specific changes. Differential Revision: https://phab.mercurial-scm.org/D10904

censor: migrate the logic to a set of `censor_revs` Instead of considering a special unique censored revision within the code, we consider a set of revision (currently always of size 1). This make the main code less censor-centric and prepare for the usage of a similar approach for stripping changesets. Differential Revision: https://phab.mercurial-scm.org/D10903

censor: rename `rl` to `revlog` in the main function Now that the bulk of the code moved to smaller function we are less restricted on line length and we can use more explicite naming. Differential Revision: https://phab.mercurial-scm.org/D10902

censor: extract the part about creating and opening new files in a function The v2_censor function is huge, now that its content has settled a bit it is a good time to split individual part inside dedicated function. The last part is the file copying and opening logic. It now have its own function. Differential Revision: https://phab.mercurial-scm.org/D10901

censor: extract the part about recomputing delta in a function The v2_censor function is huge, now that its content has settled a bit it is a good time to split individual part inside dedicated function. It is now the turn of the logic that recompute new delta to replace the one based on the revision that are going away. Differential Revision: https://phab.mercurial-scm.org/D10900

censor: extract the part about writing the other revision in a function The v2_censor function is huge, now that its content has settled a bit it is a good time to split individual part inside dedicated function. We continue with a small function that add a non-censored revision back to the revlog. Differential Revision: https://phab.mercurial-scm.org/D10899

censor: extract the part about writing the censored revision in a function The v2_censor function is huge, now that its content has settled a bit it is a good time to split individual part inside dedicated function. We start with a small function that process the censored revision. Differential Revision: https://phab.mercurial-scm.org/D10898

censor: put the tuple of open files in an explicit variable This will be simpler to pass these file around in future changesets. Differential Revision: https://phab.mercurial-scm.org/D10897

revlog: rewrite `censors.py` to `rewrite.py` The logic we use for censoring revision will be mostly common with the one we needs for stripping. So we rename the module to `rewrite` to better match its future content. Differential Revision: https://phab.mercurial-scm.org/D10896

Backed out changeset 9b8f326731ac Unfortunately, disabling the rust extensions means newer Mercurial versions no longer have the persistent-nodemap feature enabled. This means a Mercurial 5.8.1 on RPM-based Linux distributions will no longer be able to read repositories created by a Mercurial 5.8 on RPM-based Linux distributions. This violates the compatibility rules (see https://www.mercurial-scm.org/wiki/CompatibilityRules ). For this reason, I have to backout this change. I'll try to find another solution to the 'hg purge' crashes. Differential Revision: https://phab.mercurial-scm.org/D10918

copies: Keep changelog sidedata file open during copy tracing Instead of having a callback that opens and closes that file many times, a add and use a context manager method on the `revlog` class that keeps files open for its duration. Differential Revision: https://phab.mercurial-scm.org/D10888

purge: restore the `testedwith` declaration Otherwise, this extension gets highlighted as a 3rd party extension when an exception occurs. Differential Revision: https://phab.mercurial-scm.org/D10895

rust: remove dead utils module This was introduced a while back while not being imported anywhere in the module system. How pertinent this Facebook module was to the codebase was debatable at the time, nowadays with the Rust VFS layer having laid its foundations, it is clear that we should drop it. Differential Revision: https://phab.mercurial-scm.org/D10890

rebase: use str-keyed opts in remaining places Differential Revision: https://phab.mercurial-scm.org/D10892

rebase: keep str-keyed opts long enough to make `action` a str This is just another little py3 cleanup. Differential Revision: https://phab.mercurial-scm.org/D10891

benchmarks: restore `output` variable lost in D10884 Noticed by pyflakes (which I didn't have installed for Python 3.9 when I sent D10884). Differential Revision: https://phab.mercurial-scm.org/D10894

censor: implement censoring for revlogv2 It is a bit verbose and rough, but it works. Most of that logic can be common for `stripping`, so we can expect more refactoring of that code to accommodate both needs. However I wanted to keep this changesets "simple enough" and before moving forward. We also need to properly delete the older index/data/sidedata file, but this has implication for streaming clone and transaction, so this will come later. Differential Revision: https://phab.mercurial-scm.org/D10869

deltas: at a `target_rev` parameter to finddeltainfo Otherwise, recomputing a delta for a revision might result in a delta against a later revision or a full snapshot thinking we are appending a new revision. We will make use of this during censoring (and later, stripping). Differential Revision: https://phab.mercurial-scm.org/D10887

revlog: list older-but-still-around file in `files` These older files will eventually be removed, but we have to take them in account until they are. So we update `files` to keep fncache happy. Differential Revision: https://phab.mercurial-scm.org/D10868

revlog: add docket method to request new content files If we want to write content into new files, we need to be able to ask for them. Differential Revision: https://phab.mercurial-scm.org/D10867

revlog: add a way to keep track of older uids in the docket When the revlog content is rewritten, we will use new files, to avoid truncating the previous ones. We need some way to keep track of the older files before we clean them up. Differential Revision: https://phab.mercurial-scm.org/D10866

revlog: factor the logic to determine the delta compression out This make the logic clearly isolated and documented and it will help use to reuse it during censors/strip. Differential Revision: https://phab.mercurial-scm.org/D10889

nodemap: fix some comment formatting Black seems unhappy about it. Differential Revision: https://phab.mercurial-scm.org/D10865

revlogv2: fix `hg verify` with revlog v2 We added a bunch of new files but we never teached some internal about it. This is now fixed. Differential Revision: https://phab.mercurial-scm.org/D10858

revlogv2: also test that local clone works This is now the case so lets add a test. Differential Revision: https://phab.mercurial-scm.org/D10857

clone: also report the bookmark file as copied This is a small UI adjustement, but this is easy enough to do. Differential Revision: https://phab.mercurial-scm.org/D10856

clone: reuse the stream clone logic for local clone Streaming clone and local (non `--pull`) clone do mostly the same thing, however they were using different logic to do so. This means the logic frequently went out of sync and that new case had to be dealt with twice. This is fragile and anoying. So we replace this with a re-use of the logic we use for streaming clone. I can see various test changes: - a more precise progress output, - armless fncache loading during clone, - fncache is no longer hardlinked (since we write it by hand). I am not reinstalling the `reposimplestore` specific output, as far as I understand this variant have been broken for years. Differential Revision: https://phab.mercurial-scm.org/D10855

copyfiles: add a way to relax the file system checking for hardlink This is critical for transaction file, less for hardlink/copy clone as we are about to do. Since `pure` build does not have a `getfstype` implementation this would disable hardlink clone for all pure build. So we add a parameter to control that extra check. Differential Revision: https://phab.mercurial-scm.org/D10854

copyfile: add a option callback for failed hardlinking Local clone, adjust its UI depending on the success of using hardlinking, so we add a small callback making it possible for `copyfile` to signal if the requested hardlinking failed. Differential Revision: https://phab.mercurial-scm.org/D10853

streamingclone: extract the scanning part from the generation part We will reuse the scanning part for local clone, so we need it in a dedicated function. Differential Revision: https://phab.mercurial-scm.org/D10852

vfs: add a `register_file` method on the vfs class This is used by the fncache vfs to register new file. Until now, `fncache` have been doing this "automatically" by monitoring write pattern. However this is fragile and when we copy files in place by other means, we need something more robuts. So we add an explicit method to do so. Differential Revision: https://phab.mercurial-scm.org/D10851

clone: use "official" API to create local clone destination This make sure we have a properly created, fully functional repository early. This will be useful to simply the hardlink/copy phases of the local clone to make it share more of its logic with the similar "stream" cloning. This has a minor impact of the test and the resulting repository has is better initialized (eg: the `wcache` directory is pre-created.) Differential Revision: https://phab.mercurial-scm.org/D10850

localrepo: introduce a clone_requirements function This function take a source repository and return a relevant set of requirements that should be used by a copy clone. This will help make the creation of the destination repository during copy clone simpler. Differential Revision: https://phab.mercurial-scm.org/D10849

createrepository: allow to directly pass the target requirements This is useful when doing a local clone that copies store contents, it will requires the destination to use the very same store requirements so directly providing them will be simpler and safer Differential Revision: https://phab.mercurial-scm.org/D10848

narrowbundle: use new context manager for silencing the ui Same reasoning as the previous change. This affects a few tests because of the hack from d7304434390f (changegroup: move message about added changes to transaction summary, 2019-09-08). Differential Revision: https://phab.mercurial-scm.org/D10886

debugbackupbundle: use new context manager for silencing the ui A difference between setting `ui.quiet` and using `ui.silent()` is that the latter also silences `ui.write()` calls. That's practically always what one wants, including here, I think. Differential Revision: https://phab.mercurial-scm.org/D10885

ui: add a context manager for silencing the ui (pushbuffer+popbuffer) We often silence the ui by calling `ui.pushbuffer()` followed (a later in the code) by `ui.popbuffer()`. These places can be identified by the fact that they ignore the output returned from `ui.popbuffer()`. Let's create a context manager for these cases, to avoid repetition, and to avoid accidentally leaving the ui silent on exceptions. I deliberately called the new function `silent()` instead of `buffered()`, because it's just an implementation detail that it uses `pushbuffer()` and `popbuffer()`. We could later optimize it to not buffer the output. Differential Revision: https://phab.mercurial-scm.org/D10884