pyoxidizer: support code signing Newer versions of PyOxidizer feature built-in support for code signing. You simply declare a code signer in the Starlark configuration file, activate it for automatic signing, and PyOxidizer will add code signatures to signable files as it encounters them. This commit teaches our Starlark configuration file to enable automatic code signing. But only on Windows for the moment, as our immediate goal is to overhaul the Windows packaging. The feature is opt-in: you must pass variables to PyOxidizer's build context via `pyoxidizer build --var` or `pyoxidizer build --var-env` to activate code signing. Differential Revision: https://phab.mercurial-scm.org/D10684

pyoxidizer: support producing MSI installers Newer versions of PyOxidizer have support for building WiX MSI installers "natively." Essentially, you can script the definition of your WiX installer via Starlark and PyOxidizer can invoke WiX tools to produce the installer. This commit teaches our PyOxidizer config file to produce MSI installers similarly to how `contrib/packaging/packging.py wix` would do it. We had to make a very minor change to `mercurial.wxs` to reflect different paths depending on who builds. This is because when PyOxidizer builds WiX installers, it does so from an isolated directory, not Mercurial's source directory. We simply copy the files into the build environment so they are accessible. After this change, running `pyoxidizer build msi` produces a nearly identical install layout to what the previous method produces. When I applied this series on top of the 5.8 tag, here is the list of differences and explanations: * docs/*.html files are missing from the new installer because the Python build environment doesn't have docutils. * .pyd and .exe files differ, likely because I'm using a different Visual Studio toolchain on my local computer than the official build environment. * Various .dist-info/ directories have different names. This is because older versions of PyOxidizer had buggy behavior and weren't properly normalizing package names in .dist-info/ directories. e.g. we went from `cached-property-1.5.2.dist-info` to `cached_property-1.5.2.dist-info`. * Translations (.mo files) may be missing if gettext isn't in %Path%. This is because the packaging.py code installs gettext and ensures it can be found. * Some *.dist-info/RECORD files vary due to SHA-256 content digest divergence due to build environment differences. (This should be harmless.) * The new install layout ships a python3.dll because newer versions of PyOxidizer ship this file. * The new install layout has a different vcruntime140.dll and also a vcruntime140_1.dll because newer versions of PyOxidizer ship a newer version of the Visual C++ Redistributable Runtime. The new PyOxidizer functionality is not yet integrated with packaging.py. This will come in a subsequent commit. So for now, the new functionality introduced here is unused. Differential Revision: https://phab.mercurial-scm.org/D10683

pyoxidizer: use allocator_backend instead of raw_allocator The name of this attribute changed in PyOxidizer 0.11. Differential Revision: https://phab.mercurial-scm.org/D10682

pyoxidizer: simplify targets The split targets existed to enable the use of a non-default distribution flavor on Windows. Modern versions of PyOxidizer use the "standalone_dynamic" distribution flavor by default. So our split brain workaround is no longer needed. Here, we unify the targets. We also remove an unreferenced target function to create a resources file. Differential Revision: https://phab.mercurial-scm.org/D10681

contrib: install PyOxidizer 0.16.0 PyOxidizer now provides MSI installers and pre-built Linux binaries. So we install that way. This significantly reduces the time to bootstrap a new machine in automation, as building PyOxidizer from source on a low core count machine takes several minutes. This change temporarily breaks the ability of the automated environment to use the in-repo pyoxidizer.bzl configuration file, as there are backwards-incompatible changes with the upgrade that need to be reflected. We'll handle those in subsequent commits. Differential Revision: https://phab.mercurial-scm.org/D10680

pyoxidizer: pin Python to 3.8 Newer versions of PyOxidizer use version 3.9 by default. As part of upgrading PyOxidizer, we want to pin the version at 3.8 so we can compare differences more easily. Differential Revision: https://phab.mercurial-scm.org/D10679

pyoxidizer: remove some boilerplate in file This isn't used for anything and can safely be removed. Differential Revision: https://phab.mercurial-scm.org/D10678

contrib: upgrade Rust in Windows automation I'm not sure why we don't install the minimum required Rust version here like we do for Linux. Whatever: that's unrelated to wanting to stay modern. Differential Revision: https://phab.mercurial-scm.org/D10677

contrib: update Python versions in Windows automation Let's keep our Python versions modern. Differential Revision: https://phab.mercurial-scm.org/D10676

automation: update rust in Linux environment Our minimum supported Rust is 1.41.1 per rust/README.rst. We also bump the modern Rust version to latest stable to stay current. Differential Revision: https://phab.mercurial-scm.org/D10675

automation: install latest Python versions in Linux environment Let's keep our Linux environment up to date. Differential Revision: https://phab.mercurial-scm.org/D10674

automation: create Python 3.5 variant of requirements.txt The automation environment is refusing to build with the previous file because some dependencies won't install on Python 3.5. I couldn't find an easy way to salvage the situation with a single requirements.txt file. So, I decided to introduce a variant for Python 3.5. As part of this, we update packages to latest versions. (I do question why we are still supporting Python 3.5...) Differential Revision: https://phab.mercurial-scm.org/D10690

hgweb: Fix deprecation warning in Python 3.10 (issue6520) Differential Revision: https://phab.mercurial-scm.org/D10711

dirstate-tree: Remove newly-empty nodes after removing a `DirstateEntry` This is actually necessary to make `DirstateMap::has_dir` correct, since it assumes that a node without a `DirstateEntry` has at least one descedant node with a `DirstateEntry`. This bug would become apparent when a later changeset persists tree nodes on disk in the "dirstate-v2" format. Differential Revision: https://phab.mercurial-scm.org/D10706

dirstate-tree: Refactor DirstateMap::drop_file to be recursive It should behave the same as before. This will enable the next changeset to run code on the way "down" (in order to removing newly-empty nodes). Differential Revision: https://phab.mercurial-scm.org/D10705

templates: fix `revset('parents()') % ...` in amend message template I don't understand why, but putting `revset('parents()') % {desc}` in the commit message template for amend resulted in a crash because `memctx.hex()` did `hex(self.node())` and its node was None. This patch fixes that. Martin von Zweigbergk <martinvonz@google.com> Differential Revision: https://phab.mercurial-scm.org/D10707

path: teach the `hg path` command to display boolean sub-option The next changeset introduce one. Differential Revision: https://phab.mercurial-scm.org/D10450

parselist: move the function from config to stringutil We move the function in a lower level module to avoid cycle. It moves next to `parsebool` who had to migrate for the same reasons. Differential Revision: https://phab.mercurial-scm.org/D10449

bookmark: use `list_paths` to access path definition The content of the `[paths]` config section is receiving transformation that make it hard to recognise whats the actual name, and the next changeset will make it worse. So we use the official API for this instead. Differential Revision: https://phab.mercurial-scm.org/D10448

urlutil: make `paths` class old list of `path` We move from a `{name → path}` mapping to a `{name → [path]}` mapping. And update all user code accordingly. For now, all the list contains exactly one element, but we are now in a good place to make the config understand a list of url. Differential Revision: https://phab.mercurial-scm.org/D10447

urlutil: extract `chain_path` in a function This will no longer modify `path` inplace so it does not make much sense as a method. Differential Revision: https://phab.mercurial-scm.org/D10446

urlutil: add a `copy` method to `path This will be useful when inheriting from multiple path at the same time. Differential Revision: https://phab.mercurial-scm.org/D10445

template: add a `paths` field to all entry in peerurls This make it possible to display multiple path per name in the near future. Differential Revision: https://phab.mercurial-scm.org/D10444

status: Add some more tests Trying to improve coverage for various non-ovbious scenarios Differential Revision: https://phab.mercurial-scm.org/D10704

run-tests: use the same python version for shebang lines on Windows The latest py3 is used if the minor number isn't specified. After running the script to install all of the build dependencies, that moved the default from 3.8 to 3.9 on the CI system. That in turn caused a bunch of tests to be skipped that were running prior, even when the test runner was invoked with `py -3.8`. While we should almost always use the latest version, we really shouldn't make it hard to test different versions or allow things to randomly break in subtle ways like that. Differential Revision: https://phab.mercurial-scm.org/D10702

vfs: Fix deprecation warning in Python 3.10 (issue6520) Differential Revision: https://phab.mercurial-scm.org/D10710

contrib: fix typo I forgot to fix it in flight, this commit will do fine.

revlog: rename `indexdata` to entry_data Same reasoning as the previous changeset, we might not be looking at index data here. Differential Revision: https://phab.mercurial-scm.org/D10602

revlog: use "entry_point" phrasing for loading the revlog The main entry for the revlog will not necessary be the index, but a small "docket". So we change the variable names and we move the initialisation of the index_file and data_file after that first entry point have been read. Differential Revision: https://phab.mercurial-scm.org/D10601

revlog: directly use self._format_flags when loading index The shorthand variable does not bring much, so we drop it to simplify the code. Differential Revision: https://phab.mercurial-scm.org/D10600

revlog: directly use self._format_version when loading index The shorthand variable does not bring much, so we drop it to simplify the code. Differential Revision: https://phab.mercurial-scm.org/D10599

revlog: use `_format_flags` to access flags instead of `header` It seems better to reuse the variable we carefully extracted This also open the way to more flexible way to retrieve these flags. Differential Revision: https://phab.mercurial-scm.org/D10598

revlog: rename `newversionflags` to `new_header` This make it consistent with the previous changeset. Differential Revision: https://phab.mercurial-scm.org/D10597

revlog: rename `versionflags` to header The variable is named as such because it contains "version" information and "flags" information. However you mostly needs to know the code to understand the name which is not great. The fact that this is the very first four bytes in all revlog seems more relevant, so we rename the variable "header". Differential Revision: https://phab.mercurial-scm.org/D10596

revlog: move index reading logic in a dedicated method They are multiple motivation to do it: * The logic is complicated enough to deserver its own method. * We will need to reuse this once we put a docket in use. * This split the actual reading from the processing of the read data better. Differential Revision: https://phab.mercurial-scm.org/D10595

revlog: define the actual index and datafile at loading time This is just code movement, to make the code closer to where we actually use it and where it will be defined in the future. Differential Revision: https://phab.mercurial-scm.org/D10594

revlog: simplify a conditionnal in _enforceinlinesize This is a gratuitous change to make things a bit easier to read. Differential Revision: https://phab.mercurial-scm.org/D10593

revlog: drop `flush` parameter from `_peek_iscensored` This is not used anywhere. Differential Revision: https://phab.mercurial-scm.org/D10592

revlog: fix error message when data are missing The error message the message was not using the requested offset, but the adjusted offset to that read more data for improved caching. This resulted in confusing error message. Differential Revision: https://phab.mercurial-scm.org/D10591

revlog: rename `nodemap_file` to `_nodemap_file` Same reasoning as for `indexfile and datafile`, lets hide these implementation details. Differential Revision: https://phab.mercurial-scm.org/D10590

revlog: use revlog.display_id in error related to bad revisions Differential Revision: https://phab.mercurial-scm.org/D10589

revlog: use revlog.display_id in "revision too big" errors Differential Revision: https://phab.mercurial-scm.org/D10588

revlog: use revlog.display_id in censor related errors Differential Revision: https://phab.mercurial-scm.org/D10587

revlog: use revlog.display_id in integrity error Differential Revision: https://phab.mercurial-scm.org/D10586

revlog: use revlog.display_id in ambiguity errors Differential Revision: https://phab.mercurial-scm.org/D10585

revlog: use revlog.display_id for corruption error Differential Revision: https://phab.mercurial-scm.org/D10584

revlog: use revlog.display_id in format related errors Differential Revision: https://phab.mercurial-scm.org/D10583

revlog: use revlog.display_id in narrow error message Differential Revision: https://phab.mercurial-scm.org/D10582

revlog: use revlog.display_id in LookupError Differential Revision: https://phab.mercurial-scm.org/D10581

revlog: use revlog.display_id for FilteredLookupError Differential Revision: https://phab.mercurial-scm.org/D10580

revlog: introduce a `display_id` property We currently using the "index file" to identify a revlog in error output. Since we are about to make the "index file" location more volatile, we need something better. We move to use the "radix", as it is close to what we currently use. We could probably do better, as pointed out in the comment, however that would be a quite detour from my current goal. Differential Revision: https://phab.mercurial-scm.org/D10579

revlog: also use radix when computing nodemap data file We have a radix, lets use it! Differential Revision: https://phab.mercurial-scm.org/D10578

revlog: stop usage of `_indexfile` to computing nodemap path We now have the radix explicitely lets use the radix explicitely Differential Revision: https://phab.mercurial-scm.org/D10577

revlog: use a "radix" to address revlog Instead of pointing to the index directly and to derive the other file from that, we directly provide the radix and let the revlog determine the associated file path internally. This is more robust and will give us more flexibility for picking this file name in the future. Differential Revision: https://phab.mercurial-scm.org/D10576

revlog: rename `datafile` to `datafile` We want to make the actual location of the datafile and location more of an implementation details than what is is currently. In that process, we make the attribute private. Differential Revision: https://phab.mercurial-scm.org/D10575

revlog: rename `indexfile` to `_indexfile` We want to make the actual location of the indexfile and location more of an implementation details than what is is currently. In that process, we make the attribute private. Differential Revision: https://phab.mercurial-scm.org/D10574

filelog: drop `indexfile` from `filelog` Since `filelog` objects are not revlog (no really, they are not…) we drop the revlog specific attribute. We need to directly access the underlying revlog in a couple of place that already assume that we have a revlog here. This is motivated by future change to that revlog attribute. Differential Revision: https://phab.mercurial-scm.org/D10573

manifest: drop the `indexfile` from `manifestrevlog` Since `manifestrevlog` object are not revlog (no really, they are not…) we drop the revlog specific attribute. We need to directly access the underlying revlog in a couple of place that already assume that we have a revlog here. This is motivated by future change to that revlog attribute. Differential Revision: https://phab.mercurial-scm.org/D10572

revlog: deal with special "postfix" explicitely revlog usually use a straight forward '.i' and '.d' naming except for two cases "in-transaction" changelog, and censoring. Our goal is to let the revlog code deal with the internal of the file naming itself. To do so, we need to start dealing with these postfix explicitly. Differential Revision: https://phab.mercurial-scm.org/D10571

revlog: split the option initialisation in its own method The part of the code is huge, keeping it separated will keep the `_loadindex` method simpler and help keeping logic well insulated. Differential Revision: https://phab.mercurial-scm.org/D10570

revlog: always "append" full size tuple Same reasoning as the previous patch. Differential Revision: https://phab.mercurial-scm.org/D10569

revlog: make the index always return the same tuple It is simpler to manage the diferrence in on disk format in the internal index code itself and lets the rest of the code always handle the same object. This will become even more important when the data we store will be entirely different (for example the changelog does not need the "linkrev" field. We start with item reading, we will deal with item writing in the next changesets. Differential Revision: https://phab.mercurial-scm.org/D10568

revlog: introduce an explicit `format_version` member in the index struct This will allow for cleaner check than assuming each version has a different size. Unsurprisingly I am planning to use this to introduce more format variant. Differential Revision: https://phab.mercurial-scm.org/D10567

revlog: rename `hdrsize` to `entry_size` in the C code This is the size of and index entry, so lets make it clearer. Differential Revision: https://phab.mercurial-scm.org/D10566

revlog: split the `version` attribute into its two components The `revlog.version` attribute contained an integer coding 2 different informations: * the revlog version number * a bit field defining some specific feature of the revlog We now explicitly store the two components independently. This avoid exposing the implementation details all around the code and prepare for future revlog version that would encode the information in a different way. In the process we drop the `version` attribute from the interface. It was flagged for removal when that interface was created. Differential Revision: https://phab.mercurial-scm.org/D10565

verify: pass a revlog to `_checkrevlog` in `_verifymanifest` Since `manifestrevlog` is not a `revlog`, we are passing strange thing to `_checkrevlog`. We fix this to avoid breakage during future change. Differential Revision: https://phab.mercurial-scm.org/D10564

revlog: replace flag check related to generaldelta with attribute check Same logic as the previous changesets. Differential Revision: https://phab.mercurial-scm.org/D10563

revlog: replace REVLOGV2 check related to sidedata with `hassidedata` checks This is more flexible and semantically more correct. The associated revlog's attribute exist since 827cb4fe62a3, so well we start linking sidedata to revlogv2. Differential Revision: https://phab.mercurial-scm.org/D10562

revlog: explicitely pass the "indexfile" parameter Most of this was already done when introducing the `target` parameter, but some remained. Having "indexfile" passed explicitely will help us to change the way we address a revlog later in the stack. With the introduction of more generic `docket`, the entry point will not necessarly be `xxx.i` file, and the actual index files will have a variable name. Differential Revision: https://phab.mercurial-scm.org/D10561

revlog: highlight current incompatibility in `rewrite_sidedata` See comment for details. We will need to fix the test coverage when this incompatibility is lifted. Differential Revision: https://phab.mercurial-scm.org/D10544

revlog: adjust rewrite_sidedata code to not delete existing revlog content The "w+" file mode is deleting all the content of the opened file. Which is bad… This is not caught by the test because the test only check for a full, initial pull where not pre-existing content exists. So we need to fix our test coverage here. However they are another issue that prevent "incremental" pull to work here. See next changeset for details. Differential Revision: https://phab.mercurial-scm.org/D10543

status: Add tests for some more edge cases * Size-preserving file contents modification * Filtering output to a deleted or removed file Differential Revision: https://phab.mercurial-scm.org/D10701

status: Extend issue 6483 test to exclude patterns With `hg status -X`, not just include pattern with `hg status -I` Differential Revision: https://phab.mercurial-scm.org/D10700

dirstate-tree: Add a test showing that issue 6335 is fixed … when using the new status algorithm and the tree-based dirstate. The previous algorithm still has this bug. Differential Revision: https://phab.mercurial-scm.org/D10699

dirstate-tree: Add a dirstate-v1-tree variant of some tests The `dirstate-v1` variant has the previous behavior. `dirstate-v1-tree` uses the same format on disk, but uses the new `DirstateMap` with a tree data structure and the new `status` algorithm. These were untested so far. Differential Revision: https://phab.mercurial-scm.org/D10698

merge with stable

rename: add hint about --at-rev if source file doesn't exist It's quite common that users want to record copy (rename) information after committing the working copy changes (i.e. an added and a deleted file). When they try `hg mv [--after] <src> <dst>`, that just fails because the source file doesn't exist. It seems helpful if we can point them to `--at-rev=.` in this case. Differential Revision: https://phab.mercurial-scm.org/D10697

dirstate-tree: Borrow paths from the "on disk" bytes Use std::borrow::Cow to avoid some memory allocations and copying. Differential Revision: https://phab.mercurial-scm.org/D10560

dirstate-tree: Borrow copy source paths from the "on disk" bytes Use std::borrow::Cow to avoid some memory allocations and copying. These particular allocations are not visible when profiling (as many files in a typical repo don’t have a copy source). This change is "warm up" for doing the same with paths of files themselves, which is more involved since those paths are used as `HashMap` keys. This gets of the way the addition of a lifetime parameter to several types. Differential Revision: https://phab.mercurial-scm.org/D10559

rust: Use `&HgPath` instead of `&HgPathBuf` in may APIs Getting the former (through `Deref`) is almost the only useful thing one can do with the latter anyway. With this changes, API become more flexible for the "provider" of these paths which may store something else that Deref’s to HgPath, such as `std::borrow::Cow<HgPath>`. Using `Cow` can help reduce memory alloactions and copying. Differential Revision: https://phab.mercurial-scm.org/D10558

dirstate-tree: Make `DirstateMap` borrow from a bytes buffer … that has the contents of the `.hg/dirstate` file. This only applies to the tree-based flavor of `DirstateMap`. For now only the entire `&[u8]` slice is stored, so this is not useful yet. Adding a lifetime parameter to the `DirstateMap` struct (in hg-core) makes Python bindings non-trivial because we keep that struct in a Python object that has a dynamic lifetime tied to Python’s reference-counting and GC. As long as we keep the `PyBytes` that owns the borrowed bytes buffer next to the borrowing struct, the buffer will live long enough for the borrows to stay valid. However this relationship cannot be expressed in safe Rust code in a way that would statisfy they borrow-checker. We use `unsafe` code to erase that lifetime parameter, and encapsulate it in a safe abstraction similar to the owning-ref crate: https://docs.rs/owning_ref/ Differential Revision: https://phab.mercurial-scm.org/D10557

rust: Read dirstate from disk in DirstateMap constructor Before this changeset, Python code first creates an empty `DirstateMap` Rust object, then immediately calls its `read` method with a byte string of the contents of the `.hg/dirstate` file. This makes that byte string available to the constructor of `DirstateMap` in the hg-cpython crate. This is a first step towards enabling parts of `DirstateMap` in the hg-core crate to borrow from this buffer without copying. Differential Revision: https://phab.mercurial-scm.org/D10556

rust: Remove handling of `parents` in `DirstateMap` The Python wrapper class `dirstatemap` can take care of it. This removes the need to have both `_rustmap` and `_inner_rustmap`. Differential Revision: https://phab.mercurial-scm.org/D10555

dirstate-tree: Fold "tracked descendants" counter update in main walk For the purpose of implementing `has_tracked_dir` (which means "has tracked descendants) without an expensive sub-tree traversal, we maintaing a counter of tracked descendants on each "directory" node of the tree-shaped dirstate. Before this changeset, mutating or inserting a node at a given path would involve: * Walking the tree from root through ancestors to find the node or the spot where to insert it * Looking at the previous node if any to decide what counter update is needed * Performing any node mutation * Walking the tree *again* to update counters in ancestor nodes When profiling `hg status` on a large repo, this second walk takes times while loading a the dirstate from disk. It turns out we have enough information to decide before he first tree walk what counter update is needed. This changeset merges the two walks, gaining ~10% of the total time for `hg update` (in the same hyperfine benchmark as the previous changeset). --- Profiling was done by compiling with this `.cargo/config`: [profile.release] debug = true then running with: py-spy record -r 500 -n -o /tmp/hg.json --format speedscope -- \ ./hg status -R $REPO --config experimental.dirstate-tree.in-memory=1 then visualizing the recorded JSON file in https://www.speedscope.app/ Differential Revision: https://phab.mercurial-scm.org/D10554

dirstate-tree: Use HashMap instead of BTreeMap BTreeMap has the advantage of its "natural" iteration order being the one we need in the status algorithm. With HashMap however, iteration order is undefined so we need to allocate a Vec and sort it explicitly. Unfortunately many BTreeMap operations are slower than in HashMap, and skipping that extra allocation and sort is not enough to compensate. Switching to HashMap + sort makes `hg status` 17% faster in one test case, as measure with hyperfine: ``` Benchmark #1: ../hg2/hg status -R $REPO --config=experimental.dirstate-tree.in-memory=1 Time (mean ± σ): 765.0 ms ± 8.8 ms [User: 1.352 s, System: 0.747 s] Range (min … max): 751.8 ms … 778.7 ms 10 runs Benchmark #2: ./hg status -R $REPO --config=experimental.dirstate-tree.in-memory=1 Time (mean ± σ): 651.8 ms ± 9.9 ms [User: 1.251 s, System: 0.799 s] Range (min … max): 642.2 ms … 671.8 ms 10 runs Summary './hg status -R $REPO --config=experimental.dirstate-tree.in-memory=1' ran 1.17 ± 0.02 times faster than '../hg2/hg status -R $REPO --config=experimental.dirstate-tree.in-memory=1' ``` * ./hg is this revision * ../hg2/hg is its parent * $REPO is an old snapshot of mozilla-central Differential Revision: https://phab.mercurial-scm.org/D10553

dirstate-tree: Add #[timed] attribute to `status` and `DirstateMap::read` When running with a `RUST_LOG=trace` environment variable, the `micro_timer` crate prints the duration taken by each call to functions with that attribute. Differential Revision: https://phab.mercurial-scm.org/D10552

dirstate-tree: Paralellize the status algorithm with Rayon The `rayon` crate exposes "parallel iterators" that work like normal iterators but dispatch work on different items to an implicit global thread pool. Differential Revision: https://phab.mercurial-scm.org/D10551

dirstate-tree: Avoid BTreeMap double-lookup when inserting a dirstate entry The child nodes of a given node in the tree-shaped dirstate are kept in a `BTreeMap` where keys are file names as strings. Finding or inserting a value in the map takes `O(log(n))` string comparisons, which adds up when constructing the tree. The `entry` API allows finding a "spot" in the map that may or may not be occupied and then access that value or insert a new one without doing map lookup again. However the current API is limited in that calling `entry` requires an owned key (and so a memory allocation), even if it ends up not being used in the case where the map already has a value with an equal key. This is still a win, with 4% better end-to-end time for `hg status` measured here with hyperfine: ``` Benchmark #1: ../hg2/hg status -R $REPO --config=experimental.dirstate-tree.in-memory=1 Time (mean ± σ): 1.337 s ± 0.018 s [User: 892.9 ms, System: 437.5 ms] Range (min … max): 1.316 s … 1.373 s 10 runs Benchmark #2: ./hg status -R $REPO --config=experimental.dirstate-tree.in-memory=1 Time (mean ± σ): 1.291 s ± 0.008 s [User: 853.4 ms, System: 431.1 ms] Range (min … max): 1.283 s … 1.309 s 10 runs Summary './hg status -R $REPO --config=experimental.dirstate-tree.in-memory=1' ran 1.04 ± 0.02 times faster than '../hg2/hg status -R $REPO --config=experimental.dirstate-tree.in-memory=1' ``` * ./hg is this revision * ../hg2/hg is its parent * $REPO is an old snapshot of mozilla-central Differential Revision: https://phab.mercurial-scm.org/D10550

dirstate-tree: Handle I/O errors in status Errors such as insufficient permissions when listing a directory are logged, and the algorithm continues without considering that directory. Differential Revision: https://phab.mercurial-scm.org/D10549

dirstate-tree: Ignore FIFOs etc. in the status algorithm If a filesystem directory contains anything that is not: * a "normal" file * a symbolic link * or a directory … act as if that directory entry was not there. For example, if that path was previously a tracked file, mark it as deleted or removed. Differential Revision: https://phab.mercurial-scm.org/D10548

dirstate-tree: Add the new `status()` algorithm With the dirstate organized in a tree that mirrors the structure of the filesystem tree, we can traverse both trees at the same time in order to compare them. This is hopefully more efficient that building multiple big hashmaps for all of the repository’s contents. Differential Revision: https://phab.mercurial-scm.org/D10547

dirstate-tree: Give to `status()` mutable access to the `DirstateMap` Differential Revision: https://phab.mercurial-scm.org/D10546

rust: Add doc-comments to DirstateStatus fields Differential Revision: https://phab.mercurial-scm.org/D10495

rust: Move "lookup" a.k.a. "unsure" paths into `DirstateStatus` struct Instead of having `status()` returning a tuple of those paths and `DirstateStatus`. Differential Revision: https://phab.mercurial-scm.org/D10494

rust: Remove DirstateMap::file_fold_map This was a HashMap constructed on demand and then cached in the DirstateMap struct to avoid reconstructing at the next access. However the only use is in Python bindings converting it to a PyDict. That method in turn is wrapped in a @cachedproperty in Python code. This was two redudant layers of caching. This changeset removes the Rust-level one to keep the Python dict cache, and have bindings create a PyDict by iterating. Differential Revision: https://phab.mercurial-scm.org/D10493

dirstate-tree: Add "non normal" and "from other parent" sets Unlike the other DirstateMap implementation, these sets are not materialized separately in memory. Instead we traverse the main tree. Differential Revision: https://phab.mercurial-scm.org/D10492

dirstate-tree: Add add_file, remove_file, and drop_file Again, various counters need to be kept up to date. Differential Revision: https://phab.mercurial-scm.org/D10491

dirstate-tree: Add has_dir and has_tracked_dir A node without a `DirstateMap` entry represents a directory. Only some values of `EntryState` represent tracked files. A directory is considered "tracked" if it contains any descendant file that is tracked. To avoid a sub-tree traversal in `has_tracked_dir` we add a counter for this. A boolean flag would become insufficent when we implement remove_file and drop_file. `add_file_node` is more general than needed here, in anticipation of adding the `add_file` and `remove_file` methods. Differential Revision: https://phab.mercurial-scm.org/D10490

dirstate-tree: Add clear_ambiguous_times in the new DirstateMap Also drive-by refactor it in the other DirstateMap Differential Revision: https://phab.mercurial-scm.org/D10489

dirstate-tree: Add copy_map_insert and copy_map_remove Differential Revision: https://phab.mercurial-scm.org/D10488

dirstate-tree: Maintain a counter of DirstateEntry’s and copy sources This allows implementing __len__ for DirstateMap and CopyMap efficiently, without traversing the tree. Differential Revision: https://phab.mercurial-scm.org/D10487

dirstate-tree: Serialize to disk The existing `pack_dirstate` function relies on implementation details of `DirstateMap`, so extract some parts of it as separate functions for us in the tree-based `DirstateMap`. The `bytes-cast` crate is updated to a version that has an `as_bytes` method, not just `from_bytes`: https://docs.rs/bytes-cast/0.2.0/bytes_cast/trait.BytesCast.html#method.as_bytes Drive-by refactor `clear_ambiguous_times` which does part of the same thing. Differential Revision: https://phab.mercurial-scm.org/D10486

rust: Add a Timestamp struct instead of abusing Duration `SystemTime` would be the standard library type semantically appropriate instead of `Duration`. But since the value is coming from Python as a plain integer and used in dirstate packing code as an integer, let’s make a type that contains a single integer instead of using one with sub-second precision. Differential Revision: https://phab.mercurial-scm.org/D10485

dirstate-tree: Add tree traversal/iteration Like Python’s, Rust’s iterators are "external" in that they are driven by a caller who calls a `next` method. This is as opposed to "internal" iterators who drive themselves and call a callback for each item. Writing an internal iterator traversing a tree is easy with recursion, but internal iterators cannot rely on the call stack in that way, they must save in an explicit object all state that they need to be preserved across two `next` calls. This algorithm uses a `Vec` as a stack that contains what would be local variables on the call stack if we could use recursion. Differential Revision: https://phab.mercurial-scm.org/D10370

dirstate-tree: Add map `get` and `contains_key` methods Differential Revision: https://phab.mercurial-scm.org/D10369

dirstate-tree: Add parsing only dirstate parents from disk Differential Revision: https://phab.mercurial-scm.org/D10368

dirstate-tree: Implement DirstateMap::read This reads the on-disk dirstate in its current format (a flat sequence of entries) and creates a tree in memory. Differential Revision: https://phab.mercurial-scm.org/D10367

dirstate-tree: Add `WithBasename` wrapper for `HgPath` In the tree-shaped dirstate we want to have nodes representing files or directories, where directory nodes contain a map associating "base" names to child nodes for child files and directories. Many dirstate operations expect a full path from the repository root, but re-concatenating string from nested map keys all the time might be expensive. Instead, `WithBasename` stores a full path for these operations but behaves as its base name (last path component) for equality and comparison. Additionally `inclusive_ancestors` provides the successive map keys that are needed when inserting a new dirstate node at a given full path. Differential Revision: https://phab.mercurial-scm.org/D10365

dirstate-tree: Empty shell for a second Rust DirstateMap implementation For background see description of the previous changeset "Make Rust DirstateMap bindings go through a trait object". Add an empty shell for a opt-in second Rust implementation of the `DirstateMap` type and the `status` function. For now all methods panic. This can be seen in "action" with: ./hg status --config experimental.dirstate-tree.in-memory=1 Differential Revision: https://phab.mercurial-scm.org/D10364

dirstate-tree: Abstract "non-normal" and "other parent" sets Instead of exposing `HashSet`s directly, have slightly higher-level methods for the operations that Python bindings need on them. Differential Revision: https://phab.mercurial-scm.org/D10363

dirstate-tree: Make Rust DirstateMap bindings go through a trait object This changeset starts a series that adds an experiment to make status faster by changing the dirstate (first only in memory and later also on disk) to be shaped as a tree matching the directory structure, instead of the current flat collection of entries. The status algorithm can then traverse this tree dirstate at the same time as it traverses the filesystem. We (Octobus) have made prototypes that show promising results but are prone to bitrot. We would like to start upstreaming some experimental Rust code that goes in this direction, but to avoid disrupting users it should only be enabled by some run-time opt-in while keeping the existing dirstate structure and status algorithm as-is. The `DirstateMap` type and `status` function look like the appropriate boundary. This adds a new trait that abstracts everything Python bindings need and makes those bindings go through a `dyn` trait object. Later we’ll have two implementations of this trait, and the same bindings can use either. Differential Revision: https://phab.mercurial-scm.org/D10362

remotefilelog: use the correct capability when using getfilestype threaded The functon was overlooked when the capability was renamed Differential Revision: https://phab.mercurial-scm.org/D10673

test-copies: test that copies' sidedata can get computed during push If the source of the push does not have the necessary sidedata but the server needs them, the client should compute them on the fly while pushing. Differential Revision: https://phab.mercurial-scm.org/D10350

test-copies: test that copies' sidedata can get computed during pull If the source does not have the data, the pulling client should compute the necessary side-data while pulling. Differential Revision: https://phab.mercurial-scm.org/D10349

test-copies: test that copies' sidedata does not get corrupted during push This is an important usecase. Differential Revision: https://phab.mercurial-scm.org/D10348

test-copies: test that copies' sidedata does not get corrupted during pull This is an important usecase. Differential Revision: https://phab.mercurial-scm.org/D10347

test-copies: simplify some conditional output Now that all computation using sidedata give the same result, we can simplify conditional Differential Revision: https://phab.mercurial-scm.org/D10346

sidedata: move documentation about sidedata helpers to sidedata module Differential Revision: https://phab.mercurial-scm.org/D10361

sidedata: move sidedata-related utils to the dedicated module Differential Revision: https://phab.mercurial-scm.org/D10360

sidedata: replace sidedata upgrade mechanism with the new one Note: this is split into a separate change (like some other patches in this series) because it's not easy to have all patches work 100% and this seemed easier for reviewers. When cloning or upgrading a repo, we may need to compute (or remove) sidedata. This is the same mechanism that is used in exchange, so we re-use the new system to simplify the code and fix the remaining issues (correctly dropping flags and handling partial removal, etc.). This also highlighted an issue with `test-copies-in-changeset.t` that kept sidedata categories that are not relevant anymore. They should probably be dropped entirely, but that would be for another patch. Differential Revision: https://phab.mercurial-scm.org/D10359