multi-urls: document the feature We have a feature lets make it visible to people. Differential Revision: https://phab.mercurial-scm.org/D10453

multi-urls: add a boolean suboption that unlock path specification as list When this option is set, a list of patch can be specifed as value for `[paths]` entries. For the command who support it, this behave the same as providing multiple destination of the command line. Differential Revision: https://phab.mercurial-scm.org/D10452

urlutil: move url "fixing" at the time of `ui.paths` initialization Doing such fixing at the time is simpler and will be necessary to deal with urls list in a sane manner. It also reduce the size of fix-config which is always better. I wish we could get ride of the hackish way to pass the root around, I suspect that the `root` variable could be stored as part of the config value, along side the source. However getting to the end of this `root` business is a far too large detours to make now. The test change to `tests/test-hgrc.t` and `test-config.t` are expectied since we are not longer altering the config itself, but the way it is interpreted when building path. This seems more correct. I also added a couple of test call to `test-config.t` and `test-globalopts.t` to clarify that the expanding process is properly happening a the right time. Differential Revision: https://phab.mercurial-scm.org/D10451

rewriteutil: add pointer to help text when rewrite would cause divergence The evolve extension's version of the hint has this pointer. I missed it when I moved it to core. Differential Revision: https://phab.mercurial-scm.org/D10723

changelogv2: use a dedicated on disk format for changelogv2 We drop two unused entry. This is mostly a proof of concept before starting to actually rework the format. Differential Revision: https://phab.mercurial-scm.org/D10667

revlog: do not call Rust code if the index is not compatible with it This will avoid hitting the TypeError we defined in the previous changesets. This is the simplest fix but not the most elegant. Ideally we would teach the Rust code to use any kind of revlog. However this is an adventure for another time. Differential Revision: https://phab.mercurial-scm.org/D10666

revlog: signal which revlog index are compatible with Rust Otherwise, Rust may treat python object like `cindex` object, leading to trouble. The new attribute is an integer because I expect we might need a flag field in the future. As a start we get the rust code to raise a clear TypeError. We will use the information in a smarter way in the next changesets. Differential Revision: https://phab.mercurial-scm.org/D10665

revlog: pass around the `rev` we deal with when packing/unpacking entry This will help code to compute some flag on the fly. Differential Revision: https://phab.mercurial-scm.org/D10664

changelogv2: use a dedicated version number If we want to change the format, we need a new version number. We start with that. Differential Revision: https://phab.mercurial-scm.org/D10663

changelogv2: `copies-side-data` now implies `changelogv2` To store information about file changes and copies only requires sidedata support for the changelog. So we only enables `changelogv2`. This is less impactful and should help us to ship the changeset centric copy tracing sooner. Differential Revision: https://phab.mercurial-scm.org/D10662

changelogv2: allow upgrade from and to this format Differential Revision: https://phab.mercurial-scm.org/D10661

changelogv2: introduce a "changelogv2" feature Right now, this means using revlogv2, but only for the changelog. We will have the format more unique in future changesets. Differential Revision: https://phab.mercurial-scm.org/D10660

revlogv2: drop the code related to inlined revlogv2 Tevlogv2 will not use inlined revlog. As a result we can drop the associated code. Differential Revision: https://phab.mercurial-scm.org/D10659

revlogv2: no longer attempt to use inline for new revlog revlogv2 steered away for the inline feature. Differential Revision: https://phab.mercurial-scm.org/D10658

sidedata: use the "feature" to detect that sidedata are enabled This is more versatile than relying on the revlogv2 requirements. Differential Revision: https://phab.mercurial-scm.org/D10657

revlog: compress sidedata when doing "post-pull" sidedata update All path writing sidedata are now using compression (when appropriate). Differential Revision: https://phab.mercurial-scm.org/D10656

revlog: compress sidedata in `_writeentry` When appropriate we store the sidedata in a compressed form. Differential Revision: https://phab.mercurial-scm.org/D10655

revlog: apply compression mode while reading sidedata Right now, we only store things as PLAIN, but we are ready for some compression now. Future changeset will start using some. Differential Revision: https://phab.mercurial-scm.org/D10654

revlog: introduce a compression mode for sidedata in the revlog index We will use this for compression for the sidedata payload. Differential Revision: https://phab.mercurial-scm.org/D10653

revlog: implement a "default compression" mode The revlog docker is now storing a default compression engine. When a chunk use that compression, a dedicated mode is used in the revlog entry and we can directly route it to the right decompressor. We should probably make PLAIN and DEFAULT mode the only available mode for revlogv2, but this is something for later. Differential Revision: https://phab.mercurial-scm.org/D10652

revlog: add a `_get_decompressor` method This logic is non-trivial and we will need to reuse it. Differential Revision: https://phab.mercurial-scm.org/D10651

revlog: introduce a plain compression mode That mode is simple it means the chunk contains uncompressed data and can be used directly. Differential Revision: https://phab.mercurial-scm.org/D10650

revlog: use an intermediate variable for `data` in `_chunk` This will help us to dispatch this to the right decompressor. Differential Revision: https://phab.mercurial-scm.org/D10649

revlogv2: preserve the compression mode on disk The value is still the same and still not treated, but now it exists one disk. Differential Revision: https://phab.mercurial-scm.org/D10648

revlog: use `format_version` to dispatch different version in index_get This is semantically more correct. Differential Revision: https://phab.mercurial-scm.org/D10647

revlog: add a "data compression mode" entry in the index tuple That will make it possible to keep track of compression information in the revlog index, opening the way to more efficient revision restoration (in native code, but the python usage is already defeating performance work). We start with adding a new entry to the index tuple, using a value matching the current behavior. We will introduce storage and other value in later changesets. Differential Revision: https://phab.mercurial-scm.org/D10646

revlog: fix the `null_item` attribute for V0 This is not a proper index tuple. Differential Revision: https://phab.mercurial-scm.org/D10645

revlog: create a create `null_item` attribute for V0 This value is currently wrong, but at least it is now -obviously- wrong. Differential Revision: https://phab.mercurial-scm.org/D10644

revlog: improve documentation of the entry tuple The code in revlog, and outside revlog directly use the index's entry tuple, with direct integer indexing. This is a voluntary trade off to obtains better performance from the Python code at the expense of the developers sanity. Let's at least have a clear and central documentation about what this tuple is about. Differential Revision: https://phab.mercurial-scm.org/D10643

revlog: simplify entry update logic in `rewrite_sidedata` We don't need to rebuild the full entry. So lets just gather the data we need to pass to `replace_sidedata_info`. Differential Revision: https://phab.mercurial-scm.org/D10642

revlog: simplify the replace_sidedata_info code We already know how to serialize and de-serialize and enty. So lets just do that and modify the entry tuple directly. This avoid having to duplicated binary operation in complicated code. Differential Revision: https://phab.mercurial-scm.org/D10641

revlog: use `rev` instead of `i` in replace_sidedata_info This is clearer. Differential Revision: https://phab.mercurial-scm.org/D10640

revlogv2: also keep track for the size of the "data" file This is useful to make sure we always start writing at the right location, without effort. Differential Revision: https://phab.mercurial-scm.org/D10632

revlogv2: track pending write in the docket and expose it to hooks The docket is now able to write pending data. We could have used a distinct intermediate files, however keeping everything in the same file will make it simpler to keep track of the various involved files if necessary. However it might prove more complicated for streaming clone. This will be dealt with later. Note that we lifted the stderr redirection in the test since we no longer suffer from "unkown working directory parent" message. Differential Revision: https://phab.mercurial-scm.org/D10631

revlog: move the `trypending` logic from the `changelog` to the `revlog` We move the -reading- logic for the pending's '.a' suffixed index within the revlog class. This is motivated by the fact the logic could be simpler and cleaner if directly handled by the revlog docket. Before we can do so, we need to teach the revlog code about reading "pending" changes. To be honest, we already needed some special casing of the `.a` postfix, so this does not adds much complexity. The logic around -writing- the special '00changelog.i.a' remains in the `changelog` class. Note that the revlog-v2 logic no longer use this logic. The only remaining user of the `postfix` argument is the `censor` logic. We could probably also make the revlog full aware of it (most of the code is already implemented in revlog anyway) and get rid of the `postfix` argument and logic. However this is an adventure for another time. Since we have more information, we add more, paranoid, Programming error in case we detect such "pending reader" trying to do a read (which does not happens anyways). Differential Revision: https://phab.mercurial-scm.org/D10630

revlogv2: delay the update of the changelog docket to transaction end This prevent external reader to see the transaction content before it is commited. However this also prevent the hooks to see the transaction content. We will fix this in later changesets. We have to temporarily suppress the error output of the command ran during the transaction as they sometimes get confused about unknown working directory and sometimes issue message on std-err in unspecified order. Differential Revision: https://phab.mercurial-scm.org/D10629

revlogv2: track current index size in the docket This help use to fix transaction safety on repos. See next changesets for details. Differential Revision: https://phab.mercurial-scm.org/D10628

revlogv2: also test transactionality of revlog v2 Without surprise, this is broken :-) Now that it is tested, we can fix it. Differential Revision: https://phab.mercurial-scm.org/D10627

revlog: add a new test file focussed on testing transactionally issue What we test here is currently also covered by `test-hooks.t`. However having our own test file allow for more focussed/deep testing and to cover more variant when relevant. So we create a new test file. Differential Revision: https://phab.mercurial-scm.org/D10626

revlogv2: store version information in the docket only Having it duplicated in the index was both useless and a risk of discrepancy. Differential Revision: https://phab.mercurial-scm.org/D10625

revlogv2: introduce a very basic docket file This is the first stone toward using a docket file in revlogv2. Right now the docket is very basic and only store the version number (which is -also- stored into the index file…) and the other files have fixed name. This new implementation break transactionally… but they are no test checking transactionally for revlogv2… So I take this as an opportunity to start small. They are no usage of revlogv2 outside of tests anyway. The docket keeps the `.i` naming used by previous version index to preserve a unique entry point. We could decide to use a different name and look it up first, or to fully rework this in a future "store" version. However that does not seems necessary right now. We will re-introduces transactionality (and associated testing…) in a later changesets. A long list of TODOs have been added to the relevant comment. Differential Revision: https://phab.mercurial-scm.org/D10624

branching: merge stable into default

revlogv2: mark revlogv2 as requires a full upgrade This was overlooked previously. Differential Revision: https://phab.mercurial-scm.org/D10623

revlog: unify flag processing when loading index The new code use a simple declaration to do centralised processing. This is clearer, shorter and less error prone. This will be especially useful as we plan to add a fourth format: changelog-v2. Differential Revision: https://phab.mercurial-scm.org/D10622

revlog: unify checks for supported flag The new code use a simple declaration to do centralised checking. This is clearer, shorter and less error prone. This will be especially useful as we plan to add a fourth format: changelog-v2. Differential Revision: https://phab.mercurial-scm.org/D10621

side-data: drop the associated config and requirements This is no longer and extensions of revlog v1, but a part of revlog v2. We no longer needs independent config and requirement management for that. Differential Revision: https://phab.mercurial-scm.org/D10620

revlog: fix capitalisation of an error We don't start error message with capital letters. Differential Revision: https://phab.mercurial-scm.org/D10619

test: no longer directly use the sidedata config and requirements They are on their way out, so we simply use revlogv2 in these cases. Some test does not requires explicite request for the revlogv2 format since the upgrade to a side-data-capable format is impled by `exp-use-copies-side-data-changeset`. Differential Revision: https://phab.mercurial-scm.org/D10618

sidedata: add a 'side-data' repository feature and use it Most code don't really care how sidedata support is added, but it needs to know if it is present. To achieve this. we use the `repo.features` attributes with a new dedicated features". Having such centralised information is more robust and will help us to remove the sidedata requirements in a later changesets. Differential Revision: https://phab.mercurial-scm.org/D10617

statichttp: add the missing `features` attribute This make statichttp closer to the interface it is "supposed" to follow. Differential Revision: https://phab.mercurial-scm.org/D10616

sidedata: use revlogv2 requirement in the test helper We are about to drop the requirement dedicated to sidedata (since revlogv2 will support them unconditionally and previous version will not.). To prepare this more we adapt the test code to not use that requirements. Differential Revision: https://phab.mercurial-scm.org/D10615

upgrade: display sidedata before upgrade This is a small gratuitous change that help making the test simpler to understand on its own. Differential Revision: https://phab.mercurial-scm.org/D10614

revlog: determine sidedata support based on the revlog version Revlog version "2" support sidedata, previous version does not. So lets make the initialization simpler. Differential Revision: https://phab.mercurial-scm.org/D10613

requirements: no longer drop `generaldelta` requirement with revlogv2 A repository could use a mix of revlogv1 and revlogv2, making the requirements still necessary. Overall we should move away from the "requirements" file being used a way to configure the repository and stick to it "what do you need to access this repository". However this is a wider work for another time. In addition the logic we just dropped was confusing the `hg debugformat` command, breaking the upgrade code and inconsistent (eg: `sparse-revlog` is also implied by `revlogv2`). Finally, multiple other config option would imply the use of the `revlogv2` requirements, without drop the `generaldelta` one, leading to more inconsistency. Differential Revision: https://phab.mercurial-scm.org/D10612

config: drop the `format.exp-revlogv2.2` option It is oddly named and not used at all. The one used by the code is still `experimental.revlogv2`. So we drop that one option for consistency. We move associate documentation to the actual option. Differential Revision: https://phab.mercurial-scm.org/D10611

revlog: only use the `_indexfp` method for read operation This will avoid "other" code to not overlook `_writing` usage. We introduces private method dedicated to writing to make use the right option are always used. Differential Revision: https://phab.mercurial-scm.org/D10610

revlog: use `_writing` in `rewrite_sidedata` Special cases aren't special enough to break the rules. Differential Revision: https://phab.mercurial-scm.org/D10609

revlog: open files in 'r+' instead of 'a+' The code doing actual writing is already doing the necessary seeking, so we could safely use 'r+'. This make the file objecs usable in more situation, like updating the sidedata information during pull. revlog: forcibly move the file cursor at the right location before writing This is a paranoid change in case the changelog computation moved the cursors under our feets. This is not known to happens right now. Differential Revision: https://phab.mercurial-scm.org/D10608

revlog: pass a transaction object to `rewrite_sidedata` The `_writing` context need one, so we update the function signature before considering using `_writing` in rewrite_sidedata. Differential Revision: https://phab.mercurial-scm.org/D10607

revlog: rename variable in `rewrite_sidedata` to match other code Let's call the index file object and `ifh` and the data file object `dfh` as the rest of the revlog code. This will make future change clearer. Differential Revision: https://phab.mercurial-scm.org/D10606

revlog: introduce a mandatory `_writing` context to update revlog content Before this change, various revlog methods where managing the opening and closing of the revlog files manually and passing the file descriptor alors the call path. To simplify the tracking of the write operation by a future docket, we need something more organised. As a result, we introduce a `revlog._writing` context manager that will wrap each revlog update operation. The file descriptor are kept in the existing `revlog._writinghandles` parameter that was already used by the `addgroup` logic. All this change is internal to the revlog only, the "public" interface is not affected. The `addrevision` and `addgroup` logic are still responsible for setup up this context. However this new context give us multiple benefits: * all writer use a same, unified, logic, * this context is programmatically enforced, * each write "session" as a clearly identified start and end. The post-pull sidedata update logic is still doing writing by end and will be adjusted in a later changesets. This change affect the concurrency checker test, because register the state of the file in the transaction sooner in `addrevision` (about as early as what `addgroup` would do), so the abort is rollbacking the other commit. I don't want to weaken the current main logic. Differential Revision: https://phab.mercurial-scm.org/D10605

revlog: preindent some code in _enforceinlinesize Indenting this beforehand will make a future changeset much simpler. Differential Revision: https://phab.mercurial-scm.org/D10604

revlog: preindent some code in addgroup Indenting this beforehand will make a future changeset much simpler. Differential Revision: https://phab.mercurial-scm.org/D10603

pyoxidizer: use Python 3.9 (BC) Newer versions of PyOxidizer use Python 3.9 by default. We previously pinned the version to 3.8 to facilitate porting to a new PyOxidizer version and diffing results. Now that the porting work is complete, let's bump Python to Python 3.9. This will effectively change our Windows Inno and WiX Python 3 installers from Python 3.8 to 3.9. .. bc:: Windows .msi and .exe installers now use Python 3.9 instead of Python 3.8. Differential Revision: https://phab.mercurial-scm.org/D10689

packaging: use PyOxidizer for producing WiX MSI installer We recently taught our in-tree PyOxidizer configuration file to produce MSI installers with WiX using PyOxidizer's built-in support for doing so. This commit changes our WiX + PyOxidizer installer generation code to use this functionality. After this change, all the Python packaging code is doing is the following: * Building HTML documentation * Making gettext available to the build process. * Munging CLI arguments to variables for the `pyoxidizer` execution. * Invoking `pyoxidizer build`. * Copying the produced `.msi` to the `dist/` directory. Applying this stack on stable and rebuilding the 5.8 MSI installer produced the following differences from the official 5.8 installer: * .exe and .pyd files aren't byte identical (this is expected). * Various .dist-info/ directories have different names due to older versions of PyOxidizer being buggy and not properly normalizing package names. (The new behavior is correct.) * Various *.dist-info/RECORD files are different due to content divergence of files (this is expected). * The python38.dll differs due to newer PyOxidizer shipping a newer version of Python 3.8. * We now ship python3.dll because PyOxidizer now includes this file by default. * The vcruntime140.dll differs because newer PyOxidizer installs a newer version. We also now ship a vcruntime140_1.dll because newer versions of the redistributable ship 2 files now. The WiX GUIDs and IDs of installed files have likely changed as a result of PyOxidizer's different mechanism for generating those identifiers. This means that an upgrade install of the MSI will replace files instead of doing an incremental update. This is likely harmless and we've incurred this kind of breakage before. As far as I can tell, the new PyOxidizer-built MSI is functionally equivalent to the old method. Once we drop support for Python 2.7 MSI installers, we can delete the WiX code from the repository. This commit temporarily drops support for extra `.wxs` files. We raise an exception instead of silently not using them, which I think is appropriate. We should be able to add support back in by injecting state into pyoxidizer.bzl via `--var`. I just didn't want to expend cognitive load to think about the solution as part of this series. Differential Revision: https://phab.mercurial-scm.org/D10688

packaging: extract invocation of pyoxidizer to own function I'll be refactoring how the WiX installer creation calls into pyoxidizer and will need a lower level function for facilitating that. The new `run_pyoxidizer()` builds our execution environment (with gettext available) and invokes `pyoxidizer`. Differential Revision: https://phab.mercurial-scm.org/D10687

packaging: rename run_pyoxidizer() I'm going to split this function up in a future commit and I'll want the name "run_pyoxidizer()" for a lower-level function for invoking `pyoxidizer`. Differential Revision: https://phab.mercurial-scm.org/D10686

packaging: move documentation HTML building to own function This is part of some light refactoring to enable us to use PyOxidizer for WiX MSI installer generation. Differential Revision: https://phab.mercurial-scm.org/D10685

pyoxidizer: support code signing Newer versions of PyOxidizer feature built-in support for code signing. You simply declare a code signer in the Starlark configuration file, activate it for automatic signing, and PyOxidizer will add code signatures to signable files as it encounters them. This commit teaches our Starlark configuration file to enable automatic code signing. But only on Windows for the moment, as our immediate goal is to overhaul the Windows packaging. The feature is opt-in: you must pass variables to PyOxidizer's build context via `pyoxidizer build --var` or `pyoxidizer build --var-env` to activate code signing. Differential Revision: https://phab.mercurial-scm.org/D10684

pyoxidizer: support producing MSI installers Newer versions of PyOxidizer have support for building WiX MSI installers "natively." Essentially, you can script the definition of your WiX installer via Starlark and PyOxidizer can invoke WiX tools to produce the installer. This commit teaches our PyOxidizer config file to produce MSI installers similarly to how `contrib/packaging/packging.py wix` would do it. We had to make a very minor change to `mercurial.wxs` to reflect different paths depending on who builds. This is because when PyOxidizer builds WiX installers, it does so from an isolated directory, not Mercurial's source directory. We simply copy the files into the build environment so they are accessible. After this change, running `pyoxidizer build msi` produces a nearly identical install layout to what the previous method produces. When I applied this series on top of the 5.8 tag, here is the list of differences and explanations: * docs/*.html files are missing from the new installer because the Python build environment doesn't have docutils. * .pyd and .exe files differ, likely because I'm using a different Visual Studio toolchain on my local computer than the official build environment. * Various .dist-info/ directories have different names. This is because older versions of PyOxidizer had buggy behavior and weren't properly normalizing package names in .dist-info/ directories. e.g. we went from `cached-property-1.5.2.dist-info` to `cached_property-1.5.2.dist-info`. * Translations (.mo files) may be missing if gettext isn't in %Path%. This is because the packaging.py code installs gettext and ensures it can be found. * Some *.dist-info/RECORD files vary due to SHA-256 content digest divergence due to build environment differences. (This should be harmless.) * The new install layout ships a python3.dll because newer versions of PyOxidizer ship this file. * The new install layout has a different vcruntime140.dll and also a vcruntime140_1.dll because newer versions of PyOxidizer ship a newer version of the Visual C++ Redistributable Runtime. The new PyOxidizer functionality is not yet integrated with packaging.py. This will come in a subsequent commit. So for now, the new functionality introduced here is unused. Differential Revision: https://phab.mercurial-scm.org/D10683

pyoxidizer: use allocator_backend instead of raw_allocator The name of this attribute changed in PyOxidizer 0.11. Differential Revision: https://phab.mercurial-scm.org/D10682

pyoxidizer: simplify targets The split targets existed to enable the use of a non-default distribution flavor on Windows. Modern versions of PyOxidizer use the "standalone_dynamic" distribution flavor by default. So our split brain workaround is no longer needed. Here, we unify the targets. We also remove an unreferenced target function to create a resources file. Differential Revision: https://phab.mercurial-scm.org/D10681

contrib: install PyOxidizer 0.16.0 PyOxidizer now provides MSI installers and pre-built Linux binaries. So we install that way. This significantly reduces the time to bootstrap a new machine in automation, as building PyOxidizer from source on a low core count machine takes several minutes. This change temporarily breaks the ability of the automated environment to use the in-repo pyoxidizer.bzl configuration file, as there are backwards-incompatible changes with the upgrade that need to be reflected. We'll handle those in subsequent commits. Differential Revision: https://phab.mercurial-scm.org/D10680

pyoxidizer: pin Python to 3.8 Newer versions of PyOxidizer use version 3.9 by default. As part of upgrading PyOxidizer, we want to pin the version at 3.8 so we can compare differences more easily. Differential Revision: https://phab.mercurial-scm.org/D10679

pyoxidizer: remove some boilerplate in file This isn't used for anything and can safely be removed. Differential Revision: https://phab.mercurial-scm.org/D10678

contrib: upgrade Rust in Windows automation I'm not sure why we don't install the minimum required Rust version here like we do for Linux. Whatever: that's unrelated to wanting to stay modern. Differential Revision: https://phab.mercurial-scm.org/D10677

contrib: update Python versions in Windows automation Let's keep our Python versions modern. Differential Revision: https://phab.mercurial-scm.org/D10676

automation: update rust in Linux environment Our minimum supported Rust is 1.41.1 per rust/README.rst. We also bump the modern Rust version to latest stable to stay current. Differential Revision: https://phab.mercurial-scm.org/D10675

automation: install latest Python versions in Linux environment Let's keep our Linux environment up to date. Differential Revision: https://phab.mercurial-scm.org/D10674

automation: create Python 3.5 variant of requirements.txt The automation environment is refusing to build with the previous file because some dependencies won't install on Python 3.5. I couldn't find an easy way to salvage the situation with a single requirements.txt file. So, I decided to introduce a variant for Python 3.5. As part of this, we update packages to latest versions. (I do question why we are still supporting Python 3.5...) Differential Revision: https://phab.mercurial-scm.org/D10690

hgweb: Fix deprecation warning in Python 3.10 (issue6520) Differential Revision: https://phab.mercurial-scm.org/D10711

dirstate-tree: Remove newly-empty nodes after removing a `DirstateEntry` This is actually necessary to make `DirstateMap::has_dir` correct, since it assumes that a node without a `DirstateEntry` has at least one descedant node with a `DirstateEntry`. This bug would become apparent when a later changeset persists tree nodes on disk in the "dirstate-v2" format. Differential Revision: https://phab.mercurial-scm.org/D10706

dirstate-tree: Refactor DirstateMap::drop_file to be recursive It should behave the same as before. This will enable the next changeset to run code on the way "down" (in order to removing newly-empty nodes). Differential Revision: https://phab.mercurial-scm.org/D10705

templates: fix `revset('parents()') % ...` in amend message template I don't understand why, but putting `revset('parents()') % {desc}` in the commit message template for amend resulted in a crash because `memctx.hex()` did `hex(self.node())` and its node was None. This patch fixes that. Martin von Zweigbergk <martinvonz@google.com> Differential Revision: https://phab.mercurial-scm.org/D10707

path: teach the `hg path` command to display boolean sub-option The next changeset introduce one. Differential Revision: https://phab.mercurial-scm.org/D10450

parselist: move the function from config to stringutil We move the function in a lower level module to avoid cycle. It moves next to `parsebool` who had to migrate for the same reasons. Differential Revision: https://phab.mercurial-scm.org/D10449

bookmark: use `list_paths` to access path definition The content of the `[paths]` config section is receiving transformation that make it hard to recognise whats the actual name, and the next changeset will make it worse. So we use the official API for this instead. Differential Revision: https://phab.mercurial-scm.org/D10448

urlutil: make `paths` class old list of `path` We move from a `{name → path}` mapping to a `{name → [path]}` mapping. And update all user code accordingly. For now, all the list contains exactly one element, but we are now in a good place to make the config understand a list of url. Differential Revision: https://phab.mercurial-scm.org/D10447

urlutil: extract `chain_path` in a function This will no longer modify `path` inplace so it does not make much sense as a method. Differential Revision: https://phab.mercurial-scm.org/D10446

urlutil: add a `copy` method to `path This will be useful when inheriting from multiple path at the same time. Differential Revision: https://phab.mercurial-scm.org/D10445

template: add a `paths` field to all entry in peerurls This make it possible to display multiple path per name in the near future. Differential Revision: https://phab.mercurial-scm.org/D10444

cache: avoid warming the fnodetags cache after clone That cache can quite expensive to compute on large repository as not that `hg clone` is warming all cache, this can introduces a significant slowdown for clone time[1]. As a stop gap measure introduce a quick fix for that on stable, skipping the fnodetags cache post-clone. [1] https://www.mercurial-scm.org/pipermail/mercurial/2021-April/052679.html Differential Revision: https://phab.mercurial-scm.org/D10695

run-tests: fix whitelist/blacklist with directories on Windows The file name is resolved with `os.path.relpath()` in the `Test` constructor, which yields `\` on Windows. That doesn't match the `/` separator when using MSYS tools to build the list, and it isn't obvious that this is the problem because directory separators can mostly be used interchangeably. The `--test-list` argument already seems to be properly handled.

run-tests: ignore PermissionError when checking available ports I'm not sure what this is, but I'm getting it occasionally when running in WSL. When it was raised, none of the tests could run.

status: Add some more tests Trying to improve coverage for various non-ovbious scenarios Differential Revision: https://phab.mercurial-scm.org/D10704

run-tests: use the same python version for shebang lines on Windows The latest py3 is used if the minor number isn't specified. After running the script to install all of the build dependencies, that moved the default from 3.8 to 3.9 on the CI system. That in turn caused a bunch of tests to be skipped that were running prior, even when the test runner was invoked with `py -3.8`. While we should almost always use the latest version, we really shouldn't make it hard to test different versions or allow things to randomly break in subtle ways like that. Differential Revision: https://phab.mercurial-scm.org/D10702

vfs: Fix deprecation warning in Python 3.10 (issue6520) Differential Revision: https://phab.mercurial-scm.org/D10710

contrib: fix typo I forgot to fix it in flight, this commit will do fine.

revlog: rename `indexdata` to entry_data Same reasoning as the previous changeset, we might not be looking at index data here. Differential Revision: https://phab.mercurial-scm.org/D10602

revlog: use "entry_point" phrasing for loading the revlog The main entry for the revlog will not necessary be the index, but a small "docket". So we change the variable names and we move the initialisation of the index_file and data_file after that first entry point have been read. Differential Revision: https://phab.mercurial-scm.org/D10601

revlog: directly use self._format_flags when loading index The shorthand variable does not bring much, so we drop it to simplify the code. Differential Revision: https://phab.mercurial-scm.org/D10600

revlog: directly use self._format_version when loading index The shorthand variable does not bring much, so we drop it to simplify the code. Differential Revision: https://phab.mercurial-scm.org/D10599

revlog: use `_format_flags` to access flags instead of `header` It seems better to reuse the variable we carefully extracted This also open the way to more flexible way to retrieve these flags. Differential Revision: https://phab.mercurial-scm.org/D10598

revlog: rename `newversionflags` to `new_header` This make it consistent with the previous changeset. Differential Revision: https://phab.mercurial-scm.org/D10597

revlog: rename `versionflags` to header The variable is named as such because it contains "version" information and "flags" information. However you mostly needs to know the code to understand the name which is not great. The fact that this is the very first four bytes in all revlog seems more relevant, so we rename the variable "header". Differential Revision: https://phab.mercurial-scm.org/D10596

revlog: move index reading logic in a dedicated method They are multiple motivation to do it: * The logic is complicated enough to deserver its own method. * We will need to reuse this once we put a docket in use. * This split the actual reading from the processing of the read data better. Differential Revision: https://phab.mercurial-scm.org/D10595

revlog: define the actual index and datafile at loading time This is just code movement, to make the code closer to where we actually use it and where it will be defined in the future. Differential Revision: https://phab.mercurial-scm.org/D10594

revlog: simplify a conditionnal in _enforceinlinesize This is a gratuitous change to make things a bit easier to read. Differential Revision: https://phab.mercurial-scm.org/D10593

revlog: drop `flush` parameter from `_peek_iscensored` This is not used anywhere. Differential Revision: https://phab.mercurial-scm.org/D10592

revlog: fix error message when data are missing The error message the message was not using the requested offset, but the adjusted offset to that read more data for improved caching. This resulted in confusing error message. Differential Revision: https://phab.mercurial-scm.org/D10591

revlog: rename `nodemap_file` to `_nodemap_file` Same reasoning as for `indexfile and datafile`, lets hide these implementation details. Differential Revision: https://phab.mercurial-scm.org/D10590

revlog: use revlog.display_id in error related to bad revisions Differential Revision: https://phab.mercurial-scm.org/D10589

revlog: use revlog.display_id in "revision too big" errors Differential Revision: https://phab.mercurial-scm.org/D10588

revlog: use revlog.display_id in censor related errors Differential Revision: https://phab.mercurial-scm.org/D10587

revlog: use revlog.display_id in integrity error Differential Revision: https://phab.mercurial-scm.org/D10586

revlog: use revlog.display_id in ambiguity errors Differential Revision: https://phab.mercurial-scm.org/D10585

revlog: use revlog.display_id for corruption error Differential Revision: https://phab.mercurial-scm.org/D10584

revlog: use revlog.display_id in format related errors Differential Revision: https://phab.mercurial-scm.org/D10583

revlog: use revlog.display_id in narrow error message Differential Revision: https://phab.mercurial-scm.org/D10582

revlog: use revlog.display_id in LookupError Differential Revision: https://phab.mercurial-scm.org/D10581

revlog: use revlog.display_id for FilteredLookupError Differential Revision: https://phab.mercurial-scm.org/D10580

revlog: introduce a `display_id` property We currently using the "index file" to identify a revlog in error output. Since we are about to make the "index file" location more volatile, we need something better. We move to use the "radix", as it is close to what we currently use. We could probably do better, as pointed out in the comment, however that would be a quite detour from my current goal. Differential Revision: https://phab.mercurial-scm.org/D10579

revlog: also use radix when computing nodemap data file We have a radix, lets use it! Differential Revision: https://phab.mercurial-scm.org/D10578

revlog: stop usage of `_indexfile` to computing nodemap path We now have the radix explicitely lets use the radix explicitely Differential Revision: https://phab.mercurial-scm.org/D10577

revlog: use a "radix" to address revlog Instead of pointing to the index directly and to derive the other file from that, we directly provide the radix and let the revlog determine the associated file path internally. This is more robust and will give us more flexibility for picking this file name in the future. Differential Revision: https://phab.mercurial-scm.org/D10576

revlog: rename `datafile` to `datafile` We want to make the actual location of the datafile and location more of an implementation details than what is is currently. In that process, we make the attribute private. Differential Revision: https://phab.mercurial-scm.org/D10575

revlog: rename `indexfile` to `_indexfile` We want to make the actual location of the indexfile and location more of an implementation details than what is is currently. In that process, we make the attribute private. Differential Revision: https://phab.mercurial-scm.org/D10574

filelog: drop `indexfile` from `filelog` Since `filelog` objects are not revlog (no really, they are not…) we drop the revlog specific attribute. We need to directly access the underlying revlog in a couple of place that already assume that we have a revlog here. This is motivated by future change to that revlog attribute. Differential Revision: https://phab.mercurial-scm.org/D10573

manifest: drop the `indexfile` from `manifestrevlog` Since `manifestrevlog` object are not revlog (no really, they are not…) we drop the revlog specific attribute. We need to directly access the underlying revlog in a couple of place that already assume that we have a revlog here. This is motivated by future change to that revlog attribute. Differential Revision: https://phab.mercurial-scm.org/D10572

revlog: deal with special "postfix" explicitely revlog usually use a straight forward '.i' and '.d' naming except for two cases "in-transaction" changelog, and censoring. Our goal is to let the revlog code deal with the internal of the file naming itself. To do so, we need to start dealing with these postfix explicitly. Differential Revision: https://phab.mercurial-scm.org/D10571

revlog: split the option initialisation in its own method The part of the code is huge, keeping it separated will keep the `_loadindex` method simpler and help keeping logic well insulated. Differential Revision: https://phab.mercurial-scm.org/D10570

revlog: always "append" full size tuple Same reasoning as the previous patch. Differential Revision: https://phab.mercurial-scm.org/D10569

revlog: make the index always return the same tuple It is simpler to manage the diferrence in on disk format in the internal index code itself and lets the rest of the code always handle the same object. This will become even more important when the data we store will be entirely different (for example the changelog does not need the "linkrev" field. We start with item reading, we will deal with item writing in the next changesets. Differential Revision: https://phab.mercurial-scm.org/D10568

revlog: introduce an explicit `format_version` member in the index struct This will allow for cleaner check than assuming each version has a different size. Unsurprisingly I am planning to use this to introduce more format variant. Differential Revision: https://phab.mercurial-scm.org/D10567

revlog: rename `hdrsize` to `entry_size` in the C code This is the size of and index entry, so lets make it clearer. Differential Revision: https://phab.mercurial-scm.org/D10566

revlog: split the `version` attribute into its two components The `revlog.version` attribute contained an integer coding 2 different informations: * the revlog version number * a bit field defining some specific feature of the revlog We now explicitly store the two components independently. This avoid exposing the implementation details all around the code and prepare for future revlog version that would encode the information in a different way. In the process we drop the `version` attribute from the interface. It was flagged for removal when that interface was created. Differential Revision: https://phab.mercurial-scm.org/D10565

verify: pass a revlog to `_checkrevlog` in `_verifymanifest` Since `manifestrevlog` is not a `revlog`, we are passing strange thing to `_checkrevlog`. We fix this to avoid breakage during future change. Differential Revision: https://phab.mercurial-scm.org/D10564

revlog: replace flag check related to generaldelta with attribute check Same logic as the previous changesets. Differential Revision: https://phab.mercurial-scm.org/D10563

revlog: replace REVLOGV2 check related to sidedata with `hassidedata` checks This is more flexible and semantically more correct. The associated revlog's attribute exist since 827cb4fe62a3, so well we start linking sidedata to revlogv2. Differential Revision: https://phab.mercurial-scm.org/D10562

revlog: explicitely pass the "indexfile" parameter Most of this was already done when introducing the `target` parameter, but some remained. Having "indexfile" passed explicitely will help us to change the way we address a revlog later in the stack. With the introduction of more generic `docket`, the entry point will not necessarly be `xxx.i` file, and the actual index files will have a variable name. Differential Revision: https://phab.mercurial-scm.org/D10561

revlog: highlight current incompatibility in `rewrite_sidedata` See comment for details. We will need to fix the test coverage when this incompatibility is lifted. Differential Revision: https://phab.mercurial-scm.org/D10544

revlog: adjust rewrite_sidedata code to not delete existing revlog content The "w+" file mode is deleting all the content of the opened file. Which is bad… This is not caught by the test because the test only check for a full, initial pull where not pre-existing content exists. So we need to fix our test coverage here. However they are another issue that prevent "incremental" pull to work here. See next changeset for details. Differential Revision: https://phab.mercurial-scm.org/D10543

status: Add tests for some more edge cases * Size-preserving file contents modification * Filtering output to a deleted or removed file Differential Revision: https://phab.mercurial-scm.org/D10701

status: Extend issue 6483 test to exclude patterns With `hg status -X`, not just include pattern with `hg status -I` Differential Revision: https://phab.mercurial-scm.org/D10700

dirstate-tree: Add a test showing that issue 6335 is fixed … when using the new status algorithm and the tree-based dirstate. The previous algorithm still has this bug. Differential Revision: https://phab.mercurial-scm.org/D10699

dirstate-tree: Add a dirstate-v1-tree variant of some tests The `dirstate-v1` variant has the previous behavior. `dirstate-v1-tree` uses the same format on disk, but uses the new `DirstateMap` with a tree data structure and the new `status` algorithm. These were untested so far. Differential Revision: https://phab.mercurial-scm.org/D10698

merge with stable

rename: add hint about --at-rev if source file doesn't exist It's quite common that users want to record copy (rename) information after committing the working copy changes (i.e. an added and a deleted file). When they try `hg mv [--after] <src> <dst>`, that just fails because the source file doesn't exist. It seems helpful if we can point them to `--at-rev=.` in this case. Differential Revision: https://phab.mercurial-scm.org/D10697

dirstate-tree: Borrow paths from the "on disk" bytes Use std::borrow::Cow to avoid some memory allocations and copying. Differential Revision: https://phab.mercurial-scm.org/D10560

dirstate-tree: Borrow copy source paths from the "on disk" bytes Use std::borrow::Cow to avoid some memory allocations and copying. These particular allocations are not visible when profiling (as many files in a typical repo don’t have a copy source). This change is "warm up" for doing the same with paths of files themselves, which is more involved since those paths are used as `HashMap` keys. This gets of the way the addition of a lifetime parameter to several types. Differential Revision: https://phab.mercurial-scm.org/D10559

rust: Use `&HgPath` instead of `&HgPathBuf` in may APIs Getting the former (through `Deref`) is almost the only useful thing one can do with the latter anyway. With this changes, API become more flexible for the "provider" of these paths which may store something else that Deref’s to HgPath, such as `std::borrow::Cow<HgPath>`. Using `Cow` can help reduce memory alloactions and copying. Differential Revision: https://phab.mercurial-scm.org/D10558

dirstate-tree: Make `DirstateMap` borrow from a bytes buffer … that has the contents of the `.hg/dirstate` file. This only applies to the tree-based flavor of `DirstateMap`. For now only the entire `&[u8]` slice is stored, so this is not useful yet. Adding a lifetime parameter to the `DirstateMap` struct (in hg-core) makes Python bindings non-trivial because we keep that struct in a Python object that has a dynamic lifetime tied to Python’s reference-counting and GC. As long as we keep the `PyBytes` that owns the borrowed bytes buffer next to the borrowing struct, the buffer will live long enough for the borrows to stay valid. However this relationship cannot be expressed in safe Rust code in a way that would statisfy they borrow-checker. We use `unsafe` code to erase that lifetime parameter, and encapsulate it in a safe abstraction similar to the owning-ref crate: https://docs.rs/owning_ref/ Differential Revision: https://phab.mercurial-scm.org/D10557

rust: Read dirstate from disk in DirstateMap constructor Before this changeset, Python code first creates an empty `DirstateMap` Rust object, then immediately calls its `read` method with a byte string of the contents of the `.hg/dirstate` file. This makes that byte string available to the constructor of `DirstateMap` in the hg-cpython crate. This is a first step towards enabling parts of `DirstateMap` in the hg-core crate to borrow from this buffer without copying. Differential Revision: https://phab.mercurial-scm.org/D10556

rust: Remove handling of `parents` in `DirstateMap` The Python wrapper class `dirstatemap` can take care of it. This removes the need to have both `_rustmap` and `_inner_rustmap`. Differential Revision: https://phab.mercurial-scm.org/D10555

dirstate-tree: Fold "tracked descendants" counter update in main walk For the purpose of implementing `has_tracked_dir` (which means "has tracked descendants) without an expensive sub-tree traversal, we maintaing a counter of tracked descendants on each "directory" node of the tree-shaped dirstate. Before this changeset, mutating or inserting a node at a given path would involve: * Walking the tree from root through ancestors to find the node or the spot where to insert it * Looking at the previous node if any to decide what counter update is needed * Performing any node mutation * Walking the tree *again* to update counters in ancestor nodes When profiling `hg status` on a large repo, this second walk takes times while loading a the dirstate from disk. It turns out we have enough information to decide before he first tree walk what counter update is needed. This changeset merges the two walks, gaining ~10% of the total time for `hg update` (in the same hyperfine benchmark as the previous changeset). --- Profiling was done by compiling with this `.cargo/config`: [profile.release] debug = true then running with: py-spy record -r 500 -n -o /tmp/hg.json --format speedscope -- \ ./hg status -R $REPO --config experimental.dirstate-tree.in-memory=1 then visualizing the recorded JSON file in https://www.speedscope.app/ Differential Revision: https://phab.mercurial-scm.org/D10554

dirstate-tree: Use HashMap instead of BTreeMap BTreeMap has the advantage of its "natural" iteration order being the one we need in the status algorithm. With HashMap however, iteration order is undefined so we need to allocate a Vec and sort it explicitly. Unfortunately many BTreeMap operations are slower than in HashMap, and skipping that extra allocation and sort is not enough to compensate. Switching to HashMap + sort makes `hg status` 17% faster in one test case, as measure with hyperfine: ``` Benchmark #1: ../hg2/hg status -R $REPO --config=experimental.dirstate-tree.in-memory=1 Time (mean ± σ): 765.0 ms ± 8.8 ms [User: 1.352 s, System: 0.747 s] Range (min … max): 751.8 ms … 778.7 ms 10 runs Benchmark #2: ./hg status -R $REPO --config=experimental.dirstate-tree.in-memory=1 Time (mean ± σ): 651.8 ms ± 9.9 ms [User: 1.251 s, System: 0.799 s] Range (min … max): 642.2 ms … 671.8 ms 10 runs Summary './hg status -R $REPO --config=experimental.dirstate-tree.in-memory=1' ran 1.17 ± 0.02 times faster than '../hg2/hg status -R $REPO --config=experimental.dirstate-tree.in-memory=1' ``` * ./hg is this revision * ../hg2/hg is its parent * $REPO is an old snapshot of mozilla-central Differential Revision: https://phab.mercurial-scm.org/D10553

dirstate-tree: Add #[timed] attribute to `status` and `DirstateMap::read` When running with a `RUST_LOG=trace` environment variable, the `micro_timer` crate prints the duration taken by each call to functions with that attribute. Differential Revision: https://phab.mercurial-scm.org/D10552

dirstate-tree: Paralellize the status algorithm with Rayon The `rayon` crate exposes "parallel iterators" that work like normal iterators but dispatch work on different items to an implicit global thread pool. Differential Revision: https://phab.mercurial-scm.org/D10551

dirstate-tree: Avoid BTreeMap double-lookup when inserting a dirstate entry The child nodes of a given node in the tree-shaped dirstate are kept in a `BTreeMap` where keys are file names as strings. Finding or inserting a value in the map takes `O(log(n))` string comparisons, which adds up when constructing the tree. The `entry` API allows finding a "spot" in the map that may or may not be occupied and then access that value or insert a new one without doing map lookup again. However the current API is limited in that calling `entry` requires an owned key (and so a memory allocation), even if it ends up not being used in the case where the map already has a value with an equal key. This is still a win, with 4% better end-to-end time for `hg status` measured here with hyperfine: ``` Benchmark #1: ../hg2/hg status -R $REPO --config=experimental.dirstate-tree.in-memory=1 Time (mean ± σ): 1.337 s ± 0.018 s [User: 892.9 ms, System: 437.5 ms] Range (min … max): 1.316 s … 1.373 s 10 runs Benchmark #2: ./hg status -R $REPO --config=experimental.dirstate-tree.in-memory=1 Time (mean ± σ): 1.291 s ± 0.008 s [User: 853.4 ms, System: 431.1 ms] Range (min … max): 1.283 s … 1.309 s 10 runs Summary './hg status -R $REPO --config=experimental.dirstate-tree.in-memory=1' ran 1.04 ± 0.02 times faster than '../hg2/hg status -R $REPO --config=experimental.dirstate-tree.in-memory=1' ``` * ./hg is this revision * ../hg2/hg is its parent * $REPO is an old snapshot of mozilla-central Differential Revision: https://phab.mercurial-scm.org/D10550

dirstate-tree: Handle I/O errors in status Errors such as insufficient permissions when listing a directory are logged, and the algorithm continues without considering that directory. Differential Revision: https://phab.mercurial-scm.org/D10549

dirstate-tree: Ignore FIFOs etc. in the status algorithm If a filesystem directory contains anything that is not: * a "normal" file * a symbolic link * or a directory … act as if that directory entry was not there. For example, if that path was previously a tracked file, mark it as deleted or removed. Differential Revision: https://phab.mercurial-scm.org/D10548

dirstate-tree: Add the new `status()` algorithm With the dirstate organized in a tree that mirrors the structure of the filesystem tree, we can traverse both trees at the same time in order to compare them. This is hopefully more efficient that building multiple big hashmaps for all of the repository’s contents. Differential Revision: https://phab.mercurial-scm.org/D10547

dirstate-tree: Give to `status()` mutable access to the `DirstateMap` Differential Revision: https://phab.mercurial-scm.org/D10546

rust: Add doc-comments to DirstateStatus fields Differential Revision: https://phab.mercurial-scm.org/D10495

rust: Move "lookup" a.k.a. "unsure" paths into `DirstateStatus` struct Instead of having `status()` returning a tuple of those paths and `DirstateStatus`. Differential Revision: https://phab.mercurial-scm.org/D10494

rust: Remove DirstateMap::file_fold_map This was a HashMap constructed on demand and then cached in the DirstateMap struct to avoid reconstructing at the next access. However the only use is in Python bindings converting it to a PyDict. That method in turn is wrapped in a @cachedproperty in Python code. This was two redudant layers of caching. This changeset removes the Rust-level one to keep the Python dict cache, and have bindings create a PyDict by iterating. Differential Revision: https://phab.mercurial-scm.org/D10493

dirstate-tree: Add "non normal" and "from other parent" sets Unlike the other DirstateMap implementation, these sets are not materialized separately in memory. Instead we traverse the main tree. Differential Revision: https://phab.mercurial-scm.org/D10492

dirstate-tree: Add add_file, remove_file, and drop_file Again, various counters need to be kept up to date. Differential Revision: https://phab.mercurial-scm.org/D10491

dirstate-tree: Add has_dir and has_tracked_dir A node without a `DirstateMap` entry represents a directory. Only some values of `EntryState` represent tracked files. A directory is considered "tracked" if it contains any descendant file that is tracked. To avoid a sub-tree traversal in `has_tracked_dir` we add a counter for this. A boolean flag would become insufficent when we implement remove_file and drop_file. `add_file_node` is more general than needed here, in anticipation of adding the `add_file` and `remove_file` methods. Differential Revision: https://phab.mercurial-scm.org/D10490

dirstate-tree: Add clear_ambiguous_times in the new DirstateMap Also drive-by refactor it in the other DirstateMap Differential Revision: https://phab.mercurial-scm.org/D10489

dirstate-tree: Add copy_map_insert and copy_map_remove Differential Revision: https://phab.mercurial-scm.org/D10488

dirstate-tree: Maintain a counter of DirstateEntry’s and copy sources This allows implementing __len__ for DirstateMap and CopyMap efficiently, without traversing the tree. Differential Revision: https://phab.mercurial-scm.org/D10487

dirstate-tree: Serialize to disk The existing `pack_dirstate` function relies on implementation details of `DirstateMap`, so extract some parts of it as separate functions for us in the tree-based `DirstateMap`. The `bytes-cast` crate is updated to a version that has an `as_bytes` method, not just `from_bytes`: https://docs.rs/bytes-cast/0.2.0/bytes_cast/trait.BytesCast.html#method.as_bytes Drive-by refactor `clear_ambiguous_times` which does part of the same thing. Differential Revision: https://phab.mercurial-scm.org/D10486

rust: Add a Timestamp struct instead of abusing Duration `SystemTime` would be the standard library type semantically appropriate instead of `Duration`. But since the value is coming from Python as a plain integer and used in dirstate packing code as an integer, let’s make a type that contains a single integer instead of using one with sub-second precision. Differential Revision: https://phab.mercurial-scm.org/D10485

dirstate-tree: Add tree traversal/iteration Like Python’s, Rust’s iterators are "external" in that they are driven by a caller who calls a `next` method. This is as opposed to "internal" iterators who drive themselves and call a callback for each item. Writing an internal iterator traversing a tree is easy with recursion, but internal iterators cannot rely on the call stack in that way, they must save in an explicit object all state that they need to be preserved across two `next` calls. This algorithm uses a `Vec` as a stack that contains what would be local variables on the call stack if we could use recursion. Differential Revision: https://phab.mercurial-scm.org/D10370

dirstate-tree: Add map `get` and `contains_key` methods Differential Revision: https://phab.mercurial-scm.org/D10369

dirstate-tree: Add parsing only dirstate parents from disk Differential Revision: https://phab.mercurial-scm.org/D10368

dirstate-tree: Implement DirstateMap::read This reads the on-disk dirstate in its current format (a flat sequence of entries) and creates a tree in memory. Differential Revision: https://phab.mercurial-scm.org/D10367

dirstate-tree: Add `WithBasename` wrapper for `HgPath` In the tree-shaped dirstate we want to have nodes representing files or directories, where directory nodes contain a map associating "base" names to child nodes for child files and directories. Many dirstate operations expect a full path from the repository root, but re-concatenating string from nested map keys all the time might be expensive. Instead, `WithBasename` stores a full path for these operations but behaves as its base name (last path component) for equality and comparison. Additionally `inclusive_ancestors` provides the successive map keys that are needed when inserting a new dirstate node at a given full path. Differential Revision: https://phab.mercurial-scm.org/D10365

dirstate-tree: Empty shell for a second Rust DirstateMap implementation For background see description of the previous changeset "Make Rust DirstateMap bindings go through a trait object". Add an empty shell for a opt-in second Rust implementation of the `DirstateMap` type and the `status` function. For now all methods panic. This can be seen in "action" with: ./hg status --config experimental.dirstate-tree.in-memory=1 Differential Revision: https://phab.mercurial-scm.org/D10364

dirstate-tree: Abstract "non-normal" and "other parent" sets Instead of exposing `HashSet`s directly, have slightly higher-level methods for the operations that Python bindings need on them. Differential Revision: https://phab.mercurial-scm.org/D10363

dirstate-tree: Make Rust DirstateMap bindings go through a trait object This changeset starts a series that adds an experiment to make status faster by changing the dirstate (first only in memory and later also on disk) to be shaped as a tree matching the directory structure, instead of the current flat collection of entries. The status algorithm can then traverse this tree dirstate at the same time as it traverses the filesystem. We (Octobus) have made prototypes that show promising results but are prone to bitrot. We would like to start upstreaming some experimental Rust code that goes in this direction, but to avoid disrupting users it should only be enabled by some run-time opt-in while keeping the existing dirstate structure and status algorithm as-is. The `DirstateMap` type and `status` function look like the appropriate boundary. This adds a new trait that abstracts everything Python bindings need and makes those bindings go through a `dyn` trait object. Later we’ll have two implementations of this trait, and the same bindings can use either. Differential Revision: https://phab.mercurial-scm.org/D10362

remotefilelog: use the correct capability when using getfilestype threaded The functon was overlooked when the capability was renamed Differential Revision: https://phab.mercurial-scm.org/D10673

test-copies: test that copies' sidedata can get computed during push If the source of the push does not have the necessary sidedata but the server needs them, the client should compute them on the fly while pushing. Differential Revision: https://phab.mercurial-scm.org/D10350

test-copies: test that copies' sidedata can get computed during pull If the source does not have the data, the pulling client should compute the necessary side-data while pulling. Differential Revision: https://phab.mercurial-scm.org/D10349

test-copies: test that copies' sidedata does not get corrupted during push This is an important usecase. Differential Revision: https://phab.mercurial-scm.org/D10348

test-copies: test that copies' sidedata does not get corrupted during pull This is an important usecase. Differential Revision: https://phab.mercurial-scm.org/D10347

test-copies: simplify some conditional output Now that all computation using sidedata give the same result, we can simplify conditional Differential Revision: https://phab.mercurial-scm.org/D10346

sidedata: move documentation about sidedata helpers to sidedata module Differential Revision: https://phab.mercurial-scm.org/D10361

sidedata: move sidedata-related utils to the dedicated module Differential Revision: https://phab.mercurial-scm.org/D10360

sidedata: replace sidedata upgrade mechanism with the new one Note: this is split into a separate change (like some other patches in this series) because it's not easy to have all patches work 100% and this seemed easier for reviewers. When cloning or upgrading a repo, we may need to compute (or remove) sidedata. This is the same mechanism that is used in exchange, so we re-use the new system to simplify the code and fix the remaining issues (correctly dropping flags and handling partial removal, etc.). This also highlighted an issue with `test-copies-in-changeset.t` that kept sidedata categories that are not relevant anymore. They should probably be dropped entirely, but that would be for another patch. Differential Revision: https://phab.mercurial-scm.org/D10359

sidedata: add a way of replacing an existing sidedata computer This will be useful in a future patch to replace a sequential computer with a parallel computer. We only allow for explicit replacement, to force the users to think about overriding computers. Differential Revision: https://phab.mercurial-scm.org/D10358

bundle2: remove restriction around sidedata We are now capable of generating the missing sidedata on-the-fly. Differential Revision: https://phab.mercurial-scm.org/D10345

heptapod-ci: enable pytype checking The trigger is manual for now, until we get a better idea about stability, resource usage, etc. Differential Revision: https://phab.mercurial-scm.org/D10462

hghave: fix the definition of `python3` to work on Windows Both py2 and py3 executables are named `python.exe`, and may or may not be on PATH. So use the dispatcher executable that comes with py3 to fetch the version of the latest py3 executable. This allows at least one relnotes test to run on Windows. Differential Revision: https://phab.mercurial-scm.org/D10694

util: avoid echoing the password to the console on Windows py3 (issue6446) The `getpass.getpass()` implementation on Windows first checks if `sys.stdin` and `sys.__stdin__` are the same object. It's not on py3 because the former is replaced in dispatch.py with something that doesn't normalize '\n' to '\r\n'. When they aren't the same object, it simply calls `sys.stdin.readline()` instead of the mscvrt functions that read the input characters before they are echoed. This simply copies the `getpass.win_getpass()` implementation without the stdin check, and byteifies around the edges. I'm not sure if there's a reasonable replacement for the check that we could implement. When echoing input into the hg command, the `ui.interactive()` check causes `ui.getpass()` to bail before getting here. If the proper config switches are used to bypass that and call this, the process stalls until '\n' is input into the console. So there could be a deadlock here when run by another command if the wrong config settings are applied. Differential Revision: https://phab.mercurial-scm.org/D10708

sidedata: enable sidedata computers to optionally rewrite flags Sidedata computers may want to influence the flags of the revision they touch. For example, the computer for changelog-based copytracing can add a flag to signify that this revision might affect copytracing, inversely removing said flag if the information is no longer applicable. See inline documentation in `storageutil` for more details. Differential Revision: https://phab.mercurial-scm.org/D10344

cg4: introduce protocol flag to signify the presence of sidedata We need a way of signaling whether the current revision has sidedata or not, and re-using the revision flags would waste potential revlog flags and mix two normally independent layers. In this change, we add a single byte at the start of the ch4 delta header to set potential protocol flags. We also reclaim the revlog flag for sidedata, since it is no longer used, in its place now lives the (also experimental) copytracing flag. When generating deltas, apply the `CG_FLAG_SIDEDATA` flag if there is sidedata. When applying the deltas, if said flag is present, the next chunk contains the sidedata. Differential Revision: https://phab.mercurial-scm.org/D10343

changegroup: don't limit cgv4 to revlogv2 repos To help the transition from revlogv1 to revlogv2, we need to be able to enable cgv4 for revlogv1 repos, so that revlogv2 clients can handle adding/removing sidedata over the wire. Differential Revision: https://phab.mercurial-scm.org/D10342

sidedata: gate sidedata functionality to revlogv2 in more places Since revlogv1 is not capable of storing sidedata, we prevent sidedata mechanisms around the revlog layer from doing anything. We however keep the ones that allow a revlogv1 repo to generate sidedata on-the-fly on a push, the pull case simply does not add the sidedata to the revlog. Differential Revision: https://phab.mercurial-scm.org/D10341

sidedata: register copies sidedata computer regardless of the revlog version Repositories should not gate their sidedata computers based on any requirement, only their wanted sidedata. A repository might need to generate sidedata wanted by the peer that it itself does not want. Differential Revision: https://phab.mercurial-scm.org/D10340

revlog: replace the old `revlog_kind` approach with the new `target` one The new `target` attribute supersedes the previous one. Differential Revision: https://phab.mercurial-scm.org/D10353

revlog: introduce an explicit tracking of what the revlog is about Since the dawn of time, people have been forced to rely to lossy introspection of the index filename to determine what the purpose and role of the revlog they encounter is. This is hacky, error prone, inflexible, abstraction-leaky, <insert-your-own-complaints-here>. In f63299ee7e4d Raphaël introduced a new attribute to track this information: `revlog_kind`. However it is initialized in an odd place and various instances end up not having it set. In addition is only tracking some of the information we end up having to introspect in various pieces of code. So we add a new attribute that holds more data and is more strictly enforced. This work is done in collaboration with Raphaël. The `revlog_kind` one will be removed/adapted in the next changeset. We expect to be able to clean up various existing piece of code and to simplify coming work around the newer revlog format. Differential Revision: https://phab.mercurial-scm.org/D10352

config: add --source option to include source of value Showing the source of each config option is quite useful and not something the user should have to reach for the `--debug` flag for. I updates documentation and tests, except for one place in `test-hgrc.t` where I thought the test might have been intended to also test that `--debug` results in `ui.quiet` etc being test. Differential Revision: https://phab.mercurial-scm.org/D10668

rewriteutil: say how many commits would become orphan if commit is rewritten This copies the message from the evolve extension, but modifies it a bit to work with the grammar (in particular with the use of "change branch of" as `action`). I don't know why it doesn't use the same `_formatrevs()` as for public commmits. Differential Revision: https://phab.mercurial-scm.org/D10671

rewriteutil: give examples of public changesets that can't be rewritten This patch copies the feature from the evolve extension. Differential Revision: https://phab.mercurial-scm.org/D10670

convert: update p4 for Python 3 Differential Revision: https://phab.mercurial-scm.org/D10703

tests: change the fixer commands to use the buffer attribute on stdio objects Otherwise `\r` was getting injected into the fixed lines and throwing off the commit hashes on Windows when the fixer is invoked with py3. Differential Revision: https://phab.mercurial-scm.org/D10637

tests: stabilize test-persistent-nodemap.t on Windows Several issues here: - Hooks can't invoke shell scripts on Windows, so use `sh` to launch - `dd` in MSYS only recognizes `status=noxfer` - The `PATH` updating triggered a massive slowdown, but is no longer needed I have no idea why, but removing the `PATH` update substantially increased the speed of the test. It was running finishing at ~4:30 with `--debug` and ~14:50 without it, but now completes in ~2:20 on my Windows laptop. Differential Revision: https://phab.mercurial-scm.org/D10636

tests: invoke some shell scripts through the shell interpreter for Windows Otherwise, Windows was prompting what program to use to open the file (or just opening it if there was a file association configured). Differential Revision: https://phab.mercurial-scm.org/D10635

tests: run python script through quoted interpreter instead of directly This helps Windows when python is installed to %PROGRAMFILES%. Differential Revision: https://phab.mercurial-scm.org/D10634

tests: ensure `$PYTHON` is quoted for Windows Global installs of python3 go into "Program Files", and tons of tests fail with mysterious errors if this isn't quoted. Most of this is a followup to 0826d684a1b5, but a some of these were existing issues. Shebang lines are ignored because quoting breaks direct execution- these will need to be launched indirectly with the quoted `$PYTHON` command. Differential Revision: https://phab.mercurial-scm.org/D10633

black: make codebase compatible with black v21.4b2 and v20.8b1 I don't know what exact version of black made it care about these whitespace differences, but this is the version I got when I just installed it with `pip3 install black`. I'm intentionally not increasing the version of black required, as I don't want to force everyone to upgrade their version of black, and these fixes are backwards compatible with black v20.8b1. If there are more issues in the future and this becomes a maintenance burden I may do so in a future change. Tested with both versions of black (I got the older version via `pip3 install black==20.8b1`) Differential Revision: https://phab.mercurial-scm.org/D10539

debugcommands: fix some plural-agreements I noticed Differential Revision: https://phab.mercurial-scm.org/D10669

merge: with stable

Added signature for changeset 067f2c53fb24

Added tag 5.8 for changeset 067f2c53fb24

branching: merge stable into default

phabricator: adapt to the new `urlutil.url()` API This avoids a bunch of deprecation warnings in the tests. Differential Revision: https://phab.mercurial-scm.org/D10541

extensions: ignore exceptions from an extension's `getversion()` method This method is usually called when there's a stacktrace being generated, or with `hg version -v`. Raising another exception risks mangling the bug report info. I hit this issue when trying to add the method to the keyring extension to report the version of the extension and the underlying module, and ran into demandimport issues prior to py3.8. It seems like a wise thing to do anyway, though unfortunately there's no convenient `ui` object around to issue a warning. Use 'unknown' to signal that it tried to report a version and failed, unlike the default case of printing nothing. Differential Revision: https://phab.mercurial-scm.org/D10540

git: ensure all dirstate state values are bytes I'm not sure how this particular git status occurs, but after the fallout of issue 6510 and getting into the issue 6511 state where `git status` shows the files as modified in both the "to be committed" and "not staged" lists, `hg diff` was crashing in `workingctx.__contains__()`. Differential Revision: https://phab.mercurial-scm.org/D10532

tests: synchronize the git and Mercurial username The problem with the default name of "test" set by the test runner is the stringutil methods are unable to split out separate user and email addresses that git wants. This means the username is recorded in git as "test <test>". Amending a commit with that user ends up trying to use "<test>" as the person field for the new commit, and the git library complains about the angle brackets. We should probably abort with a clearer message any time this bad form is used with the git extension. One of the commit dates is tweaked to recreate the ambiguous hash prefix from before. Differential Revision: https://phab.mercurial-scm.org/D10531

git: initialize `extra` to have at least the branch name for nullid Otherwise, this crashes trying to convert to local encoding: ... File "/mnt/c/Users/Matt/hg/mercurial/logcmdutil.py", line 333, in _show branch = ctx.branch() File "/mnt/c/Users/Matt/hg/mercurial/context.py", line 675, in branch return encoding.tolocal(self._changeset.extra.get(b"branch")) File "/mnt/c/Users/Matt/hg/mercurial/encoding.py", line 181, in tolocal if isasciistr(s): TypeError: a bytes-like object is required, not 'NoneType' This was originally reported to the thg bug tracker. https://foss.heptapod.net/mercurial/tortoisehg/thg/-/issues/5629 Differential Revision: https://phab.mercurial-scm.org/D10528

git: consistently use str for parents when rebuilding the index database The tests show no changes, but when these values are overwritten shortly after when a git commit object is available, that uses str. It seems better to use that for consistency. It does materially affect the database though, because the old value stored was `X'3030..3030'` and is now '00..00' when the changelog table is dumped in sqlite3. There is one query that specifies the parents, but it passes the non null hashes as str, so it worked as expected. That likely explains the lack of test changes. Differential Revision: https://phab.mercurial-scm.org/D10527

git: pass the correct type to the sqlite3 query for baselog.hasnode() It looks like this function is mostly used in exchange (which isn't supported now) and histedit (which crashes). But I did verify in a REPL that passing bytes to `db.execute()` fails to find the row, whereas passing str works. Differential Revision: https://phab.mercurial-scm.org/D10526

git: use the correct type for stopping changelog.revs() The `tip` function returns a binary node, but the database is expecting an int. Differential Revision: https://phab.mercurial-scm.org/D10525

git: use the proper filenode for deleted files in changelog.changelogrevision Inline printing shows it still doesn't find the removed file in the test, but it is at least a str instead of bytes like the immediate query before it that does find files. Looking at the database, it doesn't look like the remove was recorded. Additionally, `hg log -r 'removes("re:.*")'` stacktraces. Differential Revision: https://phab.mercurial-scm.org/D10524

tests: add coverage for git.changelog.headrevs(...) Differential Revision: https://phab.mercurial-scm.org/D10523

git: make changelog.tiprev() return int instead of tuple (issue6510) Differential Revision: https://phab.mercurial-scm.org/D10522

git: fix partial node matching Looks like everything is meant to be a str here. Differential Revision: https://phab.mercurial-scm.org/D10521

tests: stabilize test-git-interop.t for some versions of git Not sure why this differs on my system, but it's likely due to running 2.17.1 on Ubuntu 18.04. Differential Revision: https://phab.mercurial-scm.org/D10520

manifests: push down expected node length into the parser This strictly enforces the node length in the manifest lines according to what the repository expects. One test case moves large hash testing into the non-treemanifest part as treemanifests don't provide an interface for overriding just the node length for now. Differential Revision: https://phab.mercurial-scm.org/D10533

core: don't hard-code node length Differential Revision: https://phab.mercurial-scm.org/D10536

core: don't hard-code hex node lengths Differential Revision: https://phab.mercurial-scm.org/D10535

tests: bump default timeout to 360s A number of tests hit or almost hit the default limit even on modern hardware. While the tests are ideally split into smaller pieces, that's non-trivial work. HyperThreading and similar technologies can trigger this often, even without any other load on the machine. Differential Revision: https://phab.mercurial-scm.org/D10534

enforcesinglehead-test: add the expected node output next to the error this make it simpler to validate that the test is correct. Differential Revision: https://phab.mercurial-scm.org/D10545

revlog: code for `revlogv0` in its own module This code is mostly unused compatiblity code. Yet it take a prohiminent place in the `revlog.py` module. That module is already quite big, so we move all that code in a dedicated module. Differential Revision: https://phab.mercurial-scm.org/D10511

revlog: have an explicit "pack_header" method Having to pass the version header when retrieving the binary version of every single entry is a bit silly. So we extract that special logic in its own method. This also prepare the move to newer revlog format, not storing the header within an actual entry… Differential Revision: https://phab.mercurial-scm.org/D10510

revlog: remove the revlogio class The class only contains a single `parseindex` method. Lets just make it a function and remove the `revlogio` class. It served us well but became thinner and thinner overtime. Differential Revision: https://phab.mercurial-scm.org/D10509

revlog: fix some comment style They displease check-code. Differential Revision: https://phab.mercurial-scm.org/D10542