pyoxidizer: use Python 3.9 (BC) Newer versions of PyOxidizer use Python 3.9 by default. We previously pinned the version to 3.8 to facilitate porting to a new PyOxidizer version and diffing results. Now that the porting work is complete, let's bump Python to Python 3.9. This will effectively change our Windows Inno and WiX Python 3 installers from Python 3.8 to 3.9. .. bc:: Windows .msi and .exe installers now use Python 3.9 instead of Python 3.8. Differential Revision: https://phab.mercurial-scm.org/D10689

packaging: use PyOxidizer for producing WiX MSI installer We recently taught our in-tree PyOxidizer configuration file to produce MSI installers with WiX using PyOxidizer's built-in support for doing so. This commit changes our WiX + PyOxidizer installer generation code to use this functionality. After this change, all the Python packaging code is doing is the following: * Building HTML documentation * Making gettext available to the build process. * Munging CLI arguments to variables for the `pyoxidizer` execution. * Invoking `pyoxidizer build`. * Copying the produced `.msi` to the `dist/` directory. Applying this stack on stable and rebuilding the 5.8 MSI installer produced the following differences from the official 5.8 installer: * .exe and .pyd files aren't byte identical (this is expected). * Various .dist-info/ directories have different names due to older versions of PyOxidizer being buggy and not properly normalizing package names. (The new behavior is correct.) * Various *.dist-info/RECORD files are different due to content divergence of files (this is expected). * The python38.dll differs due to newer PyOxidizer shipping a newer version of Python 3.8. * We now ship python3.dll because PyOxidizer now includes this file by default. * The vcruntime140.dll differs because newer PyOxidizer installs a newer version. We also now ship a vcruntime140_1.dll because newer versions of the redistributable ship 2 files now. The WiX GUIDs and IDs of installed files have likely changed as a result of PyOxidizer's different mechanism for generating those identifiers. This means that an upgrade install of the MSI will replace files instead of doing an incremental update. This is likely harmless and we've incurred this kind of breakage before. As far as I can tell, the new PyOxidizer-built MSI is functionally equivalent to the old method. Once we drop support for Python 2.7 MSI installers, we can delete the WiX code from the repository. This commit temporarily drops support for extra `.wxs` files. We raise an exception instead of silently not using them, which I think is appropriate. We should be able to add support back in by injecting state into pyoxidizer.bzl via `--var`. I just didn't want to expend cognitive load to think about the solution as part of this series. Differential Revision: https://phab.mercurial-scm.org/D10688

packaging: extract invocation of pyoxidizer to own function I'll be refactoring how the WiX installer creation calls into pyoxidizer and will need a lower level function for facilitating that. The new `run_pyoxidizer()` builds our execution environment (with gettext available) and invokes `pyoxidizer`. Differential Revision: https://phab.mercurial-scm.org/D10687

packaging: rename run_pyoxidizer() I'm going to split this function up in a future commit and I'll want the name "run_pyoxidizer()" for a lower-level function for invoking `pyoxidizer`. Differential Revision: https://phab.mercurial-scm.org/D10686

packaging: move documentation HTML building to own function This is part of some light refactoring to enable us to use PyOxidizer for WiX MSI installer generation. Differential Revision: https://phab.mercurial-scm.org/D10685

pyoxidizer: support code signing Newer versions of PyOxidizer feature built-in support for code signing. You simply declare a code signer in the Starlark configuration file, activate it for automatic signing, and PyOxidizer will add code signatures to signable files as it encounters them. This commit teaches our Starlark configuration file to enable automatic code signing. But only on Windows for the moment, as our immediate goal is to overhaul the Windows packaging. The feature is opt-in: you must pass variables to PyOxidizer's build context via `pyoxidizer build --var` or `pyoxidizer build --var-env` to activate code signing. Differential Revision: https://phab.mercurial-scm.org/D10684

pyoxidizer: support producing MSI installers Newer versions of PyOxidizer have support for building WiX MSI installers "natively." Essentially, you can script the definition of your WiX installer via Starlark and PyOxidizer can invoke WiX tools to produce the installer. This commit teaches our PyOxidizer config file to produce MSI installers similarly to how `contrib/packaging/packging.py wix` would do it. We had to make a very minor change to `mercurial.wxs` to reflect different paths depending on who builds. This is because when PyOxidizer builds WiX installers, it does so from an isolated directory, not Mercurial's source directory. We simply copy the files into the build environment so they are accessible. After this change, running `pyoxidizer build msi` produces a nearly identical install layout to what the previous method produces. When I applied this series on top of the 5.8 tag, here is the list of differences and explanations: * docs/*.html files are missing from the new installer because the Python build environment doesn't have docutils. * .pyd and .exe files differ, likely because I'm using a different Visual Studio toolchain on my local computer than the official build environment. * Various .dist-info/ directories have different names. This is because older versions of PyOxidizer had buggy behavior and weren't properly normalizing package names in .dist-info/ directories. e.g. we went from `cached-property-1.5.2.dist-info` to `cached_property-1.5.2.dist-info`. * Translations (.mo files) may be missing if gettext isn't in %Path%. This is because the packaging.py code installs gettext and ensures it can be found. * Some *.dist-info/RECORD files vary due to SHA-256 content digest divergence due to build environment differences. (This should be harmless.) * The new install layout ships a python3.dll because newer versions of PyOxidizer ship this file. * The new install layout has a different vcruntime140.dll and also a vcruntime140_1.dll because newer versions of PyOxidizer ship a newer version of the Visual C++ Redistributable Runtime. The new PyOxidizer functionality is not yet integrated with packaging.py. This will come in a subsequent commit. So for now, the new functionality introduced here is unused. Differential Revision: https://phab.mercurial-scm.org/D10683

pyoxidizer: use allocator_backend instead of raw_allocator The name of this attribute changed in PyOxidizer 0.11. Differential Revision: https://phab.mercurial-scm.org/D10682

pyoxidizer: simplify targets The split targets existed to enable the use of a non-default distribution flavor on Windows. Modern versions of PyOxidizer use the "standalone_dynamic" distribution flavor by default. So our split brain workaround is no longer needed. Here, we unify the targets. We also remove an unreferenced target function to create a resources file. Differential Revision: https://phab.mercurial-scm.org/D10681

contrib: install PyOxidizer 0.16.0 PyOxidizer now provides MSI installers and pre-built Linux binaries. So we install that way. This significantly reduces the time to bootstrap a new machine in automation, as building PyOxidizer from source on a low core count machine takes several minutes. This change temporarily breaks the ability of the automated environment to use the in-repo pyoxidizer.bzl configuration file, as there are backwards-incompatible changes with the upgrade that need to be reflected. We'll handle those in subsequent commits. Differential Revision: https://phab.mercurial-scm.org/D10680

pyoxidizer: pin Python to 3.8 Newer versions of PyOxidizer use version 3.9 by default. As part of upgrading PyOxidizer, we want to pin the version at 3.8 so we can compare differences more easily. Differential Revision: https://phab.mercurial-scm.org/D10679

pyoxidizer: remove some boilerplate in file This isn't used for anything and can safely be removed. Differential Revision: https://phab.mercurial-scm.org/D10678

contrib: upgrade Rust in Windows automation I'm not sure why we don't install the minimum required Rust version here like we do for Linux. Whatever: that's unrelated to wanting to stay modern. Differential Revision: https://phab.mercurial-scm.org/D10677

contrib: update Python versions in Windows automation Let's keep our Python versions modern. Differential Revision: https://phab.mercurial-scm.org/D10676

automation: update rust in Linux environment Our minimum supported Rust is 1.41.1 per rust/README.rst. We also bump the modern Rust version to latest stable to stay current. Differential Revision: https://phab.mercurial-scm.org/D10675

automation: install latest Python versions in Linux environment Let's keep our Linux environment up to date. Differential Revision: https://phab.mercurial-scm.org/D10674

automation: create Python 3.5 variant of requirements.txt The automation environment is refusing to build with the previous file because some dependencies won't install on Python 3.5. I couldn't find an easy way to salvage the situation with a single requirements.txt file. So, I decided to introduce a variant for Python 3.5. As part of this, we update packages to latest versions. (I do question why we are still supporting Python 3.5...) Differential Revision: https://phab.mercurial-scm.org/D10690

hgweb: Fix deprecation warning in Python 3.10 (issue6520) Differential Revision: https://phab.mercurial-scm.org/D10711

dirstate-tree: Remove newly-empty nodes after removing a `DirstateEntry` This is actually necessary to make `DirstateMap::has_dir` correct, since it assumes that a node without a `DirstateEntry` has at least one descedant node with a `DirstateEntry`. This bug would become apparent when a later changeset persists tree nodes on disk in the "dirstate-v2" format. Differential Revision: https://phab.mercurial-scm.org/D10706

dirstate-tree: Refactor DirstateMap::drop_file to be recursive It should behave the same as before. This will enable the next changeset to run code on the way "down" (in order to removing newly-empty nodes). Differential Revision: https://phab.mercurial-scm.org/D10705

templates: fix `revset('parents()') % ...` in amend message template I don't understand why, but putting `revset('parents()') % {desc}` in the commit message template for amend resulted in a crash because `memctx.hex()` did `hex(self.node())` and its node was None. This patch fixes that. Martin von Zweigbergk <martinvonz@google.com> Differential Revision: https://phab.mercurial-scm.org/D10707

path: teach the `hg path` command to display boolean sub-option The next changeset introduce one. Differential Revision: https://phab.mercurial-scm.org/D10450

parselist: move the function from config to stringutil We move the function in a lower level module to avoid cycle. It moves next to `parsebool` who had to migrate for the same reasons. Differential Revision: https://phab.mercurial-scm.org/D10449

bookmark: use `list_paths` to access path definition The content of the `[paths]` config section is receiving transformation that make it hard to recognise whats the actual name, and the next changeset will make it worse. So we use the official API for this instead. Differential Revision: https://phab.mercurial-scm.org/D10448

urlutil: make `paths` class old list of `path` We move from a `{name → path}` mapping to a `{name → [path]}` mapping. And update all user code accordingly. For now, all the list contains exactly one element, but we are now in a good place to make the config understand a list of url. Differential Revision: https://phab.mercurial-scm.org/D10447

urlutil: extract `chain_path` in a function This will no longer modify `path` inplace so it does not make much sense as a method. Differential Revision: https://phab.mercurial-scm.org/D10446

urlutil: add a `copy` method to `path This will be useful when inheriting from multiple path at the same time. Differential Revision: https://phab.mercurial-scm.org/D10445

template: add a `paths` field to all entry in peerurls This make it possible to display multiple path per name in the near future. Differential Revision: https://phab.mercurial-scm.org/D10444

cache: avoid warming the fnodetags cache after clone That cache can quite expensive to compute on large repository as not that `hg clone` is warming all cache, this can introduces a significant slowdown for clone time[1]. As a stop gap measure introduce a quick fix for that on stable, skipping the fnodetags cache post-clone. [1] https://www.mercurial-scm.org/pipermail/mercurial/2021-April/052679.html Differential Revision: https://phab.mercurial-scm.org/D10695

run-tests: fix whitelist/blacklist with directories on Windows The file name is resolved with `os.path.relpath()` in the `Test` constructor, which yields `\` on Windows. That doesn't match the `/` separator when using MSYS tools to build the list, and it isn't obvious that this is the problem because directory separators can mostly be used interchangeably. The `--test-list` argument already seems to be properly handled.

run-tests: ignore PermissionError when checking available ports I'm not sure what this is, but I'm getting it occasionally when running in WSL. When it was raised, none of the tests could run.

status: Add some more tests Trying to improve coverage for various non-ovbious scenarios Differential Revision: https://phab.mercurial-scm.org/D10704

run-tests: use the same python version for shebang lines on Windows The latest py3 is used if the minor number isn't specified. After running the script to install all of the build dependencies, that moved the default from 3.8 to 3.9 on the CI system. That in turn caused a bunch of tests to be skipped that were running prior, even when the test runner was invoked with `py -3.8`. While we should almost always use the latest version, we really shouldn't make it hard to test different versions or allow things to randomly break in subtle ways like that. Differential Revision: https://phab.mercurial-scm.org/D10702

vfs: Fix deprecation warning in Python 3.10 (issue6520) Differential Revision: https://phab.mercurial-scm.org/D10710

contrib: fix typo I forgot to fix it in flight, this commit will do fine.

revlog: rename `indexdata` to entry_data Same reasoning as the previous changeset, we might not be looking at index data here. Differential Revision: https://phab.mercurial-scm.org/D10602

revlog: use "entry_point" phrasing for loading the revlog The main entry for the revlog will not necessary be the index, but a small "docket". So we change the variable names and we move the initialisation of the index_file and data_file after that first entry point have been read. Differential Revision: https://phab.mercurial-scm.org/D10601

revlog: directly use self._format_flags when loading index The shorthand variable does not bring much, so we drop it to simplify the code. Differential Revision: https://phab.mercurial-scm.org/D10600

revlog: directly use self._format_version when loading index The shorthand variable does not bring much, so we drop it to simplify the code. Differential Revision: https://phab.mercurial-scm.org/D10599

revlog: use `_format_flags` to access flags instead of `header` It seems better to reuse the variable we carefully extracted This also open the way to more flexible way to retrieve these flags. Differential Revision: https://phab.mercurial-scm.org/D10598

revlog: rename `newversionflags` to `new_header` This make it consistent with the previous changeset. Differential Revision: https://phab.mercurial-scm.org/D10597

revlog: rename `versionflags` to header The variable is named as such because it contains "version" information and "flags" information. However you mostly needs to know the code to understand the name which is not great. The fact that this is the very first four bytes in all revlog seems more relevant, so we rename the variable "header". Differential Revision: https://phab.mercurial-scm.org/D10596

revlog: move index reading logic in a dedicated method They are multiple motivation to do it: * The logic is complicated enough to deserver its own method. * We will need to reuse this once we put a docket in use. * This split the actual reading from the processing of the read data better. Differential Revision: https://phab.mercurial-scm.org/D10595

revlog: define the actual index and datafile at loading time This is just code movement, to make the code closer to where we actually use it and where it will be defined in the future. Differential Revision: https://phab.mercurial-scm.org/D10594

revlog: simplify a conditionnal in _enforceinlinesize This is a gratuitous change to make things a bit easier to read. Differential Revision: https://phab.mercurial-scm.org/D10593

revlog: drop `flush` parameter from `_peek_iscensored` This is not used anywhere. Differential Revision: https://phab.mercurial-scm.org/D10592

revlog: fix error message when data are missing The error message the message was not using the requested offset, but the adjusted offset to that read more data for improved caching. This resulted in confusing error message. Differential Revision: https://phab.mercurial-scm.org/D10591

revlog: rename `nodemap_file` to `_nodemap_file` Same reasoning as for `indexfile and datafile`, lets hide these implementation details. Differential Revision: https://phab.mercurial-scm.org/D10590

revlog: use revlog.display_id in error related to bad revisions Differential Revision: https://phab.mercurial-scm.org/D10589

revlog: use revlog.display_id in "revision too big" errors Differential Revision: https://phab.mercurial-scm.org/D10588

revlog: use revlog.display_id in censor related errors Differential Revision: https://phab.mercurial-scm.org/D10587

revlog: use revlog.display_id in integrity error Differential Revision: https://phab.mercurial-scm.org/D10586

revlog: use revlog.display_id in ambiguity errors Differential Revision: https://phab.mercurial-scm.org/D10585

revlog: use revlog.display_id for corruption error Differential Revision: https://phab.mercurial-scm.org/D10584

revlog: use revlog.display_id in format related errors Differential Revision: https://phab.mercurial-scm.org/D10583

revlog: use revlog.display_id in narrow error message Differential Revision: https://phab.mercurial-scm.org/D10582