sharesafe: introduce functionality to automatically upgrade shares In past few months, we have developed a `share-safe` mode for sharing repository in which share source requirements and config values are shared with the shares. To get it rolling, an important task is to get these shares automatically upgraded. We are focusing on an installation where shares are created by scripts and test jobs. It will be difficult to manually upgrade these and we need some functionality to do so automatically. This patch introduces a config option to deal with it. If all of the following conditions are met, we upgrade the share repository automatically: * If the config option is enabled * Share source repository is share-safe enabled * Share is not share-safe enabled * Any command is run in the share Upgrading the share is pretty easy as it involves only editing the requirements file. Differential Revision: https://phab.mercurial-scm.org/D9679

localrepo: move storevfs calculation out of if statement In next patch, we will need this variable in else statement too. So, let's take it out. Differential Revision: https://phab.mercurial-scm.org/D9681

setup: don't import distutils prior to checking FORCE_SETUPTOOLS I've seen warnings about importing distutils before setuptools, although I can't reproduce them at the moment. Differential Revision: https://phab.mercurial-scm.org/D9722

largefiles: remove unused imports This fixes test-check-pyflakes.t Differential Revision: https://phab.mercurial-scm.org/D9747

upgrade: don't perform anything if nothing to do Before this patch, upgrade will process everything, re-clone all revlogs, write requirements file again even there is no change to be made. This patch makes it exit earlier. Differential Revision: https://phab.mercurial-scm.org/D9695

downgrade: if a compression is removed, consider that too For compression format variant, the result of `fromrepo()` and `fromconfig()` is not a boolean and we have to actually equate those to find change. Differential Revision: https://phab.mercurial-scm.org/D9693

engine: prevent a function call for each store file Python function calls are not cheap. Instead of calling the function once for each file in store, we use a dedicated function which filters and yields the required files. Differential Revision: https://phab.mercurial-scm.org/D9673

rhg: use a release-mode executable in tests This allows the rhg build for test-rhg.t to share compiled dependencies such as hg-core with the hg-cpython build for other tests. For context, my wrapper script for the typical edit-compile-test cycle now looks like this: (cd rust && cargo +nightly-2020-10-04 fmt) (cd rust && cargo build --release -p rhg) make --silent local PURE=--rust python test/run-tests.py --local "$@" Differential Revision: https://phab.mercurial-scm.org/D9728

rust: fix file folding map The file folding map, frequently used on macOS, had two issues: * the means for converting it to Python didn't work * a minor typo when copying the python code, where `!=` became `==` With this, the rust code passes all tests on macOS. Test Plan: I'm currently doing a full test run on a case-insensitive file system. If it passes, perhaps we can change the platform check from an error to a warning? Differential Revision: https://phab.mercurial-scm.org/D9671

test: replace a many occurence of `python` with `$PYTHON` Otherwise this can use the wrong python version, or worse, not find any python at all. Differential Revision: https://phab.mercurial-scm.org/D9730

tests: make test-subrepo-git.t compatible with git's master->main rename Git is about to change the default branch from "master" to "main". Recent versions has started warning about that, which makes tests fail. I assume the tests would fail in a different way once the default has changed. To make us compatible with that name change, let's set configure the default to be "master". That value makes the tests still work on older Git version (those where the default branch name is not configurable). Differential Revision: https://phab.mercurial-scm.org/D9746

revlog: use size_t for nodetree capacity This allows handling revlog containing more than 33554432 (INT_MAX / sizeof(nodetreenode)) revisions on x64 platforms. Differential Revision: https://phab.mercurial-scm.org/D9745

upgrade: demonstrate that a no-op upgrade still performs everything An upgrade operation which is not adding or removing anything should ideally error out. However, it does the whole cloning and everything. Next patch will fix it. Differential Revision: https://phab.mercurial-scm.org/D9694

upgrade: migrated -> upgraded in ui messages Differential Revision: https://phab.mercurial-scm.org/D9676

upgrade: remove unnecessary `is None` check `upgrade_engine.upgrade()` always return the `backuppath` value and there are not early returns. Hence I don't see how `backuppath` can be None. Adding extra unncessary safe checks hides unknown bugs. Hence removing it. Differential Revision: https://phab.mercurial-scm.org/D9675

engine: refactor code to replace stores in separate function In not all upgrades, we need to change the whole store. For example, when upgrading repository to share-safe mode, we don't touch revlogs at all hence store cloning and copying is not required. The store replacing code needs to be made aware about what all has changed and hence only copy/rename those things. To kickstart that, this patch moves existing logic into a separate function. Differential Revision: https://phab.mercurial-scm.org/D9674

engine: make hook point for extension a public function Well there are no private public functions in Python, but we generally treat functions and variables starting with `_` as private ones. A function which needs to be overrided in extension should be a public one. Differential Revision: https://phab.mercurial-scm.org/D9672

engine: prevent multiple checking of re-delta-multibase The _perform_clone function is called for each revlog cloned, hence we should prevent this function call overhead. Differential Revision: https://phab.mercurial-scm.org/D9669

engine: pass upgrade operation inside `_perform_clone()` Same as previous patch. Differential Revision: https://phab.mercurial-scm.org/D9668

engine: pass upgrade operation inside _clonerevlogs() Better to pass the operation instead of passing three of it's members (one of the them is a function call) separately. This will also be useful in future when we will like to control which things are upgraded. Differential Revision: https://phab.mercurial-scm.org/D9667

actions: store deltareuse mode of whole operation in UpgradeOperation UpgradeOperation should provide easy access to all the things related to the current operation. Clients should not need to compute them. Differential Revision: https://phab.mercurial-scm.org/D9666

engine: refactor how total dstsize is calculated Instead of increasing it with each revlog, we just get the sum of total destination changelog, manifest and filelogs sizes. Differential Revision: https://phab.mercurial-scm.org/D9665

upgrade: introduce post upgrade and downgrade message for improvements For certain imporvements, we will like to show a message after the operation completed. This patch introduces that functionality. Right now it's only used by share-safe feature. Differential Revision: https://phab.mercurial-scm.org/D9619

actions: introduce function to calculate downgrades An upgrade operation can also downgrade/remove some format variants. Before this patch there was no clean way to find out all such variants which will be removed. This patch adds a function for that. It will be used in next patch. Differential Revision: https://phab.mercurial-scm.org/D9618

debugupgraderepo: minor documentation fix When we specify `--no-changelog --no-manifest --no-filelog` we skip all revlog optimization instead of all filelog optimization. Also while I was here, for consistency, I did `optimisation` -> `optimization` to make it consistent with rest of occurrences. Note: I am not native speaker and I only changed it because of consistency. I don't know which one is correct. Differential Revision: https://phab.mercurial-scm.org/D9617

upgrade: rename actions to upgrade_actions The `actions` were a list of optimizations and format upgrades which will be upgraded. This does not include things which will be downgraded. Let's rename the variable for clarity. It now makes obvious that we don't have any concrete information on what things are downgraded. Differential Revision: https://phab.mercurial-scm.org/D9616

upgrade: move optimization addition to determineactions() The documentation of `determineactions()` mention that it is given a list returned from `findoptimizations()` however it was not true before this patch. The code extending actions with optimizations also mentioned about it that this should be in determineactions. So let's do what comments at couple of places say. Differential Revision: https://phab.mercurial-scm.org/D9615

upgrade: drop support for old style optimization names Old style optimization names like `redeltaparent` were converted into `re-delta-parent` more than two years ago. The old names were kept around for sometime because of BC reasons. Refer 5608b5a6c3231c4ec771171cc3079142c7672be6. The commit states the map is there for a while and we can drop them as the underlying command is a debug command. Differential Revision: https://phab.mercurial-scm.org/D9614

upgrade: add a missing space in status message Found while reading the code to refactor. Differential Revision: https://phab.mercurial-scm.org/D9583

actions: rename DEFICIENCY constant to FORMAT_VARIANT It was not obvious what does deficieny means and every format change can't be a deficiency. There are some format changes like compression levels, share-safe which can't be understood at deficiencies. This patch renames the constant to make things clearer. Differential Revision: https://phab.mercurial-scm.org/D9664