match: convert O(n) to O(log n) in exactmatcher.visitchildrenset When using narrow, during rebase this is called (at least) once per directory in the set of files in the commit being rebased. Every time it's called, we did the set arithmetic (now extracted and cached), which was probably pretty cheap but not necessary to repeat each time, looped over every item in the matcher and kept things that started with the directory we were querying. With very large narrowspecs, and a commit that touched a file in a large number of directories, this was slow. In a pathological repo, the rebase of a single commit (that touched over 17k files, I believe in approximately as many directories) with a narrowspec that had >32k entries took 8,246s of profiled time, with 5,007s of that spent in visitchildrenset (transitively). With this change, the time spent in visitchildrenset is less than 34s (which is where my profile cut off). Most of the remaining time was network access due to our custom remotefilelog-based setup not properly prefetching. Differential Revision: https://phab.mercurial-scm.org/D10294

exthelper: improve docs to indicate what module vars are needed I recently tried creating an extension "from scratch" using exthelper, and it wasn't obvious that you needed these. I believe that a careful reading of one of the comments would tell you that they were required, but it's easy to miss and having the examples be "complete" is helpful. Differential Revision: https://phab.mercurial-scm.org/D10295

tests: restore the ability to run `black` on Windows The hghave test for black silently stopped working with the change in 08fd76a553c9. This was the output of what it was hitting when run in the test environment: Traceback (most recent call last):\r (esc) File "c:\\users\\matt\\appdata\\local\\programs\\python\\python39\\lib\\runpy.py", line 197, in _run_module_as_main\r (esc) return _run_code(code, main_globals, None,\r (esc) File "c:\\users\\matt\\appdata\\local\\programs\\python\\python39\\lib\\runpy.py", line 87, in _run_code\r (esc) exec(code, run_globals)\r (esc) File "c:\\Users\\Matt\\AppData\\Roaming\\Python\\Python39\\Scripts\\black.exe\\__main__.py", line 4, in <module>\r (esc) File "C:\\Users\\Matt\\AppData\\Roaming\\Python\\Python39\\site-packages\\black\\__init__.py", line 70, in <module>\r (esc) CACHE_DIR = Path(user_cache_dir("black", version=__version__))\r (esc) File "C:\\Users\\Matt\\AppData\\Roaming\\Python\\Python39\\site-packages\\appdirs.py", line 293, in user_cache_dir\r (esc) path = os.path.normpath(_get_win_folder("CSIDL_LOCAL_APPDATA"))\r (esc) File "C:\\Users\\Matt\\AppData\\Roaming\\Python\\Python39\\site-packages\\appdirs.py", line 481, in _get_win_folder_with_pywin32\r (esc) dir = shell.SHGetFolderPath(0, getattr(shellcon, csidl_name), 0, 0)\r (esc) pywintypes.com_error: (-2147024893, '$ENOTDIR$.', None, None)\r (esc) [1] Differential Revision: https://phab.mercurial-scm.org/D10310

setup: copy python3.dll next to hg.exe when building on Windows for hgext.git I thought I took care of this already, but it must have been that I just manually copied the file over locally when debugging why the pygit2 library wasn't loading. The problem with that is what was copied over was from py38, and then running a py39 build hard crashed when the extension was loaded. Differential Revision: https://phab.mercurial-scm.org/D10301

tests: stablize test-hook.t on Windows Apparently, hooks can't run `echo` directly, even from MSYS. Differential Revision: https://phab.mercurial-scm.org/D10300

tests: handle Windows file separator differences in test-config.t Differential Revision: https://phab.mercurial-scm.org/D10299

tests: update the detailed exit codes for icasefs gated tests The fact that there's already a detailed exit code after the last change here in test-casecollision-merge.t gives me some pause, but maybe it was found and changed manually? Differential Revision: https://phab.mercurial-scm.org/D10298

tests: update the detailed exit codes in test-phabricator.t I'm guessing this has slipped through because most people don't have the necessary `pytest-vcr` package installed. Differential Revision: https://phab.mercurial-scm.org/D10297

tests: avoid using rebaseskipobsolete=0 I'm about to delete the `rebaseskipobsolete` config. This patch updates a test to get hidden commits into the rebase state by using `hg debugobsolete` instead of setting `rebaseskipobsolete=0`. Differential Revision: https://phab.mercurial-scm.org/D10270

rebase: when using --keep, don't care about pruned commits or divergence `hg rebase --keep` creates duplicate commits (not successors), so I was surprised that it still skips pruned commits and errors out if it "would cause divergence" (it wouldn't). I guess this was just an oversight. We didn't have any tests for it, so I also included that. Differential Revision: https://phab.mercurial-scm.org/D10269

rebase: remove duplicate initialization of a field `obsolete_with_successor_in_destination` is already initialized in the constructor. Differential Revision: https://phab.mercurial-scm.org/D10268

rust: bump rust-cpython version to 0.5.2 we need a newer version to define "property" on Rust defined object. Differential Revision: https://phab.mercurial-scm.org/D10308

revlog: directly use the Struct object for related operation The Struct object has all the piece we needs, so no need to duplicate information on the revlog itself. Differential Revision: https://phab.mercurial-scm.org/D10307

revlog: move the "index header" struct inside revlog.utils.constants The struct was previous called "version", but this is actually "version" + "flags". So header seems like a better name. The move to the `constants` module has the same motivation as the INDEX_ENTRY_V# ones. Differential Revision: https://phab.mercurial-scm.org/D10306

revlog: move the details of revlog "v2" index inside revlog.utils.constants the revlog module is quite large and this kind of format information would handy for other module. So let us start to gather this information about the format in a more appropriate place. We update various reference to this information to use the new "source of truth" in the process. Differential Revision: https://phab.mercurial-scm.org/D10305

revlog: move the details of revlog "v1" index inside revlog.utils.constants The revlog module is quite large and this kind of format information would handy for other module. So let us start to gather this information about the format in a more appropriate place. We update various reference to this information to use the new "source of truth" in the process. Differential Revision: https://phab.mercurial-scm.org/D10304

revlog: move the details of revlog "v0" index inside revlog.utils.constants the revlog module is quite large and this kind of format information would handy for other module. So let us start to gather this information about the format in a more appropriate place. Differential Revision: https://phab.mercurial-scm.org/D10303

revlog: add some comment in the header sections We are about to add more content so let us organise the existing content first. Differential Revision: https://phab.mercurial-scm.org/D10302

store: drop the `filefilter` argument to `_walk` No code use it anywhere. Dropping it will help replacing the function with something with a more precise semantic. Differential Revision: https://phab.mercurial-scm.org/D10314

store: document the `walk` method Differential Revision: https://phab.mercurial-scm.org/D10313

revlog: fix error about unknown compression format in py3 In py2, the error is something like: abort: unknown compression type 'x'! In py3, we get the following unhelpful message: abort: unknown compression type <memory at 0x7f4650b5cdc8>! Switch to something like: abort: unknown compression type 78! Differential Revision: https://phab.mercurial-scm.org/D10318

revlog-compression: fix computation of engine availability We don't just need the engine to be define, we need it to be available and able to do be used for revlog compression. Without this change, `zstd` could be selected as a viable option for repository creation on platform where it is not available. Differential Revision: https://phab.mercurial-scm.org/D10325

test: explicitly use zlib compression in tests/test-share-safe.t We need the implicit value to be explicit until we can change the default in some case. Differential Revision: https://phab.mercurial-scm.org/D10323

test: explicitly use zlib compression in tests/test-upgrade-repo.t We need the implicit value to be explicit until we can change the default in some case. Differential Revision: https://phab.mercurial-scm.org/D10322

test: explicitly use zlib compression in tests/test-repo-compengines.t We need the implicit value to be explicit until we can change the default in some case. Differential Revision: https://phab.mercurial-scm.org/D10321

remotefilelog: include file contents in bundles produced during strip `hg strip` and other things that use repair.strip (such as the narrow extension's `hg tracked --removeinclude`) will "save" some commits that have a higher revision number than the oldest commit we're stripping, but aren't actually descended from any of the commits that we're stripping. It saves them in a bundle, and then reapplies them to the repo. Remotefilelog doesn't generally participate in strip, it doesn't contribute files to either the backup bundle or the "saved" bundle, and doesn't adjust linknodes when commits are stripped. This can break things like push, which rely on the linknodes. This change makes it so that remotefilelog includes files in these bundles during strip operations. During reapplication, the files are reapplied from the bundle, and the linknode is properly updated. Differential Revision: https://phab.mercurial-scm.org/D10320

tests: add test-remotefilelog-strip.t to demonstrate an issue with linknodes ### Background Every time a commit is modified, remotefilelog updates the metadata for the file object to point to the new commit (I believe that this is different from non-remotefilelog hg, which leaves the linkrevs pointing to the obsolete commits; doing otherwise would involve changing data in the middle of revlogs). With `hg strip` (or other things that use repair.strip()), when you strip a commit that's not the tip of the revlog, there may be commits after it in revnum order that aren't descended from it and don't need to be (and shouldn't be) stripped. These are "saved" by strip in a bundle, and that bundle is reapplied after truncating the relevant revlogs. ### The problem Remotefilelog generally avoids being involved at all in strip. Currently, that includes even providing file contents to this backup bundle. This can cause the linknode to point to a changeset that is no longer in the repository. Example: ``` @ 3 df91f74b871e | | x 2 70494d7ec5ef |/ | x 1 1e423846dde0 |/ o 0 b292c1e3311f ``` Commits 1, 2, and 3 are related via obsolescence, and are description-only changes. The linknode for the file in these commits changed each time we updated the description, so it's currently df91f7. If I strip commits 1 and 3, however, the linknode *remains* df91f7, which no longer exists in the repository. Commit 70494d was "saved", stripped, and then reapplied, so it is in the repository (as revision 1 instead of 2 now), and was unobsoleted since the obsmarker was stripped as well. The linknode for the file should point to 70494d, the most recent commit that is in the repository that modified the file. Remotefilelog has some logic to handle broken linknodes, but it can be slow. We have actually disabled it internally because it's too slow for our purposes. Differential Revision: https://phab.mercurial-scm.org/D10319

mergestate: remove unused import Differential Revision: https://phab.mercurial-scm.org/D10291

deb: avoid use of [[ in 'rules' file It's not supported by posix shell, and apparently my build system uses that. Differential Revision: https://phab.mercurial-scm.org/D10292

refactor: prefer checks against nullrev over nullid A common pattern is using a changeset context and obtaining the node to compare against nullid. Change this to obtain the nullrev instead. In the future, the nullid becomes a property of the repository and is no longer a global constant, so using nullrev is much easier to reason about. Python function call overhead makes the difference moot, but future changes will result in more dictionary lookups otherwise, so prefer the simpler pattern. Differential Revision: https://phab.mercurial-scm.org/D10290

refactor: prefer lookup by revision, even for null While the nullid lookup is a special case, it is still more complicated. The common pattern is to lookup via nullrev so be consistent here. Differential Revision: https://phab.mercurial-scm.org/D10280

setdiscovery: simplify by using tiprev directly tip() uses tiprev() and reads the node from it, so drop a layer of indirection. Differential Revision: https://phab.mercurial-scm.org/D10289