test: enable sparse-revlog for test-revlog-v2.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5333

test: enable sparse-revlog for test-repo-compengines.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5332

test: enable sparse-revlog for test-remotefilelog-log.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5331

test: enable sparse-revlog for test-remotefilelog-clone-tree.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5330

test: enable sparse-revlog for test-remotefilelog-clone.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5329

test: enable sparse-revlog for test-phases.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5328

test: enable sparse-revlog for test-narrow-clone.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5327

test: enable sparse-revlog for test-narrow-clone-stream.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5326

test: enable sparse-revlog for test-narrow-clone-no-ellipsis.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5325

test: enable sparse-revlog for test-lfs-serve.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5324

test: enable sparse-revlog for test-lfs-largefiles.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5323

test: enable sparse-revlog for test-lfconvert.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5322

test: enable sparse-revlog for test-init.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5321

test: enable sparse-revlog for test-http-protocol.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5320

test: enable sparse-revlog for test-hgweb-commands.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5319

test: enable sparse-revlog for test-debugcommands.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5318

test: enable sparse-revlog for test-clone-uncompressed.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5317

test: enable sparse-revlog for test-clonebundles.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5316

test: enable sparse-revlog for test-bundle.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5315

test: make sure sparse-revlog does not interfer with test-parseindex.t That test carefully craft revlog to reproduce issues. We do not want new revlog behavior to interfere with that. Differential Revision: https://phab.mercurial-scm.org/D5314

test: explicitly disable sparse-revlog in test-http-bad-server.t The `test-http-bad-server.t` test is very fragile with hard-coded number all around. Adding any new capability have massive consequences on it that are long and tiresome to handle. Since the test check more generic protocol level behavior, it does not need to follow all new capabilities. Disabling the capabilities is a much simpler way to avoid the issue. We make sure `sparse-revlog` will not impact this test once turned on by default. Differential Revision: https://phab.mercurial-scm.org/D5313

test: explicitly disable sparse-revlog in test-generaldelta.t That test is about testing generaldelta without further improvement. We ensure it will stay that way even if sparse-revlog is turned on by default. The sparse-revlog format is tested on his own in a different test file. Differential Revision: https://phab.mercurial-scm.org/D5312

sparse-revlog: disable sparse-revlog if config disable general-delta Sparse-revlog requires general-delta in order to work. If general-delta is explicitly disabled through configuration, we should honor that and also disabled sparse-revlog. This change will be more relevant when sparse-revlog is enabled by default. Differential Revision: https://phab.mercurial-scm.org/D5311

test: fix config typo in test-upgrade-repo.t Differential Revision: https://phab.mercurial-scm.org/D5310

tests: use $RUNTESTDIR instead of $TESTDIR in narrow-library.sh This is done so that third party extensions can use narrow-library.sh in tests. Differential Revision: https://phab.mercurial-scm.org/D5400

tests: update network related errors for Debian 9 We have a CI job that runs the Mercurial tests in parallel. Some of the network related failures seems to be different on the environment. Oddly, those failures happens only when running the tests in parallel, not when running the test file only. I have no idea how to get the windows formatted message for the error, if someone could give me an hand, I will update this changeset with the value. Differential Revision: https://phab.mercurial-scm.org/D5401

util: implement pop() on lrucachedict This moves __delitem__() to pop() as the requirement is pretty much the same, and reimplement __delitem__() by using pop().

util: add method to peek item in lrucachedict I want a function that doesn't unnecessarily update the internal state of the cache dict after fork().

commandserver: loop over selector events An IPC socket will be waited by the same selector.

commandserver: remove redundant "if True" block

commandserver: extract handler of new socket connection This prepares for adding an IPC socket.

commandserver: pass around option to hook repo instance creation This is necessary to wrap a repo instance so the master process will be notified on repo.close().

py3: stop subscripting socket.error In 3.3 and later, this is now an alias for OSError. I hacked up the server code enough that I was able to trigger the exception handler in server.py from test-http-bundle1.t. Other instances of this either subscript through the `args` member, or reference the errno or strerror attributes. Note that on Windows, the errno value seems to reflect the Winsock error, so the various tests for EPIPE seem like they would always fail. But that seems to be the case in py2 as well.

test: update test-logtoprocess.t to not requires less The test previously assumed that a working pager was present in the test environment. Update it to use the fakepager instead. Differential Revision: https://phab.mercurial-scm.org/D5399

py3: quote several instances of $PYTHON for Windows Python3 lives in Program Files by default. The last remaining unquoted instance is in test-hghave.t. I can't figure out how to quote that, as it either complains: 'c:\\Program' is not recognized as an internal or external command, operable program or batch file. or $TESTTMP.sh: line 22: "C:/Program Files/Python37/python.exe": $ENOENT$

py3: adapt test-largefiles-wireproto.t to python3

py3: byteify test-push-race.t

py3: convert filename to bytes in test-hardlinks.t Windows was complaining about passing str to CreateFileA() in win32.py.

narrow: keep bookmarks temporarily stripped for as long as commits are The narrow extension also has support for shallowness and for inserting older commits on pull. It works by temporarily stripping newer commits, adding the older commits, then re-applying the stripped bundle. The regular Mercurial server uses that when you widen, although it shouldn't be necessary there. Our Google-internal server does it when the user requests an older commit. Our Google-internal tests fail since 7caf632e30c3 (filecache: unimplement __set__() and __delete__() (API), 2018-10-20). I haven't quite understood the problem, but it's related to the way we temporarily hide bookmarks while the commits they point to are stripped. When a transaction is started, Mercurial tries to read various things from the repo for the transaction summary. That leads to computation of hidden commits, which leads to an attempt to find commits pinned by bookmarks. This is the reason we temporarily hide the bookmarks. I think the aforementioned commit makes the restored bookmarks visible earlier than before (which seems like an improvement), so we can no longer incorrectly rely on the repo._bookmarks field being cached too long (IIUC). This patch makes it so we restore the temporarily hidden bookmarks only after the temporary bundle has been re-applied. It also adapts the code to update the repo.__bookmarks field using the pattern described in the aforementioned commit instead of writing directly to the fiels. I have spent many hours trying to understand what was going on here, but I still don't know if this can also happen without our custom server. So this patch unfortunately does not add any tests; I have only been able to test the fix using our Google-internal tests. Differential Revision: https://phab.mercurial-scm.org/D5398

narrow: drop obsolete support for old Mercurial Not needed since narrow is shipped with core. Differential Revision: https://phab.mercurial-scm.org/D5397

tests: test that narrow preserves bookmarks on widen When widening inserts older commits in the changelog, we have to preserve bookmarks so they are not removed by the call to repair.strip() we temporarily do. We didn't have any tests for that before. Differential Revision: https://phab.mercurial-scm.org/D5396

rebase: clarify that commits that become empty are skipped Our message just said that it created no changes, but we didn't explicitly say that that meant that the the commit was not rebased. It also wasn't clear why it created no changes, so this patch also clarifies that that was because the changes were already in the destination. Differential Revision: https://phab.mercurial-scm.org/D5395

rebase: format commit in usual way when explaining that it became empty We use the _ctxdesc() for other similar messages, so it seems we should use it here too. Differential Revision: https://phab.mercurial-scm.org/D5394

rust: propagate error of index_get_parents() properly Before, rustla_contains() would return 0 on error, and the exception would be cleared or noticed somewhere else. We need to propagate the error from AncestorsIterator up to the FFI surface.

rust: look up HgRevlogIndex_GetParents() from symbol table And removes the unused index_get_parents_checked() function. I expect the Index struct will be turned into a pyobject type, though I haven't written any PoC-level patches yet.

revlog: add public CPython function to get parent revisions Since this is a public function, it validates the input revision, and supports nullrev. index_get_parents_checked() will be replaced by this function.

revlog: rename indexType to HgRevlogIndex_Type as it's a global symbol It follows the CPython naming convention, but uses Hg instead of Py.

revlog: export symbol of indexType The idea is to wrap the index object with rust-cpython. I haven't tried it, but it should be doable. We'll probably need a better interface than raw function pointers to do more in Rust.

setup: fix path of rust source dependencies Appears that they were screwed up while moving the source files around. Since rust-direct-ffi sources are automatically added to the depends list, we only need to list hg-core sources.

rebase: remove now-unnecessary arguments to _abort() Since _abort() is now an instance method, we don't need to pass state that's available on "self". Differential Revision: https://phab.mercurial-scm.org/D5393

rebase: move abort() onto rebaseruntime The function depends on a lot of the state of rebaseruntime, so it makes sense for it to be an instance method. This will let us remove many of the arguments to the method. That will be done in a later patch. Differential Revision: https://phab.mercurial-scm.org/D5392

cleanupnodes: trust caller when "moves" is not None If "moves" (indicating how to move bookmarks) is None, we fill it out based on "replacements" (indicating which obsmarkers to add). If "moves" is not None, we would still add items based on "replacements". This makes it impossible to pass "moves={}" and not move bookmarks, which surprised me. The only caller that currently passes a value for "moves" was the rebase extension and there we were already adding bookmark moves corresponding to obsmarker additions, so it should not be impacted. Differential Revision: https://phab.mercurial-scm.org/D5391

shelve: change transaction description from "commit" to "shelve" "commit" was probably a copy&paste mistake. Differential Revision: https://phab.mercurial-scm.org/D5390

shelve: drop unnecessary backup of dirstate for phase-based case Regular shelve has a hack using an uncommitted transaction that's then aborted at the end of the operation. It preserves the dirstate across the abort, however, by saving a backup copy of it. Phase-based shelve instead commits the transaction, so the hack shouldn't be necessary there. Differential Revision: https://phab.mercurial-scm.org/D5389

tests: split test-shelve.t in two test-shelve.t dominated run time for all shelve tests. Before: # Ran 9 tests, 1 skipped, 0 failed. real 0m43.568s user 2m15.822s sys 0m40.857s After: # Ran 11 tests, 1 skipped, 0 failed. real 0m24.574s user 2m21.354s sys 0m40.435s Differential Revision: https://phab.mercurial-scm.org/D5388

remotefilelog: rely on progress helper for keeping track of position The progress helper class keeps track of its current position, so we don't need a "count" variable for that. Differential Revision: https://phab.mercurial-scm.org/D5386

remotefilelog: reduce use of "count" container We had already given the initial value of "count[0]" a name, so just use that when applicable. Differential Revision: https://phab.mercurial-scm.org/D5385

remotefilelog: replace a "a=[expr]; b=a[0]" by "b=expr; a = [b]" Simpler is better. Differential Revision: https://phab.mercurial-scm.org/D5384

remotefilelog: remove an unnecessary update of "count" container The "count" container is never used after this point, so there is no need to update it. Differential Revision: https://phab.mercurial-scm.org/D5383

remotefilelog: avoid temporarily using "count" variable as synonym for "total" The "count" variable is generally used for updating progress, but early in fileserverclient.request(), its used to mean the total count. We already have another "total" variable for that, so it seems much clearer to use that. Differential Revision: https://phab.mercurial-scm.org/D5382

remotefilelog: use progress helper in fileserverclient Differential Revision: https://phab.mercurial-scm.org/D5381

remotefilelog: use progress helper in shallowrepo Differential Revision: https://phab.mercurial-scm.org/D5380

remotefilelog: use progress helper in shallowbundle Differential Revision: https://phab.mercurial-scm.org/D5379

remotefilelog: use progress helper in repack Differential Revision: https://phab.mercurial-scm.org/D5378

remotefilelog: use progress helper in remotefilelogserver Differential Revision: https://phab.mercurial-scm.org/D5377

remotefilelog: use progress helper in basestore Differential Revision: https://phab.mercurial-scm.org/D5376

remotefilelog: use progress helper when analyzing repos Differential Revision: https://phab.mercurial-scm.org/D5375

fastannotate: use progress helper Differential Revision: https://phab.mercurial-scm.org/D5374

debugbuildannotatecache: use progress helper Differential Revision: https://phab.mercurial-scm.org/D5373

upgrade: clarify "aggressivemergedelta" handling We rename "aggressivemergedelta" argument to "forceaggressivemergedelta". The previous argument naming could infer an absolute control on the behavior. However, the code show we respect the config option if set.

upgrade: test that fncache is valid after repository upgrade The behavior was correct. Now it is correct and tested.

tests: update `rm` invocation for new location of checklink This should fix the test on FreeBSD. Differential Revision: https://phab.mercurial-scm.org/D5387

bookflow: new extension for bookmark-based branching This extension should be helpful for feature branches - based workflows. At my company we first considered branches, but weren't sure about creating a lot of permanent objects. We tried bookmarks, but found some scenarios to be difficult to control. The main problem, was the active bookmark being moved on update. Disabling that made everything a lot more predictable. Bookmarks move on commit, and updating means switching between them. The extension also implements a few minor features to better guide the workflow: - hg bookmark NAME can be ambiguous (create or move), unlike hg branch. The extension requires -rev to move. - require an active bookmark on commit. - some bookmarks can be protected (like @), useful for teams, that require code reviews. - block creation of new branches. The initial implementation requires no changes in the core, but it does rely on some implementation details. I thought it may be useful to discuss the functionality first, and then focus making the code more robust. Differential Revision: https://phab.mercurial-scm.org/D4312

rust: make clean takes care of rust/target This isn't the prettiest way of doing it, but it doesn't require looking up cargo, or wondering whether that should be part of setup.py clean. Differential Revision: https://phab.mercurial-scm.org/D5369

rust: rename local variables in AncestorsIterator::next It was confusing to have p1 and parents.1 ; (p1, p2) is clearer. Differential Revision: https://phab.mercurial-scm.org/D5365

tests: stabilize test-inherit-mode.t on FreeBSD and macOS again This is the same fix as 90e26ef4cbb1, just repeated on the new file location. Differential Revision: https://phab.mercurial-scm.org/D5371

tests: drop redundant "#if execbit" from test-rebase-inmemory.t The whole file is already guarded by "#require execbit". Differential Revision: https://phab.mercurial-scm.org/D5366

narrow: don't resurrect old commits when narrowing (don't strip obsmarkers) If you have an old obsolescence-chain of commits that has been pruned and you narrow your repo so that some of those commits get stripped (because they affected the removed paths), then we would currently resurrect the commit that came before (along the obsmarker chain) the last stripped commit. That happens by the usual rules for obsmarker-stripping. However, it's quite surprising when it happens when you narrow your repo. This patch makes narrowing not strip obsmarkers. Differential Revision: https://phab.mercurial-scm.org/D5364

commandserver: turn server debug messages into logs They were ui.debug() just because commandserver.log() was noop if no client connected.

commandserver: add config knob for various logging options The default rotating options are copied from the blackbox extension.

commandserver: expand log path for convenience This allows us to set the log path relative to $XDG_RUNTIME_DIR, for instance. [cmdserver] log = $XDG_RUNTIME_DIR/chg/server.log

commandserver: switch logging facility to ui.log() interface The "pager subcommand" message is removed since ui isn't accessible there. I think that's okay as cmdtable[cmd]() will call attachio() and some debug message will be printed.

commandserver: install logger to record server events through canonical API The global commandserver.log() will be replaced with this.

commandserver: enable logging when server process started This allows us to keep track of server events before client connects to the server. Tests will be added later. Currently there's no log() call to check if things are working well.

test-commandserver: change way of triggering early crash Future patches will move the logging facility out of the server class, so cmdserver.log can't be (ab)used for this purpose. Instead, let's hook the factory function to raise exception.

loggingutil: add basic logger backends These classes will be used in command server. They are similar to the blackboxlogger, but it can't be factored out since the blackbox is so tightly coupled with a repo object.

merge with stable

rust: peek_mut optim for lazy ancestors This is one of the two optimizations that are also present in the Python code: replacing pairs of pop/push on the BinaryHeap by single updates, hence having it under the hood maintain its consistency (sift) only once. On Mozilla central, the measured gain (see details below) is around 7%. Creating the PeekMut object by calling peek_mut() right away instead of peek() first is less efficient (gain is only 4%, stats not included). Our interpretation is that its creation has a cost which is vasted in the cases where it ends by droping the value (Peekmut::pop() just does self.heap.pop() anyway). On the other hand, the immutable peek() is very fast: it's just taking a reference in the underlying vector. The Python version still has another optimization: if parent(current) == current-1, then the heap doesn't need to maintain its consistency, since we already know that it's bigger than all the others in the heap. Rust's BinaryHeap doesn't allow us to mutate its biggest element with no housekeeping, but we tried it anyway, with a copy of the BinaryHeap implementation with a dedicaded added method: it's not worth the technical debt in our opinion (we measured only a further 1.6% improvement). One possible explanation would be that the sift is really fast anyway in that case, whereas it's not in the case of Python, because it's at least partly done in slow Python code. Still it's possible that replacing BinaryHeap by something more dedicated to discrete ordered types could be faster. Measurements on mozilla-central: Three runs of 'hg perfancestors' on the parent changeset: Moyenne des médianes: 0.100587 ! wall 0.100062 comb 0.100000 user 0.100000 sys 0.000000 (best of 98) ! wall 0.135804 comb 0.130000 user 0.130000 sys 0.000000 (max of 98) ! wall 0.102864 comb 0.102755 user 0.099286 sys 0.003469 (avg of 98) ! wall 0.101486 comb 0.110000 user 0.110000 sys 0.000000 (median of 98) ! wall 0.096804 comb 0.090000 user 0.090000 sys 0.000000 (best of 100) ! wall 0.132235 comb 0.130000 user 0.120000 sys 0.010000 (max of 100) ! wall 0.100258 comb 0.100300 user 0.096000 sys 0.004300 (avg of 100) ! wall 0.098384 comb 0.100000 user 0.100000 sys 0.000000 (median of 100) ! wall 0.099925 comb 0.100000 user 0.100000 sys 0.000000 (best of 98) ! wall 0.133518 comb 0.140000 user 0.130000 sys 0.010000 (max of 98) ! wall 0.102381 comb 0.102449 user 0.098265 sys 0.004184 (avg of 98) ! wall 0.101891 comb 0.090000 user 0.090000 sys 0.000000 (median of 98) Mean of the medians: 0.100587 On the present changeset: ! wall 0.091344 comb 0.090000 user 0.090000 sys 0.000000 (best of 100) ! wall 0.122728 comb 0.120000 user 0.110000 sys 0.010000 (max of 100) ! wall 0.093268 comb 0.093300 user 0.089300 sys 0.004000 (avg of 100) ! wall 0.092567 comb 0.100000 user 0.090000 sys 0.010000 (median of 100) ! wall 0.093294 comb 0.080000 user 0.080000 sys 0.000000 (best of 100) ! wall 0.144887 comb 0.150000 user 0.140000 sys 0.010000 (max of 100) ! wall 0.097708 comb 0.097700 user 0.093400 sys 0.004300 (avg of 100) ! wall 0.094980 comb 0.100000 user 0.090000 sys 0.010000 (median of 100) ! wall 0.091262 comb 0.090000 user 0.080000 sys 0.010000 (best of 100) ! wall 0.123772 comb 0.130000 user 0.120000 sys 0.010000 (max of 100) ! wall 0.093188 comb 0.093200 user 0.089300 sys 0.003900 (avg of 100) ! wall 0.092364 comb 0.100000 user 0.090000 sys 0.010000 (median of 100) Mean of the medians is 0.0933 Differential Revision: https://phab.mercurial-scm.org/D5358

fuzz: grep away HAVE_GETC_UNLOCKED in pyconfig.h to avoid msan badness Per discussion with Greg Smith and the patches on https://bugs.python.org/issue35214. This, combined with the previous patch, fixes msan builds on oss-fuzz. Differential Revision: https://phab.mercurial-scm.org/D5363

fuzz: more correctly specify CFLAGS and LDFLAGS when building Python Gets us closer to a working msan build alongside our asan build. Differential Revision: https://phab.mercurial-scm.org/D5362

tests: stabilize test-blackbox.t on Windows I didn't look into why the error is more detailed, but that seems like it's a good thing (other than for recording tests).

tests: stabilize for recent wcache changes This goes with 47e3f554df35::d5622dfe4ba3. I'm not sure if it was really expected that there would be no wcache directory if neither execbit nor symlink is supported.

extdiff: avoid double backslashes in the displayed tool path on Windows This shows the tool path in the help, and changed in 67b180c0e263. uirepr() already does the same thing, but that undoes the mangling in its call to repr().

contrib: add a helper script that help to build interesting repositories The script is dedicated to building a couple of repositories that should be interesting to run discovery from one another. It seems a common enough need to contribute it upstream.

py3: listify filter() to call len() on it Differential Revision: https://phab.mercurial-scm.org/D5354

loggingutil: document openlogfile() This function will be used later for command-server logging.

loggingutil: extract openlogfile() and proxylogger to new module This module isn't placed under the "utils" package since it needs "ui" to process things. It's called "loggingutil", not "logutil" because the word "log" is too obscure in our codebase.

blackbox: pass in options to _openlogfile() as arguments This prepares for extracting utility function from the blackbox module.

blackbox: just try writing to repo.vfs and update lastlogger on success This is simpler and more robust. Before, an empty ".hg" directory would be created if it's removed after checking vfs.isdir('.').

vfs: add option to not create parent directories implicitly In blackbox, we don't want to create a ".hg" directory by mistake. This provides a race-safe option to achieve that.

repo: add a `wcachevfs` to access the `.hg/wcache/` directory This wvfs will allow us to migrate various cache to the new `wcache` directory. Helping with cache issues with "share".

cache: create `wcache` directory at init time The cache directory will be needed very quickly, so it seems simpler to create it early to make sure it has the same owner and permission than the other directory in the repository.

cache: create `cache` directory at init time The cache directory will be needed very quickly, so it seems simpler to create it early to make sure it has the same owner and permission than the other directory in the repository.

check-exec: write file in 'wcache' instead of 'cache' Some cache are relevant or affected by the working copy used. So the `.hg/cache` directory is not the best place for them because multiple shared repository can end up fighting over them. To address this issue, we introduce a new 'wcache' directory to host this kind of cache. The first user are the `checkisexec` type file. These files describe property of the working copy and fit the use-case well.

mmapindex: set default to 1MB mmapping index is more efficient if we only need a small part of it. The 1MB value has been picked arbitrarily, a lower value might be better. On a large repository with a 60MB index, we see the following performance gain: hg perfindex before: ! wall 0.032023 comb 0.040000 user 0.000000 sys 0.040000 (best of 100) after: ! wall 0.000196 comb 0.000000 user 0.000000 sys 0.000000 (best of 1060) The speed boost benefit all cases, including the one where the full index needs to be parsed. hg perfindex --rev 0 before: ! wall 0.040673 comb 0.030000 user 0.000000 sys 0.030000 (best of 100) after ! wall 0.010713 comb 0.020000 user 0.010000 sys 0.010000 (best of 212) This gain reflect in higher level operation: hg perfbookmarks --clear-revlogs before: ! wall 0.161339 comb 0.160000 user 0.130000 sys 0.030000 (best of 56) after: ! wall 0.123228 comb 0.120000 user 0.120000 sys 0.000000 (best of 68)

mmapindex: move the 'mmapindexthreshold' option out of experimental The option is useful and should be advertised more. We move it out of experimental as a first step. The `storage` section is selected as this is related to how the storage is accessed. A new 'performance' section might be more appropriate. We move from 'mmapindexthreshold` to `mmap-threshold` as non-index item are also suitable for mmap (eg: the rev-branch-cache). If relevant, we can introduce sub-option `mmap-threshold.revlog-index` later.

perf: add a --rev attribute to perfindex This allow for benchmarking the time necessary to look for other version than the tip.

perf: update perfindex to be more realistic The previous code was creating a revlog manually, we now use the actual `localrepo` method to create it. We have to jump though extra hops to work around the impact of filecache.

match: drop unnecessary wrapping of regex in group It seems the regexes have been wrapped in an unnamed group since b6c42714d900 (Add locate command., 2005-07-05). In that commit, the grouping was needed because there was a "head" ('^') added before the group and a "tail" (os.sep) added after it. It seems the head was moved inside the group in 1c0c413cccdd (Get add and locate to use new repo and dirstate walk code., 2005-07-18) and the tail was moved inside the group in 89985a1b3427 (Clean up walk and changes code to use normalised names properly., 2005-07-31), So it seems to me that we've carried around the unnecessary group for 13 years. This patch removes it. Differential Revision: https://phab.mercurial-scm.org/D5352

match: use _BASE_SIZE instead of magic value 4 Differential Revision: https://phab.mercurial-scm.org/D5351

match: make "groupsize" include the trailing "|" I think this is a little easier to follow and it will simplify later patches too. Differential Revision: https://phab.mercurial-scm.org/D5350

match: fix an unaligned (but harmless) indent Differential Revision: https://phab.mercurial-scm.org/D5349

match: raise an Abort error instead of OverflowError This case of OverflowError (one single pattern being too large) has never been properly caught in the past.

match: avoid translating glob to matcher multiple times for large sets For hgignore with many globs, the resulting regexp might not fit under the 20K length limit. So the patterns need to be broken up in smaller pieces. Before this change, the logic was re-starting the full process from scratch for each smaller pieces, including the translation of globs into regexp. Effectively doing the work over and over. If the 20K limit is reached, we are likely in a case where there is many such glob, so exporting them is especially expensive and we should be careful not to do that work more than once. To work around this, we now translate glob to regexp once and for all. Then, we assemble the resulting individual regexp into valid blocks. This raises a very significant performance win for large `.hgignore file`: Before: ! wall 0.153153 comb 0.150000 user 0.150000 sys 0.000000 (median of 66) After: ! wall 0.059793 comb 0.060000 user 0.060000 sys 0.000000 (median of 100)

match: extract function that group regexps

match: test for overflow error in pattern If a single pattern is too large to handle, we raise an exception. This case is now doctested.

match: extract a literal constant into a symbolic one

tests: apply binary mode to output in seq.py I noticed this when playing with running tests using WSL, and iterating over the output yielded '0\r', '1\r',... Most of the other *.py tools do this, and `seq` on MSYS lacks '\r' in the output, so this is more consistent.

perf: add a `--clear-caches` to `perfbranchmapupdate` This flag will help to measure the time we spend loading various cache that support the branchmap update. Example for an 500 000 revisions repository: hg perfbranchmapupdate --base 'not tip' --target 'tip' ! wall 0.000860 comb 0.000000 user 0.000000 sys 0.000000 (best of 336) hg perfbranchmapupdate --base 'not tip' --target 'tip' --clear-caches ! wall 0.029494 comb 0.030000 user 0.030000 sys 0.000000 (best of 100)

perf: start from an existing branchmap if possible If the --base set if a superset of one of the cached branchmap, we should use as a starting point. This greatly help the overall runtime of `hg perfbranchmapupdate` For example, for a repository with about 500 000 revisions, using this trick make the command runtime move from about 200 second to about 10 seconds. A 20x gain.