win32: close the handles associated with a spawned child process Probably not a big deal because at this point, the call is only used when spawning a daemonized server. In that case, the parent process goes away first, so it won't prevent the child from being cleaned up.

rust: remove comment about error handling of AncestorsIterator To be align with 443eb4bc41af "rust: propagate error of index_get_parents() properly." Spotted by Georges Racinet.

test: fix test-http-bad-server with current python 2.7 https://github.com/python/cpython/pull/2825 changed the exception message for empty http status line. Differential Revision: https://phab.mercurial-scm.org/D5412

perf: add perfprogress command I've noticed that progress bars can add significant overhead to tight loops. Let's add a perf command that attempts to isolate that overhead. With a default hgrc, iteration over 1M items appears to take ~3.75s on my machine. Profiling reveals ~28% of time is spent in ui.configbool() resolving the value of the progress.debug config option. Even if I set progress.disable=true, execution still takes ~2.60s, with ~59% of the time spent in ui.configbool(). Differential Revision: https://phab.mercurial-scm.org/D5407

wireprotov2: unify file revision collection and linknode derivation The old mechanism for choosing which file revisions to send in the haveparents=True case was buggy in multiple ways - the most severe of which being that file revisions were excluded when they shouldn't have been. This commit unifies the logic for deriving the filenodes that will be sent by the "filesdata" command. We now consistently read files data from manifests. The "haveparents" argument now controls whether we iterate ctx.files() or use the full manifest to derive relevant files. The logic here is still woefully lacking to fully support shallow clones. It will require an API break to fully address. This commit should at least make the server APIs emit proper data, which is strictly better than before. Differential Revision: https://phab.mercurial-scm.org/D5406

wireprotov2: send linknodes to emitfilerevisions() Previously, linknodes were calculated within emitfilerevisions() by using filectx.introrev(), which would always use the linkrev/linknode as recorded by storage. This is wrong for cases where the receiver doesn't have the changeset the linknode refers to. This commit changes the logic for linknode emission so the mapping of filenode to linknode is computed by the caller and passed into emitfilerevisions(). As part of the change, linknodes for "filesdata" in the haveparents=False case are now correct: the existing code performed a manifest walk and it was trivial to plug in the correct linknode. However, behavior for the haveparents=True case is still wrong because it relies on filtering linkrevs against the outgoing set in order to determine what to send. This will be fixed in a subsequent commit. The change test test-wireproto-exchangev2-shallow.t is a bit wonky. The test repo has 6 revisions. The changed test is performing a shallow clone with depth=1. So, only file data for revision 5 is present locally. So, the new behavior of associating the linknode with revision 5 for every file revision seems correct. Of course, when backfilling old revisions, we'll want to update the linknode. But this problem requires wire protocol support and we'll cross that bridge later. Differential Revision: https://phab.mercurial-scm.org/D5405

tests: add tests for server-side linknode adjustment with wireprotov2 The current implementation of linknode serving in wireprotov2 simply serves up the linkrev/linknode as stored: it doesn't attempt to adjust the linknode to what the receiver is aware of. This can result in the client seeing a linknode referencing a changeset that is unknown to it. This commit adds test coverage of that scenario. The tests in test-wireproto-command-filesdata.t demonstrate two failures. First, the linknode refers to a changeset not in the available set. Second, the server doesn't send a file revision that it should have (because of linkrev filtering). The test in test-wireproto-exchange.t demonstrates that the lack of a file revision results in a corrupted repository on the client. Differential Revision: https://phab.mercurial-scm.org/D5404

tests: fix empty commit in test This was almost certainly a bug. Differential Revision: https://phab.mercurial-scm.org/D5403

tests: add sparserevlog requirement to test-sqlitestore.t Looks like this test was missed when enabling sparse revlogs by default. Differential Revision: https://phab.mercurial-scm.org/D5402

tests: remove all transitional configuration Now that sparse-revlog is enabled by default, we no longer needs it. Differential Revision: https://phab.mercurial-scm.org/D5346

sparse-revlog: enabled by default The feature provides large benefits. It now seems mature enough to be enabled by default. * It solves catastrophic issues regarding delta storage in revlog, * It allows for shorter delta chain in all repositories, improving performances. Running benchmark of a wide range of operation did not reveal problematic impact. Performance gains are observed where expected. The format is supported by Mercurial version 4.7. So it seems safe to enable it by default now. Here is a reminder of key numbers regarding this delta strategy effect on repository size and performance. Effect on Size: =============== For repositories with a lot of branches, sparse-revlog significantly improve size, fixing limitation associated with the span of a delta chain. In addition, sparse-revlog, deal well with limitations of the delta chain length. For large repositories, this allows for a stiff reduction of the delta chain without a problematic impact on the repository size. This delta chain length improvement helps all repositories, not just the ones with many branches. As a reminder, here are the default chain limits for each "format": * no-sparse: none * sparse: 1000 Mercurial --------- Manifest Size: limit | none | 1000 ------------|-------------|------------ no-sparse | 6 143 044 | 6 269 496 sparse | 5 798 796 | 5 827 025 Manifest Chain length data limit || none || 1000 value || average | max || average | max ------------||---------|---------||---------|--------- no-sparse || 429 | 1 397 || 397 | 1 000 sparse || 326 | 1 290 || 313 | 1 000 Full Store Size limit | none | 1000 ------------|-------------|------------ no-sparse | 46 944 775 | 47 166 129 sparse | 46 622 445 | 46 723 774 pypy ---- Manifest Size: limit | none | 1000 ------------|-------------|------------ no-sparse | 52 941 760 | 56 200 970 sparse | 26 348 229 | 27 384 133 Manifest Chain length data limit || none || 1000 value || average | max || average | max ------------||---------|---------||---------|--------- no-sparse || 769 | 3 889 || 390 | 1 000 sparse || 1 223 | 3 846 || 495 | 1 000 Full Store Size limit | none | 1000 ------------|-------------|------------ no-sparse | 336 050 203 | 339 309 413 sparse | 338 673 985 | 339 709 889 Mozilla ------- Manifest Size: limit | none | 1000 ------------|----------------|--------------- no-sparse | 215 096 339 | 1 708 853 525 sparse | 188 947 271 | 278 894 170 Manifest Chain length data limit || none || 1000 value || average | max || average | max ------------||---------|---------||---------|-------- no-sparse || 20 454 | 59 562 || 491 | 1 000 sparse || 23 509 | 69 891 || 489 | 1 000 Full Store Size limit | none | 1000 ------------|----------------|--------------- no-sparse | 2 377 578 715 | 3 876 258 798 sparse | 2 441 677 137 | 2 535 997 381 Netbeans -------- Manifest Size: limit | none | 1000 ------------|----------------|--------------- no-sparse | 130 088 982 | 741 590 565 sparse | 118 836 887 | 159 161 207 Manifest Chain length data limit || none || 1000 value || average | max || average | max ------------||---------|---------||---------|--------- no-sparse || 19 321 | 61 397 || 510 | 1 000 sparse || 21 240 | 61 583 || 503 | 1 000 Full Store Size limit | none | 1000 ------------|----------------|--------------- no-sparse | 1 160 013 008 | 1 771 514 591 sparse | 1 164 959 988 | 1 205 284 308 Private repo #1 --------------- Manifest Size: limit | none | 1000 ------------|-----------------|--------------- no-sparse | 33 725 285 081 | 33 724 834 190 sparse | 350 542 420 | 423 470 579 Manifest Chain length data limit || none || 1000 value || average | max || average | max ------------||---------|---------||---------|--------- no-sparse || 282 | 8 885 || 113 | 1 000 snapshot || 3 655 | 8 951 || 530 | 1 000 Full Store Size limit | none | 1000 ------------|----------------|--------------- no-sparse | 41 544 149 652 | 41 543 698 761 sparse | 8 448 037 300 | 8 520 965 459 Effect on speed: ================ Performances are strongly impacted by the delta chain length. Longer chain results in slower revision restoration. For this reason, the 1000 chain limit introduced by sparse-revlog helps repository with previously large chains a lot. In our corpus, this means `netbeans` and `mozilla-central` who suffered from unreasonable manifest delta chain length. Another way sparse revlog helps, is by producing better delta's. For repositories with many branches, the pathological patterns that resulted in many sub-optimal deltas are gone. Smaller delta help with operations where deltas are directly relevant, like bundle. However, the sparse-revlog logic introduces some extra processing and a more throughout testing of possible delta candidates. Adding an extra cost in some cases. This cost is usually counterbalanced by the other performance gain. However, for smaller repositories not affected by delta chain length issues or branching related issues, this might make things a bit slower. However, these are also repository where revlog performance is dwarfed by other costs. Below are the summary of some timing from the performance test suite running at `http://perf.octobus.net/` for a handful of key commands and operation. It is important to keep in mind that most of this command works on the tip part of the repository. The non-sparse and sparse version produce different delta chains and the tip revision can end up at an arbitrary point of these chains. This will impact some performance number listed in this summary. For the record: here is the delta chain length for the tip revision of manifest log in the benchmarked repository: | no-sparse | sparse | mercurial | 94 | 904 | pypy | 23 | 673 | netbeans | 4158 | 258 | mozilla | 63263 | 781 | As you can see, the chain length for mercurial and pypy turn out to be significantly longer. The netbeans and mozilla one get shorter because these repositories benefit from the maximum chain length. Timing for `hg commit`: ----------------------- The time taken by `hg commit` does not varies significantly, no drawback for using sparse here. | no-sparse | sparse | mercurial | 68.1ms | 66.7ms | pypy | 95.0ms | 94.1ms | netbeans | 614.0ms | 611.0ms | mozilla | 1340.0ms | 1.320.0ms | Check the final section for statistics on a wider array of write. Timing for bundling 10 000 changesets ------------------------------------- The repository that benefits from better delta see a good performance boost. The other ones are not significantly affected. | no-sparse | sparse | mercurial | 3.1s | 3.0s | pypy | 25.1s | 7.5s | netbeans | 24.2s | 17.0s | mozilla | 23.7s | 25.0s | Timing for unbundling 1 000 changesets -------------------------------------- Mercurial and mozilla are unaffected. The pypy repository benefit well from the better delta. However, the netbeans repository takes a visible hit. Digging that difference reveals that it comes from the sparse-revlog bundle having to deal with a snapshot that was re-encoded in the bundle. The slow path for adding new a revision had to be triggered for it, slowing things down. The Sparse versions do not have such snapshot to handle similar cases in the tested configuration. | no-sparse | sparse | mercurial | 519ms | 502ms | pypy | 1.270ms | 886ms | netbeans | 1.370ms | 2.250ms | mozilla | 3.230ms | 3.210ms | Netbeans benefits from the better deltas in other dimensions too. For example, the produced bundle is significantly smaller: * netbeans-no-sparse.hg: 2.3MB * netbeans-sparse.hg: 1.9MB Timing to restore the tip most manifest entry: ---------------------------------------------- Nothing surprising here. The timing for mercurial and pypy are within a small range where they won't affect performance much. In our tested case, they are slower as they use a longer chain. Timing for netbeans and mozilla improves a lot. Removing a significant amount of time. | no-sparse | sparse | mercurial | 1.09ms | 3.15ms | pypy | 4.11ms | 10.70ms | netbeans | 239.00ms | 112.00ms | mozilla | 688.00ms | 198.00ms | Reading 100 revision in descending order: ----------------------------------------- We see the same kind of effect when reading the last 100 revisions. Large boost for netbeans and mozilla, as they use much smaller delta chain. Mercurial and pypy longer chain means slower reads, but nothing gets out of control. | no-sparse | sparse | mercurial | 0.089s | 0.268s | pypy | 0.259s | 0.698s | netbeans | 125.000s | 20.600s | mozilla | 23.000s | 11.400s | Writing from full text: statistic for the last 30K revisions ------------------------------------------------------------ This benchmark adds revisions to revlog from their full text. This is similar to the work done during a commit, but for a large amount of revisions so that we get a more relevant view. We see better overall performances with sparse-revlog. The very worst case is usually slower with sparse-revlog, but does not gets out of control. For the vast majorities of the other writes, sparse-revlog is significantly faster for larger repositories. This is reflected in the accumulated rewrite time for netbeans and mozilla. The notable exception is the pypy repository where things get slower. The extra processing is not balanced by shorter delta chain. However, this is to be seen as a blocking issue. First, the overall time spend dealing with revlog for the repository pypy size is small compared to the other costs, so we get slower on operations that matter less than for other larger repository. Second, we still get nice size benefit from using sparse-revlog, smaller repo size brings other usability and speed benefit (eg: bundle size). max time | no-sparse | sparse | mercurial | 0.010143s | 0.045280s | pypy | 0.034924s | 0.243288s | netbeans | 0.605371s | 2.130876s | mozilla | 1.478342s | 3.424541s | 99% time | no-sparse | sparse | mercurial | 0.003774s | 0.003758s | pypy | 0.017387s | 0.025310s | netbeans | 0.576913s | 0.271195s | mozilla | 1.478342s | 0.449661s | 95% time | no-sparse | sparse | mercurial | 0.002069s | 0.002120s | pypy | 0.010141s | 0.014797s | netbeans | 0.540202s | 0.258644s | mozilla | 0.654830s | 0.243440s | full time | no-sparse | sparse | mercurial | 14.15s | 14.87s | pypy | 90.50s | 137.12s | netbeans | 6401.06s | 3411.14s | mozilla | 3086.89s | 1991.97s | Differential Revision: https://phab.mercurial-scm.org/D5345

test: preemptively disable sparse-revlog for some of test-upgrade-repo Some part of this test assumes sparse-revlog is off. We enforce it explicitly. Differential Revision: https://phab.mercurial-scm.org/D5344

test: enable sparse-revlog for test-wireproto-content-redirects.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5343

test: enable sparse-revlog for test-wireproto-command-capabilities.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5342

test: enable sparse-revlog for test-wireproto-caching.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5341

test: enable sparse-revlog for test-treemanifest.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5340

test: enable sparse-revlog for test-stream-bundle-v2.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5339

test: enable sparse-revlog for test-ssh.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5338

test: enable sparse-revlog for test-ssh-proto-unbundle.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5337

test: enable sparse-revlog for test-ssh-proto.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5336

test: enable sparse-revlog for test-ssh-bundle1.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5335

test: enable sparse-revlog for test-sparse-requirement.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5334

test: enable sparse-revlog for test-revlog-v2.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5333

test: enable sparse-revlog for test-repo-compengines.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5332

test: enable sparse-revlog for test-remotefilelog-log.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5331

test: enable sparse-revlog for test-remotefilelog-clone-tree.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5330

test: enable sparse-revlog for test-remotefilelog-clone.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5329

test: enable sparse-revlog for test-phases.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5328

test: enable sparse-revlog for test-narrow-clone.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5327

test: enable sparse-revlog for test-narrow-clone-stream.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5326

test: enable sparse-revlog for test-narrow-clone-no-ellipsis.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5325

test: enable sparse-revlog for test-lfs-serve.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5324

test: enable sparse-revlog for test-lfs-largefiles.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5323

test: enable sparse-revlog for test-lfconvert.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5322

test: enable sparse-revlog for test-init.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5321

test: enable sparse-revlog for test-http-protocol.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5320

test: enable sparse-revlog for test-hgweb-commands.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5319

test: enable sparse-revlog for test-debugcommands.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5318

test: enable sparse-revlog for test-clone-uncompressed.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5317

test: enable sparse-revlog for test-clonebundles.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5316

test: enable sparse-revlog for test-bundle.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5315

test: make sure sparse-revlog does not interfer with test-parseindex.t That test carefully craft revlog to reproduce issues. We do not want new revlog behavior to interfere with that. Differential Revision: https://phab.mercurial-scm.org/D5314

test: explicitly disable sparse-revlog in test-http-bad-server.t The `test-http-bad-server.t` test is very fragile with hard-coded number all around. Adding any new capability have massive consequences on it that are long and tiresome to handle. Since the test check more generic protocol level behavior, it does not need to follow all new capabilities. Disabling the capabilities is a much simpler way to avoid the issue. We make sure `sparse-revlog` will not impact this test once turned on by default. Differential Revision: https://phab.mercurial-scm.org/D5313

test: explicitly disable sparse-revlog in test-generaldelta.t That test is about testing generaldelta without further improvement. We ensure it will stay that way even if sparse-revlog is turned on by default. The sparse-revlog format is tested on his own in a different test file. Differential Revision: https://phab.mercurial-scm.org/D5312

sparse-revlog: disable sparse-revlog if config disable general-delta Sparse-revlog requires general-delta in order to work. If general-delta is explicitly disabled through configuration, we should honor that and also disabled sparse-revlog. This change will be more relevant when sparse-revlog is enabled by default. Differential Revision: https://phab.mercurial-scm.org/D5311

test: fix config typo in test-upgrade-repo.t Differential Revision: https://phab.mercurial-scm.org/D5310

tests: use $RUNTESTDIR instead of $TESTDIR in narrow-library.sh This is done so that third party extensions can use narrow-library.sh in tests. Differential Revision: https://phab.mercurial-scm.org/D5400

tests: update network related errors for Debian 9 We have a CI job that runs the Mercurial tests in parallel. Some of the network related failures seems to be different on the environment. Oddly, those failures happens only when running the tests in parallel, not when running the test file only. I have no idea how to get the windows formatted message for the error, if someone could give me an hand, I will update this changeset with the value. Differential Revision: https://phab.mercurial-scm.org/D5401

util: implement pop() on lrucachedict This moves __delitem__() to pop() as the requirement is pretty much the same, and reimplement __delitem__() by using pop().

util: add method to peek item in lrucachedict I want a function that doesn't unnecessarily update the internal state of the cache dict after fork().

commandserver: loop over selector events An IPC socket will be waited by the same selector.

commandserver: remove redundant "if True" block

commandserver: extract handler of new socket connection This prepares for adding an IPC socket.

commandserver: pass around option to hook repo instance creation This is necessary to wrap a repo instance so the master process will be notified on repo.close().

py3: stop subscripting socket.error In 3.3 and later, this is now an alias for OSError. I hacked up the server code enough that I was able to trigger the exception handler in server.py from test-http-bundle1.t. Other instances of this either subscript through the `args` member, or reference the errno or strerror attributes. Note that on Windows, the errno value seems to reflect the Winsock error, so the various tests for EPIPE seem like they would always fail. But that seems to be the case in py2 as well.

test: update test-logtoprocess.t to not requires less The test previously assumed that a working pager was present in the test environment. Update it to use the fakepager instead. Differential Revision: https://phab.mercurial-scm.org/D5399

py3: quote several instances of $PYTHON for Windows Python3 lives in Program Files by default. The last remaining unquoted instance is in test-hghave.t. I can't figure out how to quote that, as it either complains: 'c:\\Program' is not recognized as an internal or external command, operable program or batch file. or $TESTTMP.sh: line 22: "C:/Program Files/Python37/python.exe": $ENOENT$

py3: adapt test-largefiles-wireproto.t to python3

py3: byteify test-push-race.t

py3: convert filename to bytes in test-hardlinks.t Windows was complaining about passing str to CreateFileA() in win32.py.

narrow: keep bookmarks temporarily stripped for as long as commits are The narrow extension also has support for shallowness and for inserting older commits on pull. It works by temporarily stripping newer commits, adding the older commits, then re-applying the stripped bundle. The regular Mercurial server uses that when you widen, although it shouldn't be necessary there. Our Google-internal server does it when the user requests an older commit. Our Google-internal tests fail since 7caf632e30c3 (filecache: unimplement __set__() and __delete__() (API), 2018-10-20). I haven't quite understood the problem, but it's related to the way we temporarily hide bookmarks while the commits they point to are stripped. When a transaction is started, Mercurial tries to read various things from the repo for the transaction summary. That leads to computation of hidden commits, which leads to an attempt to find commits pinned by bookmarks. This is the reason we temporarily hide the bookmarks. I think the aforementioned commit makes the restored bookmarks visible earlier than before (which seems like an improvement), so we can no longer incorrectly rely on the repo._bookmarks field being cached too long (IIUC). This patch makes it so we restore the temporarily hidden bookmarks only after the temporary bundle has been re-applied. It also adapts the code to update the repo.__bookmarks field using the pattern described in the aforementioned commit instead of writing directly to the fiels. I have spent many hours trying to understand what was going on here, but I still don't know if this can also happen without our custom server. So this patch unfortunately does not add any tests; I have only been able to test the fix using our Google-internal tests. Differential Revision: https://phab.mercurial-scm.org/D5398

narrow: drop obsolete support for old Mercurial Not needed since narrow is shipped with core. Differential Revision: https://phab.mercurial-scm.org/D5397

tests: test that narrow preserves bookmarks on widen When widening inserts older commits in the changelog, we have to preserve bookmarks so they are not removed by the call to repair.strip() we temporarily do. We didn't have any tests for that before. Differential Revision: https://phab.mercurial-scm.org/D5396

rebase: clarify that commits that become empty are skipped Our message just said that it created no changes, but we didn't explicitly say that that meant that the the commit was not rebased. It also wasn't clear why it created no changes, so this patch also clarifies that that was because the changes were already in the destination. Differential Revision: https://phab.mercurial-scm.org/D5395

rebase: format commit in usual way when explaining that it became empty We use the _ctxdesc() for other similar messages, so it seems we should use it here too. Differential Revision: https://phab.mercurial-scm.org/D5394

rust: propagate error of index_get_parents() properly Before, rustla_contains() would return 0 on error, and the exception would be cleared or noticed somewhere else. We need to propagate the error from AncestorsIterator up to the FFI surface.

rust: look up HgRevlogIndex_GetParents() from symbol table And removes the unused index_get_parents_checked() function. I expect the Index struct will be turned into a pyobject type, though I haven't written any PoC-level patches yet.

revlog: add public CPython function to get parent revisions Since this is a public function, it validates the input revision, and supports nullrev. index_get_parents_checked() will be replaced by this function.

revlog: rename indexType to HgRevlogIndex_Type as it's a global symbol It follows the CPython naming convention, but uses Hg instead of Py.

revlog: export symbol of indexType The idea is to wrap the index object with rust-cpython. I haven't tried it, but it should be doable. We'll probably need a better interface than raw function pointers to do more in Rust.

setup: fix path of rust source dependencies Appears that they were screwed up while moving the source files around. Since rust-direct-ffi sources are automatically added to the depends list, we only need to list hg-core sources.

rebase: remove now-unnecessary arguments to _abort() Since _abort() is now an instance method, we don't need to pass state that's available on "self". Differential Revision: https://phab.mercurial-scm.org/D5393

rebase: move abort() onto rebaseruntime The function depends on a lot of the state of rebaseruntime, so it makes sense for it to be an instance method. This will let us remove many of the arguments to the method. That will be done in a later patch. Differential Revision: https://phab.mercurial-scm.org/D5392

cleanupnodes: trust caller when "moves" is not None If "moves" (indicating how to move bookmarks) is None, we fill it out based on "replacements" (indicating which obsmarkers to add). If "moves" is not None, we would still add items based on "replacements". This makes it impossible to pass "moves={}" and not move bookmarks, which surprised me. The only caller that currently passes a value for "moves" was the rebase extension and there we were already adding bookmark moves corresponding to obsmarker additions, so it should not be impacted. Differential Revision: https://phab.mercurial-scm.org/D5391

shelve: change transaction description from "commit" to "shelve" "commit" was probably a copy&paste mistake. Differential Revision: https://phab.mercurial-scm.org/D5390

shelve: drop unnecessary backup of dirstate for phase-based case Regular shelve has a hack using an uncommitted transaction that's then aborted at the end of the operation. It preserves the dirstate across the abort, however, by saving a backup copy of it. Phase-based shelve instead commits the transaction, so the hack shouldn't be necessary there. Differential Revision: https://phab.mercurial-scm.org/D5389

tests: split test-shelve.t in two test-shelve.t dominated run time for all shelve tests. Before: # Ran 9 tests, 1 skipped, 0 failed. real 0m43.568s user 2m15.822s sys 0m40.857s After: # Ran 11 tests, 1 skipped, 0 failed. real 0m24.574s user 2m21.354s sys 0m40.435s Differential Revision: https://phab.mercurial-scm.org/D5388

remotefilelog: rely on progress helper for keeping track of position The progress helper class keeps track of its current position, so we don't need a "count" variable for that. Differential Revision: https://phab.mercurial-scm.org/D5386

remotefilelog: reduce use of "count" container We had already given the initial value of "count[0]" a name, so just use that when applicable. Differential Revision: https://phab.mercurial-scm.org/D5385

remotefilelog: replace a "a=[expr]; b=a[0]" by "b=expr; a = [b]" Simpler is better. Differential Revision: https://phab.mercurial-scm.org/D5384

remotefilelog: remove an unnecessary update of "count" container The "count" container is never used after this point, so there is no need to update it. Differential Revision: https://phab.mercurial-scm.org/D5383

remotefilelog: avoid temporarily using "count" variable as synonym for "total" The "count" variable is generally used for updating progress, but early in fileserverclient.request(), its used to mean the total count. We already have another "total" variable for that, so it seems much clearer to use that. Differential Revision: https://phab.mercurial-scm.org/D5382

remotefilelog: use progress helper in fileserverclient Differential Revision: https://phab.mercurial-scm.org/D5381

remotefilelog: use progress helper in shallowrepo Differential Revision: https://phab.mercurial-scm.org/D5380

remotefilelog: use progress helper in shallowbundle Differential Revision: https://phab.mercurial-scm.org/D5379

remotefilelog: use progress helper in repack Differential Revision: https://phab.mercurial-scm.org/D5378

remotefilelog: use progress helper in remotefilelogserver Differential Revision: https://phab.mercurial-scm.org/D5377

remotefilelog: use progress helper in basestore Differential Revision: https://phab.mercurial-scm.org/D5376

remotefilelog: use progress helper when analyzing repos Differential Revision: https://phab.mercurial-scm.org/D5375

fastannotate: use progress helper Differential Revision: https://phab.mercurial-scm.org/D5374

debugbuildannotatecache: use progress helper Differential Revision: https://phab.mercurial-scm.org/D5373

upgrade: clarify "aggressivemergedelta" handling We rename "aggressivemergedelta" argument to "forceaggressivemergedelta". The previous argument naming could infer an absolute control on the behavior. However, the code show we respect the config option if set.

upgrade: test that fncache is valid after repository upgrade The behavior was correct. Now it is correct and tested.

tests: update `rm` invocation for new location of checklink This should fix the test on FreeBSD. Differential Revision: https://phab.mercurial-scm.org/D5387

bookflow: new extension for bookmark-based branching This extension should be helpful for feature branches - based workflows. At my company we first considered branches, but weren't sure about creating a lot of permanent objects. We tried bookmarks, but found some scenarios to be difficult to control. The main problem, was the active bookmark being moved on update. Disabling that made everything a lot more predictable. Bookmarks move on commit, and updating means switching between them. The extension also implements a few minor features to better guide the workflow: - hg bookmark NAME can be ambiguous (create or move), unlike hg branch. The extension requires -rev to move. - require an active bookmark on commit. - some bookmarks can be protected (like @), useful for teams, that require code reviews. - block creation of new branches. The initial implementation requires no changes in the core, but it does rely on some implementation details. I thought it may be useful to discuss the functionality first, and then focus making the code more robust. Differential Revision: https://phab.mercurial-scm.org/D4312

rust: make clean takes care of rust/target This isn't the prettiest way of doing it, but it doesn't require looking up cargo, or wondering whether that should be part of setup.py clean. Differential Revision: https://phab.mercurial-scm.org/D5369

rust: rename local variables in AncestorsIterator::next It was confusing to have p1 and parents.1 ; (p1, p2) is clearer. Differential Revision: https://phab.mercurial-scm.org/D5365

tests: stabilize test-inherit-mode.t on FreeBSD and macOS again This is the same fix as 90e26ef4cbb1, just repeated on the new file location. Differential Revision: https://phab.mercurial-scm.org/D5371

tests: drop redundant "#if execbit" from test-rebase-inmemory.t The whole file is already guarded by "#require execbit". Differential Revision: https://phab.mercurial-scm.org/D5366

narrow: don't resurrect old commits when narrowing (don't strip obsmarkers) If you have an old obsolescence-chain of commits that has been pruned and you narrow your repo so that some of those commits get stripped (because they affected the removed paths), then we would currently resurrect the commit that came before (along the obsmarker chain) the last stripped commit. That happens by the usual rules for obsmarker-stripping. However, it's quite surprising when it happens when you narrow your repo. This patch makes narrowing not strip obsmarkers. Differential Revision: https://phab.mercurial-scm.org/D5364

commandserver: turn server debug messages into logs They were ui.debug() just because commandserver.log() was noop if no client connected.

commandserver: add config knob for various logging options The default rotating options are copied from the blackbox extension.

commandserver: expand log path for convenience This allows us to set the log path relative to $XDG_RUNTIME_DIR, for instance. [cmdserver] log = $XDG_RUNTIME_DIR/chg/server.log

commandserver: switch logging facility to ui.log() interface The "pager subcommand" message is removed since ui isn't accessible there. I think that's okay as cmdtable[cmd]() will call attachio() and some debug message will be printed.

commandserver: install logger to record server events through canonical API The global commandserver.log() will be replaced with this.

commandserver: enable logging when server process started This allows us to keep track of server events before client connects to the server. Tests will be added later. Currently there's no log() call to check if things are working well.

test-commandserver: change way of triggering early crash Future patches will move the logging facility out of the server class, so cmdserver.log can't be (ab)used for this purpose. Instead, let's hook the factory function to raise exception.

loggingutil: add basic logger backends These classes will be used in command server. They are similar to the blackboxlogger, but it can't be factored out since the blackbox is so tightly coupled with a repo object.

hgweb: register web.comparisoncontext to the config table This was caught in some server side logging added to debug py3 issues.

merge with stable

Added signature for changeset 1c8c54cf9725

Added tag 4.8.1 for changeset 1c8c54cf9725

rebase: fix path auditing to audit path relative to repo root (issue5818) Before this patch, when rebasing a file called "foo/bar", we would check e.g. if "/foo" (i.e. rooted at the file system root) was a symlink. Differential Revision: https://phab.mercurial-scm.org/D5361

tests: show bad path auditing in in-memory rebase Thanks to Yuya for providing this test case in https://bz.mercurial-scm.org/show_bug.cgi?id=5818. Differential Revision: https://phab.mercurial-scm.org/D5368

tests: add a missing "cd .." to test-rebase-inmemory.t Differential Revision: https://phab.mercurial-scm.org/D5367

rust: fix possible out-of-bounds read through index_get_parents() index_get_parents() is an internal function, which doesn't check if the specified rev is valid. If rustlazyancestors() were instantiated with an invalid stoprev, it would access to invalid memory region. This is NOT a security fix as there's no Python code triggering the bug, but included in this series to not give a notion about the memory issue fixed by the previous patch.

revlog: fix out-of-bounds access by negative parents read from revlog (SEC) 82d6a35cf432 wasn't enough. Several callers don't check negative revisions but for -1 (nullrev), which would directly lead to out-of-bounds read, and buffer overflow could follow. RCE might be doable with carefully crafted revlog structure, though I don't think this would be useful attack surface.

rust: peek_mut optim for lazy ancestors This is one of the two optimizations that are also present in the Python code: replacing pairs of pop/push on the BinaryHeap by single updates, hence having it under the hood maintain its consistency (sift) only once. On Mozilla central, the measured gain (see details below) is around 7%. Creating the PeekMut object by calling peek_mut() right away instead of peek() first is less efficient (gain is only 4%, stats not included). Our interpretation is that its creation has a cost which is vasted in the cases where it ends by droping the value (Peekmut::pop() just does self.heap.pop() anyway). On the other hand, the immutable peek() is very fast: it's just taking a reference in the underlying vector. The Python version still has another optimization: if parent(current) == current-1, then the heap doesn't need to maintain its consistency, since we already know that it's bigger than all the others in the heap. Rust's BinaryHeap doesn't allow us to mutate its biggest element with no housekeeping, but we tried it anyway, with a copy of the BinaryHeap implementation with a dedicaded added method: it's not worth the technical debt in our opinion (we measured only a further 1.6% improvement). One possible explanation would be that the sift is really fast anyway in that case, whereas it's not in the case of Python, because it's at least partly done in slow Python code. Still it's possible that replacing BinaryHeap by something more dedicated to discrete ordered types could be faster. Measurements on mozilla-central: Three runs of 'hg perfancestors' on the parent changeset: Moyenne des médianes: 0.100587 ! wall 0.100062 comb 0.100000 user 0.100000 sys 0.000000 (best of 98) ! wall 0.135804 comb 0.130000 user 0.130000 sys 0.000000 (max of 98) ! wall 0.102864 comb 0.102755 user 0.099286 sys 0.003469 (avg of 98) ! wall 0.101486 comb 0.110000 user 0.110000 sys 0.000000 (median of 98) ! wall 0.096804 comb 0.090000 user 0.090000 sys 0.000000 (best of 100) ! wall 0.132235 comb 0.130000 user 0.120000 sys 0.010000 (max of 100) ! wall 0.100258 comb 0.100300 user 0.096000 sys 0.004300 (avg of 100) ! wall 0.098384 comb 0.100000 user 0.100000 sys 0.000000 (median of 100) ! wall 0.099925 comb 0.100000 user 0.100000 sys 0.000000 (best of 98) ! wall 0.133518 comb 0.140000 user 0.130000 sys 0.010000 (max of 98) ! wall 0.102381 comb 0.102449 user 0.098265 sys 0.004184 (avg of 98) ! wall 0.101891 comb 0.090000 user 0.090000 sys 0.000000 (median of 98) Mean of the medians: 0.100587 On the present changeset: ! wall 0.091344 comb 0.090000 user 0.090000 sys 0.000000 (best of 100) ! wall 0.122728 comb 0.120000 user 0.110000 sys 0.010000 (max of 100) ! wall 0.093268 comb 0.093300 user 0.089300 sys 0.004000 (avg of 100) ! wall 0.092567 comb 0.100000 user 0.090000 sys 0.010000 (median of 100) ! wall 0.093294 comb 0.080000 user 0.080000 sys 0.000000 (best of 100) ! wall 0.144887 comb 0.150000 user 0.140000 sys 0.010000 (max of 100) ! wall 0.097708 comb 0.097700 user 0.093400 sys 0.004300 (avg of 100) ! wall 0.094980 comb 0.100000 user 0.090000 sys 0.010000 (median of 100) ! wall 0.091262 comb 0.090000 user 0.080000 sys 0.010000 (best of 100) ! wall 0.123772 comb 0.130000 user 0.120000 sys 0.010000 (max of 100) ! wall 0.093188 comb 0.093200 user 0.089300 sys 0.003900 (avg of 100) ! wall 0.092364 comb 0.100000 user 0.090000 sys 0.010000 (median of 100) Mean of the medians is 0.0933 Differential Revision: https://phab.mercurial-scm.org/D5358

fuzz: grep away HAVE_GETC_UNLOCKED in pyconfig.h to avoid msan badness Per discussion with Greg Smith and the patches on https://bugs.python.org/issue35214. This, combined with the previous patch, fixes msan builds on oss-fuzz. Differential Revision: https://phab.mercurial-scm.org/D5363

fuzz: more correctly specify CFLAGS and LDFLAGS when building Python Gets us closer to a working msan build alongside our asan build. Differential Revision: https://phab.mercurial-scm.org/D5362