revset: use changelog's `headrevs` method to compute heads Instead of implementing our own algorithm, we reuse a more generic one. This previous algorithm did not leave much room for laziness so we do not really regress in that regards. A small impact is visible for first/last value in some of the simpler cases. The time needed to compute all values improves overall. Small optimization in the dagop.headrevs function will help to buy this back in the next changesets. There is room to introduce actual laziness in this algorithm, but this is out of scope for this series. This has no visible effect on expensive cases: revset: heads(matching(tip, "author")) plain min max first last reverse rev..rst rev..ast sort sor..rst sor..ast 0) 7.574666 7.545950 7.570743 7.578697 7.525725 7.509929 7.443854 7.488442 7.452880 7.445411 7.689107 1) 7.549390 7.389162 7.529790 7.536297 7.450467 7.555347 7.404586 7.514948 7.542794 7.524787 7.536918 revset: heads(matching(tip, "author")) and -10000:-1 plain min max first last reverse rev..rst rev..ast sort sor..rst sor..ast 0) 7.512533 7.605877 7.382894 7.462109 7.420086 7.575034 7.448452 7.549374 7.457880 7.450308 7.515019 1) 7.548677 7.551832 7.629598 7.494857 7.550554 7.521838 7.451794 error 7.321781 7.546885 7.557523 revset: (-10000:-1) and heads(matching(tip, "author")) plain min max first last reverse rev..rst rev..ast sort sor..rst sor..ast 0) 7.465419 7.570089 7.439594 7.521221 7.498716 7.492922 7.479108 7.552397 7.407888 error 7.468264 1) 7.539866 7.548045 7.491761 7.517170 7.469824 7.501990 7.579102 7.502568 7.578102 7.555754 7.567622 In simpler cases, we see a 10-15% impact when retrieving a single value, the full computation time is equivalent or improved: revset: (-5000:-1000) and heads(-10000:-1) plain min max first last reverse rev..rst rev..ast sort sor..rst sor..ast 0) 0.004244 0.003368 0.003313 0.003367 0.003327 0.004325 0.003401 0.003379 0.004310 0.003359 0.003396 1) 0.003969 93% 0.003862 114% 0.003834 115% 0.003810 113% 0.003822 114% 0.003940 91% 0.003908 114% 0.003814 112% 0.003986 92% 0.003954 117% 0.003816 112% revset: heads(all()) plain min max first last reverse rev..rst rev..ast sort sor..rst sor..ast 0) 0.036503 0.032564 0.030024 0.032378 0.030887 0.036367 0.031713 0.032205 0.036467 0.032286 0.030300 1) 0.036668 0.035347 108% 0.035611 118% 0.035358 109% 0.035726 115% 0.036411 0.035261 111% 0.036096 112% 0.036052 0.035095 108% 0.035792 118% revset: heads(-10000:-1) plain min max first last reverse rev..rst rev..ast sort sor..rst sor..ast 0) 0.003936 0.003218 0.003227 0.003302 0.003328 0.003848 0.003305 0.003252 0.003839 0.003306 0.003279 1) 0.003870 0.003785 117% 0.003821 118% 0.003780 114% 0.003769 113% 0.003776 0.003792 114% 0.003805 117% 0.003810 0.003798 114% 0.003840 117%

revlog: accept a revs argument in `headrevs` Computing the heads of an arbitrary set of revision is useful, we make it possible to do so through the `headrevs` method of the revlog. Right now, this is just calling dagop's implementation. However, we expect to plug a native implementation soon.

revset: inline parents computation to reuse the input argument Before this change, using `heads(xxx)` would compute `xxx` multiple time. Once to select the possible candidates, and once to compute the parent set. The code used to compute parents is a direct copy past from the `parents` revset. We expect to replace it quickly in a later changeset. So we did not bother with extracting a function. In case where the input set is expensive to compute this provides a significant performance boost. (output are from contrib/revsetbenchmarks.py) revset: heads(matching(tip, "author")) plain min max first last reverse rev..rst rev..ast sort sor..rst sor..ast 0) 15.06746 14.92766 7.335694 15.03092 7.635580 15.04133 7.454806 15.27565 14.97796 14.87607 7.480900 1) 7.529300 49% 7.592152 50% 7.480548 7.544528 50% 7.421248 7.522279 50% 7.484876 7.613154 49% 7.599553 50% 7.561410 50% 7.508990 In other cases, with a faster input set, we still see a (smaller) performance boost. revset: heads(all()) plain min max first last reverse rev..rst rev..ast sort sor..rst sor..ast 0) 0.038994 0.035981 0.033345 0.035751 0.033569 0.039833 0.033653 0.035428 0.039483 0.035750 0.033657 1) 0.036359 93% 0.032613 90% 0.031479 94% 0.032790 91% 0.030681 91% 0.036456 91% 0.031128 92% 0.032461 91% 0.036276 91% 0.032721 91% 0.031024 92% revset: heads(-10000:-1) plain min max first last reverse rev..rst rev..ast sort sor..rst sor..ast 0) 0.004184 0.003576 0.003593 0.003628 0.003569 0.004277 0.003590 0.003719 0.004194 0.003659 0.003690 1) 0.003850 92% 0.003267 91% 0.003256 90% 0.003261 89% 0.003204 89% 0.003855 90% 0.003294 91% 0.003164 85% 0.003848 91% 0.003302 90% 0.003296 89% revset: (-5000:-1000) and heads(-10000:-1) plain min max first last reverse rev..rst rev..ast sort sor..rst sor..ast 0) 0.004730 0.003429 0.003359 0.003391 0.003369 0.004787 0.003418 0.003469 0.004772 0.003445 0.003454 1) 0.004277 90% 0.003430 0.003423 0.003353 0.003340 0.004250 88% 0.003387 0.003385 0.004325 90% 0.003413 0.003373 revset: heads(matching(tip, "author")) and -10000:-1 plain min max first last reverse rev..rst rev..ast sort sor..rst sor..ast 0) 8.250275 8.231453 7.508579 8.230028 7.529777 8.358590 7.531636 8.301830 8.137196 8.421402 7.540355 1) 7.474707 90% 7.587345 92% 7.486192 7.548340 91% 7.485288 7.659108 91% 7.485307 7.628890 91% 7.523479 92% 7.558384 89% 7.467524 revset: (-10000:-1) and heads(matching(tip, "author")) plain min max first last reverse rev..rst rev..ast sort sor..rst sor..ast 0) 8.341504 8.315248 7.489414 8.320746 7.548816 8.244137 7.514663 8.281701 8.218862 8.412644 7.456793 1) 7.553704 90% 7.570679 91% 7.391438 7.724237 92% 7.527400 7.570637 91% 7.580622 7.450912 89% 7.556154 91% 7.514726 89% 7.494328

revsetbenchmarks: add various examples around the 'heads()' revset We are about to work on the performance of this revset. Before doing so we defines various ways to use it.

revsetbenchmarks: support revset starting with a "-" Before this change, there was no strict separation between arguments and the benchmarked revset. This is easy to fix.

py3: two more passing tests from the ratchet Differential Revision: https://phab.mercurial-scm.org/D5627

py3: test*gendoc*.t passes on Python 3 The buildbot didn't notice because docutils isn't installed in Python 3 there yet. I verified this locally. Differential Revision: https://phab.mercurial-scm.org/D5617

bdiff: drop duplicate definition of splitnewlines() It was added in 29dd37a418aa (bdiff: write a native version of splitnewlines, 2018-01-25). Differential Revision: https://phab.mercurial-scm.org/D5618

tests: also skip remotefilelog *.py tests on Windows Otherwise, the buildbot won't even be green on stable with the RC. This should have gone with 0800d9e6e216. Previous discussion in this thread: https://www.mercurial-scm.org/pipermail/mercurial-devel/2018-November/125421.html

py3: add 10 more passing tests caught by ratchet Thanks to Augie who fixed these tests recently. Differential Revision: https://phab.mercurial-scm.org/D5616

remotefilelog: import Queue on Python 2, and queue on Python 3 Differential Revision: https://phab.mercurial-scm.org/D5599

py3: all fastannotate tests now pass Differential Revision: https://phab.mercurial-scm.org/D5615

fastannotate: adapt to buffer() going a way in Python 3 There's probably something more efficient I could do here, but I'm disinclined to spend much time on this at the moment. Differential Revision: https://phab.mercurial-scm.org/D5614

fastannotate: use pycompat.maplist instead of map Differential Revision: https://phab.mercurial-scm.org/D5613

fastannotate: slice strings to get single character Behaves identically on Python 3 and Python 2. Differential Revision: https://phab.mercurial-scm.org/D5612

fastannotate: fix isinstance checks to be against bytes instead of str Differential Revision: https://phab.mercurial-scm.org/D5611

absorb: add a pycompat.bytestr() to fix --edit-lines functionality on Python 3 Differential Revision: https://phab.mercurial-scm.org/D5610

remotefilelog: fix some bytes/str portability issues for Python 3 A few remotefilelog tests still fail on Python 3, but it's a much better story now. Differential Revision: https://phab.mercurial-scm.org/D5609

shallowutil: fsdecode the bytes group name before passing to os Differential Revision: https://phab.mercurial-scm.org/D5608

shallowutil: slice off a byte instead of subscripting This behaves identically on Python 2 and 3. Differential Revision: https://phab.mercurial-scm.org/D5607

remotefilelog: check against bytes type instead of buffer and coerce to bytes Fixes Python 3 compat here. Differential Revision: https://phab.mercurial-scm.org/D5606

remotefilelog: use list comprehension instead of filter for py3 portability Differential Revision: https://phab.mercurial-scm.org/D5605

tests: fix up uses of xrange in remotefilelog tests for py3 Differential Revision: https://phab.mercurial-scm.org/D5604

tests: add missing b prefixes in remotefilelog-getflogheads.py # skip-blame just b prefixes Differential Revision: https://phab.mercurial-scm.org/D5603

tests: make python oneliner portable to python 3 in remotefilelog test Differential Revision: https://phab.mercurial-scm.org/D5602

remotefilelog: implement __bool__ as well as __nonzero__ for py3 Differential Revision: https://phab.mercurial-scm.org/D5601

remotefilelog: fix logging in retry decorator This still fails with an error about no exception being available to re-raise, but so it goes. Differential Revision: https://phab.mercurial-scm.org/D5600

basepack: avoid 'rbe' mode in Python 3 Differential Revision: https://phab.mercurial-scm.org/D5598

remotefilelog: do file IO in terms of bytes Differential Revision: https://phab.mercurial-scm.org/D5597

rust-cpython: using MissingAncestors from Python code As precedently done with LazyAncestors on cpython.rs, we test for the presence of the 'rustext' module. incrementalmissingrevs() has two callers within the Mercurial core: `setdiscovery.partialdiscovery` and the `only()` revset. This move shows a significant discovery performance improvement in cases where the baseline is slow: using perfdiscovery on the PyPy repos, prepared with `contrib/discovery-helper <repo> 50 100`, we get averaged medians of 403ms with the Rust version vs 742ms without (about 45% better). But there are still indications that performance can be worse in cases the baseline is fast, possibly due to the conversion from Python to Rust and back becoming the bottleneck. We could measure this on mozilla-central in cases were the delta is just a few changesets. This requires confirmation, but if that's the reason, then an upcoming `partialdiscovery` fully in Rust should solve the problem. Differential Revision: https://phab.mercurial-scm.org/D5551

rust: MissingAncestors.basesheads() This new API method on `MissingAncestors` leverages directly the Rust implementation for relative heads of a set, and also lowers the cost of returning the results to Python in the context of discovery. These interchange costs can probably be further reduced by implementing the `partialdiscovery` class in Rust, but that will be investigated in the 5.0 development cycle. Differential Revision: https://phab.mercurial-scm.org/D5584

discovery: using the new basesheads() Our ultimate goal is to switch eventually to a Rust implementation, but this move actually seems to increase the performance in a pure Python build. What follows is a quick measurement done on PyPy on repos prepared with `contrib/discovery-helper.sh 50 100`. Before: ! wall 0.894384 comb 0.890000 user 0.890000 sys 0.000000 (best of 11) ! wall 0.971199 comb 0.970000 user 0.950000 sys 0.020000 (max of 11) ! wall 0.927993 comb 0.925455 user 0.919091 sys 0.006364 (avg of 11) ! wall 0.921619 comb 0.920000 user 0.910000 sys 0.010000 (median of 11) After: ! wall 0.614278 comb 0.610000 user 0.610000 sys 0.000000 (best of 14) ! wall 0.789459 comb 0.790000 user 0.770000 sys 0.020000 (max of 14) ! wall 0.722765 comb 0.720000 user 0.715714 sys 0.004286 (avg of 14) ! wall 0.734448 comb 0.720000 user 0.720000 sys 0.000000 (median of 14) Differential Revision: https://phab.mercurial-scm.org/D5583

ancestor: incrementalmissingancestors.basesheads() This new method will avoid the need to access the `bases` attribute directly in `setdiscovery`, and to prefilter `nullrev` before passing it to the `heads()` revset. Being a method, it can transparently be reimplemented in a Rust (or any native) version. Differential Revision: https://phab.mercurial-scm.org/D5582

rust-cpython: set conversion for MissingAncestors.bases() Also I hope that the separate `py_set()` helper will help transition to proper `PySet` support in `rust-cpython` Took the opportunity to replace explict for loop with iteration and collect(). Differential Revision: https://phab.mercurial-scm.org/D5581

rust: dagop.headrevs() Rust counterparts This introduces two Rust implementations for `mercurial.dagop.headrevs`. The algorithm is identical to the Python version. Depending on the caller, one or the other could be the most practical, or the most performant, by minimizing the amount of memory copy and allocations. Differential Revision: https://phab.mercurial-scm.org/D5580

rust: factorized testing Graphs it will useful to use these outside of `ancestors`, too. Differential Revision: https://phab.mercurial-scm.org/D5579

rust-cpython: moved generic conversion fn out of ancestors module This will allow to use it easily from other submodules Differential Revision: https://phab.mercurial-scm.org/D5578

revset: transparently forward _intlist argument in all case We took a safe approach for the first take, we can get bolder now.

narrow: reuse narrowspec.updateworkingcopy() when narrowing Similar to the previous patch for widening, but here we also need to teach updateworkingcopy() to forcefully delete files that are not recorded in the dirstate as clean. That should be safe because the narrowing command (e.g. `hg tracked --removeinclude`) has already checked that the working copy is clean. Differential Revision: https://phab.mercurial-scm.org/D5511

narrow: reuse narrowspec.updateworkingcopy() when widening The widening of the working copy we do after widening a repo is practically the same as we do in a repo share after the store narrowspec has been changed in a different share. Let's reuse the code for that that we now have in the narrowspec module. Differential Revision: https://phab.mercurial-scm.org/D5510

narrow: move copytonarrowspec() out of setnarrowpats() I think it was a mistake to write the working copy's narrowspec every time the store narrowspec is written. This starts separating those actions. Differential Revision: https://phab.mercurial-scm.org/D5509

narrow: drop now-unnecessary reassignment of repo attributes Differential Revision: https://phab.mercurial-scm.org/D5507

packaging: allow running packaging with custom uid+gid for CentOS rpmbuild in CentOS 7 has a bug causing rpmbuild to fail with "Bad owner/group" if spec or source files are owned by a different user: https://github.com/rpm-software-management/rpm/issues/2 This makes it very annoying to try and build the CentOS RPMs on CentOS with Docker. As an alternative, this change makes it possible to do so, using an environment variable. Differential Revision: https://phab.mercurial-scm.org/D5571

hg-docker: fix Python 3.4 compatibility (for CentOS 7) I realize Mercurial is not targetting Python 3.4 compatibility, but without this change, it's not even possible to build it on CentOS 7 (and I assume the same is true for RHEL 7). Differential Revision: https://phab.mercurial-scm.org/D5570

copies: use node.nullrev instead of literal -1 Differential Revision: https://phab.mercurial-scm.org/D5593

copies: use node.wdirrev instead of inventing another constant for it Differential Revision: https://phab.mercurial-scm.org/D5592

narrow: extract repo property for store narrowmatcher When a repo lock is released, we try to persist the manifest cache. That involves getting the narrowmatcher for the manifestlog. That should not fail if the store and working copy narrowspecs are out of sync. Without this patch, the later patches in this series will fail because of that. Differential Revision: https://phab.mercurial-scm.org/D5508

narrow: copy store narrowspec to working copy immediately We no longer need to delay it until the end of the transaction since we now restore a backup if the transaction aborts. Differential Revision: https://phab.mercurial-scm.org/D5506

narrow: include working copy narrowspec in transaction journal Now that we have separate narrowspecs for the store and the working copy, we need to include both in the transaction journal. Differential Revision: https://phab.mercurial-scm.org/D5505

narrow: make dirstateguard back up and restore working copy narrowspec instead We used to have only one narrowspec for the store and the working copy, but now that we have one narrowspec for each, it seems clear that the dirstateguard was supposed to back up and restore the narrowspec associated with the working copy, not the one associated with the store. clearbackup() (for the store narrowspec) is not needed because the presence of the file in localrepository._journalfiles() takes care of that. Differential Revision: https://phab.mercurial-scm.org/D5504

narrow: include journal.narrowspec in transaction journal We had missed this file before, which led to it lying around after the transaction completed. Differential Revision: https://phab.mercurial-scm.org/D5556

progress: deprecate ui.progress() It is now just a weird wrapper for ui.makeprogress(). Differential Revision: https://phab.mercurial-scm.org/D5531

context: use scmutil.matchfiles instead of matchmod.match(exact=True) Differential Revision: https://phab.mercurial-scm.org/D5591

histedit: fix call to _getgoal() by adding a byteskwargs() wrapper I also added some b-prefixes while I was here because I got confused and it seems silly to not just add them since it clarifies the whole change. Differential Revision: https://phab.mercurial-scm.org/D5585

revset: introduce an API that avoids `formatspec` input serialization Instead of having the data fully serialized, the input can be directly inserted in the tree at a later stage. Just using it for simple "%ld" case provide a significant boost. For example here are the impact on a sample discovery run between two pypy repositories with arbitrary differences (using hg perfdiscovery). $ hg perfdiscovery before: ! wall 0.700435 comb 0.710000 user 0.700000 sys 0.010000 (median of 15) after: ! wall 0.501305 comb 0.510000 user 0.490000 sys 0.020000 (median of 20)

revset: detect integer list on parsing Right now, using "%ld" with `repo.revs("…%ld…", somerevs)` is very inefficient, all items in `somerevs` will be serialized to ascii and then reparsed as integers. If `somerevs` contains just an handful of entry this is fine, however, when you get to thousands or hundreds of thousands of revisions this becomes very slow. To avoid this serialization we need to first detect this situation. The code involved in the whole process is quite complex so we start simple and focus on some "simple" but widespread cases. So far we only detect the situation and don't do anything special about it. The singled out will be serialized in `formatspec` in the same way as before.

revert: extract "%ld" formatting in a _formatintlist function We'll have to reuse this logic in different places.

revset: extract parsing logic out of formatspec We want to be able to perform better handling of some input when running revset (eg: `repo.revs("%ld", somerevs)`). The first step is to be able to access some of the parsed content before it gets substituted. There are many possible different substitutions, we'll add support for them gradually. In this changeset we support none, we just split some logic in a sub function as a preparatory step.

revset: enforce "%d" to be interpreted as literal revision number (API) (BC) Before this change, `formatspec("%d", x)` results in `"%d" % int(x)`. This seems simple and correct until you consider `nullrev`. In revset, a direct "-1" symbol is equivalent to `tip` not `nullrev`. This is a subtle error that went undetected for a while. Wrapping the revision number inside 'rev()' remove the ambiguity, preserving nullrev value passed to formatspec. It got caught by the rebase code, were the following wrongly returned `[1]`: repo.revs("children(%d) and ancestors(%ld)", 0, [nullrev]) This is flagged as API, because `%d` can be used for non-revision integer argument of revset function. We probably need to introduce a new '%…' substitution to allow literal integer (maybe `%i`). However, the `%d` usage is currently widespread for revision number so it is important to fix this issue for `%d`. This choice is reinforced by the fact _intlist is implemented as revisions only. Restricting `%d` to revision only makes things more consistent. This bug can become especially tricky since `_intlist` recognize `nullrev` right. So `revs('%ld', [-1, 0])` → select `[nullrev, 0]` but `revs('%ld', [-1])` is simplified and treated as `%d` selecting `[tip]`. Another side effect is that "%d" of an unknown revision simply match nothing. It was previously raising and error. This is consistent with what "%ld" (and `_intlist`) is doing, so it seems like a good move.

revset: remove the last usage of "%d" for a non-revision entry In order to fix an important bug, we are about to narrow the semantic of "%d" in revset. This is one of the few exceptions that we need to get rid of before being able to fix the bug. See the later semantic narrowing changeset for full rationale on the semantic change.