tests: suppress "Checked out 1 paths of <hash>" from modern git test-convert-git.t is failiing since git commit 0f086e6dca (checkout: print something when checking out paths, 2018-11-13). Suppress the new output by disambiguating the arguments with a "--" separator. Differential Revision: https://phab.mercurial-scm.org/D5625

revlog: document that mmap resources are released implicitly by GC It's okay-ish, but currently the open fd and the mapping itself are leaked until the indexdata is deallocated. If revlog had close(), the underlying resources should be closed there as well, but AFAIK there's no such hook point.

ui: proxy protect/restorestdio() calls to update internal flag It should be better to manage the redirection flag solely by the ui class.

ui: move protectedstdio() context manager from procutil This is a follow-up series for 23a00bc90a3c, "chgserver: do not send system() back to client if stdio redirected." The function is renamed using ui terms.

cext: clang-format new code coming from stable branch

match: support rooted globs in hgignore In a .hgignore, "glob:foo" always means "**/foo". This cannot be avoided because there is no syntax like "^" in regexes to say you don't want the implied "**/" (of course one can use regexes, but glob syntax is nice). When you have a long list of fairly specific globs like path/to/some/thing, this has two consequences: 1. unintended files may be ignored (not too common though) 2. matching performance can suffer significantly Here is vanilla hg status timing on a private repository: Using syntax:glob everywhere real 0m2.199s user 0m1.545s sys 0m0.619s When rooting the appropriate globs real 0m1.434s user 0m0.847s sys 0m0.565s (tangentially, none of this shows up in --profile's output. It seems that C code doesn't play well with profiling) The code already supports this but there is no syntax to make use of it, so it seems reasonable to create such syntax. I create a new hgignore syntax "rootglob". Differential Revision: https://phab.mercurial-scm.org/D5493

resolve: fix mark-check when a file was deleted on one side (issue6020) wvfs.open raises an error if one of the files does not exist. Ignoring the error if it's ENOENT is done in several other places in this code, so I'm in good company :) Differential Revision: https://phab.mercurial-scm.org/D5243

discovery: compute newly discovered missing in a more efficient way Calling "descendants" is expensive, instead, we bound the walk inside the know set of undecided revision. This help with discovery performance: # without the revset '%ld' improvement $ hg perfdiscovery -R pypy-left pypy-right before: wall 0.675631 comb 0.680000 user 0.670000 sys 0.010000 (median of 15) after: wall 0.520145 comb 0.530000 user 0.510000 sys 0.020000 (median of 19) There is another series in flight that greatly improves performances of "%ld" substitution in `repo.revs` call. If this changeset is applied above it, we see a similar performance boost. # with the revset '%ld' improvement $ hg perfdiscovery -R pypy-left pypy-right before: wall 0.477848 comb 0.480000 user 0.480000 sys 0.000000 (median of 22) after: wall 0.404163 comb 0.400000 user 0.400000 sys 0.000000 (median of 24)

exthelper: drop the addattr() decorator Yuya pointed out that this goes against the typical advice to not add attributes to classes[1]. The evolve extension still uses this a handful of times, so maybe it should be brought back in the future if a general use is found. But it isn't nice to have a new helper API that can lead to easy problems. [1] https://www.mercurial-scm.org/pipermail/mercurial-devel/2018-December/126330.html

revsetbenchmark: add more example for roots usages We test the `roots` revset in setting similar to our test for `heads`. Note that the algorithm used for roots can give result without consuming the full input set. This provides a significant speedup when testing or accessing a single value. We can't just replace it with simple, full algorithm like we did for `heads`. See performance number below: 0) roots((tip~100::) - (tip~100::tip)) 1) roots((0::) - (0::tip)) 2) roots(tip~100:) 3) roots(:42) 4) roots(not public()) 5) roots((0:tip)::) 6) roots(0::tip) 7) 42:68 and roots(42:tip) 8) roots(0:tip) 9) roots((:42) + (tip~42:)) 10) roots(all()) 11) roots(-10000:-1) 12) (-5000:-1000) and roots(-10000:-1) 13) roots(matching(tip, "author")) 14) roots(matching(tip, "author")) and -10000:-1 15) (-10000:-1) and roots(matching(tip, "author")) plain min max first last reverse rev..rst rev..ast sort sor..rst sor..ast 00) 0.000789 0.000801 0.000801 0.000819 0.000784 0.000774 0.000793 0.000816 0.000815 0.000831 0.000799 01) 0.097610 0.002717 0.096706 0.002615 0.059189 0.089033 0.059862 0.002644 0.098058 0.002640 0.058992 02) 0.000709 0.000117 0.000382 0.000136 0.000384 0.000724 0.000412 0.000133 0.000733 0.000159 0.000416 03) 0.000075 0.000064 0.000093 0.000080 0.000097 0.000089 0.000123 0.000079 0.000105 0.000102 0.000126 04) 0.000055 0.000071 0.000070 0.000087 0.000075 0.000066 0.000100 0.000085 0.000082 0.000110 0.000102 05) 0.088043 0.001084 0.087816 0.001097 0.048049 0.072454 0.047673 0.001089 0.088491 0.001163 0.047824 06) 0.058761 0.001727 0.059324 0.001850 0.058562 0.059198 0.058998 0.001743 0.058556 0.001874 0.059420 07) 0.000131 0.000121 0.000145 0.000138 0.000150 0.000142 0.000178 0.000135 0.000160 0.000163 0.000179 08) 0.058003 0.000077 0.032327 0.000093 0.031966 0.056812 0.031753 0.000092 0.057113 0.000116 0.031933 09) 0.000503 0.000145 0.000469 0.000161 0.000476 0.000564 0.000502 0.000160 0.000537 0.000187 0.000500 10) 0.056654 0.000058 0.033104 0.000073 0.032157 0.056598 0.031877 0.000071 0.056433 0.000094 0.031819 11) 0.005842 0.000081 0.001907 0.000101 0.001883 0.005868 0.001915 0.000099 0.005836 0.000122 0.001896 12) 0.003237 0.000634 0.001784 0.000655 0.001803 0.003245 0.001837 0.000649 0.003231 0.000680 0.001858

dagop: minor python optimization to `headrevs` Less lookup and less function call never hurt. This provides a small speedup on various run of the 'heads()' revset. This also buys back some of the slow down we observed in the previous changesets for single value lookup. Performance number: 0) before dagop.headrevs usage 1) after dagop.headrevs usage 2) after this change revset: heads(all()) plain min max first last reverse rev..rst rev..ast sort sor..rst sor..ast 0) 0.036503 0.032564 0.030024 0.032378 0.030887 0.036367 0.031713 0.032205 0.036467 0.032286 0.030300 1) 0.036668 0.035347 108% 0.035611 118% 0.035358 109% 0.035726 115% 0.036411 0.035261 111% 0.036096 112% 0.036052 0.035095 108% 0.035792 118% 2) 0.034254 93% 0.034482 0.035003 0.034353 0.033754 94% 0.034689 0.034361 0.035059 0.034636 0.034662 0.035465 revset: heads(-10000:-1) plain min max first last reverse rev..rst rev..ast sort sor..rst sor..ast 0) 0.003936 0.003218 0.003227 0.003302 0.003328 0.003848 0.003305 0.003252 0.003839 0.003306 0.003279 1) 0.003870 0.003785 117% 0.003821 118% 0.003780 114% 0.003769 113% 0.003776 0.003792 114% 0.003805 117% 0.003810 0.003798 114% 0.003840 117% 2) 0.003666 94% 0.003577 94% 0.003632 0.003644 0.003614 0.003638 0.003652 0.003632 0.003661 0.003660 0.003658 revset: (-5000:-1000) and heads(-10000:-1) plain min max first last reverse rev..rst rev..ast sort sor..rst sor..ast 0) 0.004244 0.003368 0.003313 0.003367 0.003327 0.004325 0.003401 0.003379 0.004310 0.003359 0.003396 1) 0.003969 93% 0.003862 114% 0.003834 115% 0.003810 113% 0.003822 114% 0.003940 91% 0.003908 114% 0.003814 112% 0.003986 92% 0.003954 117% 0.003816 112% 2) 0.003728 93% 0.003638 94% 0.003659 0.003685 0.003628 94% 0.003716 94% 0.003653 93% 0.003655 0.003748 94% 0.003740 94% 0.003686 revset: heads(matching(tip, "author")) plain min max first last reverse rev..rst rev..ast sort sor..rst sor..ast 0) 7.574666 7.545950 7.570743 7.578697 7.525725 7.509929 7.443854 7.488442 7.452880 7.445411 7.689107 1) 7.549390 7.389162 7.529790 7.536297 7.450467 7.555347 7.404586 7.514948 7.542794 7.524787 7.536918 2) 7.568294 7.479326 7.578624 7.380375 7.440102 7.454218 7.515189 7.556511 7.524585 7.537566 7.507418 revset: heads(matching(tip, "author")) and -10000:-1 plain min max first last reverse rev..rst rev..ast sort sor..rst sor..ast 0) 7.512533 7.605877 7.382894 7.462109 7.420086 7.575034 7.448452 7.549374 7.457880 7.450308 7.515019 1) 7.548677 7.551832 7.629598 7.494857 7.550554 7.521838 7.451794 error 7.321781 7.546885 7.557523 2) 7.451985 7.541044 7.506563 7.470928 7.512618 7.474988 7.498887 7.547930 7.560276 7.618599 7.465442 revset: (-10000:-1) and heads(matching(tip, "author")) plain min max first last reverse rev..rst rev..ast sort sor..rst sor..ast 0) 7.465419 7.570089 7.439594 7.521221 7.498716 7.492922 7.479108 7.552397 7.407888 error 7.468264 1) 7.539866 7.548045 7.491761 7.517170 7.469824 7.501990 7.579102 7.502568 7.578102 7.555754 7.567622 2) 7.370463 7.514712 7.497024 7.679428 7.638138 7.490775 7.472273 7.652587 7.584139 7.511893 7.466384

revset: use changelog's `headrevs` method to compute heads Instead of implementing our own algorithm, we reuse a more generic one. This previous algorithm did not leave much room for laziness so we do not really regress in that regards. A small impact is visible for first/last value in some of the simpler cases. The time needed to compute all values improves overall. Small optimization in the dagop.headrevs function will help to buy this back in the next changesets. There is room to introduce actual laziness in this algorithm, but this is out of scope for this series. This has no visible effect on expensive cases: revset: heads(matching(tip, "author")) plain min max first last reverse rev..rst rev..ast sort sor..rst sor..ast 0) 7.574666 7.545950 7.570743 7.578697 7.525725 7.509929 7.443854 7.488442 7.452880 7.445411 7.689107 1) 7.549390 7.389162 7.529790 7.536297 7.450467 7.555347 7.404586 7.514948 7.542794 7.524787 7.536918 revset: heads(matching(tip, "author")) and -10000:-1 plain min max first last reverse rev..rst rev..ast sort sor..rst sor..ast 0) 7.512533 7.605877 7.382894 7.462109 7.420086 7.575034 7.448452 7.549374 7.457880 7.450308 7.515019 1) 7.548677 7.551832 7.629598 7.494857 7.550554 7.521838 7.451794 error 7.321781 7.546885 7.557523 revset: (-10000:-1) and heads(matching(tip, "author")) plain min max first last reverse rev..rst rev..ast sort sor..rst sor..ast 0) 7.465419 7.570089 7.439594 7.521221 7.498716 7.492922 7.479108 7.552397 7.407888 error 7.468264 1) 7.539866 7.548045 7.491761 7.517170 7.469824 7.501990 7.579102 7.502568 7.578102 7.555754 7.567622 In simpler cases, we see a 10-15% impact when retrieving a single value, the full computation time is equivalent or improved: revset: (-5000:-1000) and heads(-10000:-1) plain min max first last reverse rev..rst rev..ast sort sor..rst sor..ast 0) 0.004244 0.003368 0.003313 0.003367 0.003327 0.004325 0.003401 0.003379 0.004310 0.003359 0.003396 1) 0.003969 93% 0.003862 114% 0.003834 115% 0.003810 113% 0.003822 114% 0.003940 91% 0.003908 114% 0.003814 112% 0.003986 92% 0.003954 117% 0.003816 112% revset: heads(all()) plain min max first last reverse rev..rst rev..ast sort sor..rst sor..ast 0) 0.036503 0.032564 0.030024 0.032378 0.030887 0.036367 0.031713 0.032205 0.036467 0.032286 0.030300 1) 0.036668 0.035347 108% 0.035611 118% 0.035358 109% 0.035726 115% 0.036411 0.035261 111% 0.036096 112% 0.036052 0.035095 108% 0.035792 118% revset: heads(-10000:-1) plain min max first last reverse rev..rst rev..ast sort sor..rst sor..ast 0) 0.003936 0.003218 0.003227 0.003302 0.003328 0.003848 0.003305 0.003252 0.003839 0.003306 0.003279 1) 0.003870 0.003785 117% 0.003821 118% 0.003780 114% 0.003769 113% 0.003776 0.003792 114% 0.003805 117% 0.003810 0.003798 114% 0.003840 117%

revlog: accept a revs argument in `headrevs` Computing the heads of an arbitrary set of revision is useful, we make it possible to do so through the `headrevs` method of the revlog. Right now, this is just calling dagop's implementation. However, we expect to plug a native implementation soon.

revset: inline parents computation to reuse the input argument Before this change, using `heads(xxx)` would compute `xxx` multiple time. Once to select the possible candidates, and once to compute the parent set. The code used to compute parents is a direct copy past from the `parents` revset. We expect to replace it quickly in a later changeset. So we did not bother with extracting a function. In case where the input set is expensive to compute this provides a significant performance boost. (output are from contrib/revsetbenchmarks.py) revset: heads(matching(tip, "author")) plain min max first last reverse rev..rst rev..ast sort sor..rst sor..ast 0) 15.06746 14.92766 7.335694 15.03092 7.635580 15.04133 7.454806 15.27565 14.97796 14.87607 7.480900 1) 7.529300 49% 7.592152 50% 7.480548 7.544528 50% 7.421248 7.522279 50% 7.484876 7.613154 49% 7.599553 50% 7.561410 50% 7.508990 In other cases, with a faster input set, we still see a (smaller) performance boost. revset: heads(all()) plain min max first last reverse rev..rst rev..ast sort sor..rst sor..ast 0) 0.038994 0.035981 0.033345 0.035751 0.033569 0.039833 0.033653 0.035428 0.039483 0.035750 0.033657 1) 0.036359 93% 0.032613 90% 0.031479 94% 0.032790 91% 0.030681 91% 0.036456 91% 0.031128 92% 0.032461 91% 0.036276 91% 0.032721 91% 0.031024 92% revset: heads(-10000:-1) plain min max first last reverse rev..rst rev..ast sort sor..rst sor..ast 0) 0.004184 0.003576 0.003593 0.003628 0.003569 0.004277 0.003590 0.003719 0.004194 0.003659 0.003690 1) 0.003850 92% 0.003267 91% 0.003256 90% 0.003261 89% 0.003204 89% 0.003855 90% 0.003294 91% 0.003164 85% 0.003848 91% 0.003302 90% 0.003296 89% revset: (-5000:-1000) and heads(-10000:-1) plain min max first last reverse rev..rst rev..ast sort sor..rst sor..ast 0) 0.004730 0.003429 0.003359 0.003391 0.003369 0.004787 0.003418 0.003469 0.004772 0.003445 0.003454 1) 0.004277 90% 0.003430 0.003423 0.003353 0.003340 0.004250 88% 0.003387 0.003385 0.004325 90% 0.003413 0.003373 revset: heads(matching(tip, "author")) and -10000:-1 plain min max first last reverse rev..rst rev..ast sort sor..rst sor..ast 0) 8.250275 8.231453 7.508579 8.230028 7.529777 8.358590 7.531636 8.301830 8.137196 8.421402 7.540355 1) 7.474707 90% 7.587345 92% 7.486192 7.548340 91% 7.485288 7.659108 91% 7.485307 7.628890 91% 7.523479 92% 7.558384 89% 7.467524 revset: (-10000:-1) and heads(matching(tip, "author")) plain min max first last reverse rev..rst rev..ast sort sor..rst sor..ast 0) 8.341504 8.315248 7.489414 8.320746 7.548816 8.244137 7.514663 8.281701 8.218862 8.412644 7.456793 1) 7.553704 90% 7.570679 91% 7.391438 7.724237 92% 7.527400 7.570637 91% 7.580622 7.450912 89% 7.556154 91% 7.514726 89% 7.494328

revsetbenchmarks: add various examples around the 'heads()' revset We are about to work on the performance of this revset. Before doing so we defines various ways to use it.

revsetbenchmarks: support revset starting with a "-" Before this change, there was no strict separation between arguments and the benchmarked revset. This is easy to fix.

py3: two more passing tests from the ratchet Differential Revision: https://phab.mercurial-scm.org/D5627

py3: test*gendoc*.t passes on Python 3 The buildbot didn't notice because docutils isn't installed in Python 3 there yet. I verified this locally. Differential Revision: https://phab.mercurial-scm.org/D5617

bdiff: drop duplicate definition of splitnewlines() It was added in 29dd37a418aa (bdiff: write a native version of splitnewlines, 2018-01-25). Differential Revision: https://phab.mercurial-scm.org/D5618

tests: also skip remotefilelog *.py tests on Windows Otherwise, the buildbot won't even be green on stable with the RC. This should have gone with 0800d9e6e216. Previous discussion in this thread: https://www.mercurial-scm.org/pipermail/mercurial-devel/2018-November/125421.html

py3: add 10 more passing tests caught by ratchet Thanks to Augie who fixed these tests recently. Differential Revision: https://phab.mercurial-scm.org/D5616

remotefilelog: import Queue on Python 2, and queue on Python 3 Differential Revision: https://phab.mercurial-scm.org/D5599

py3: all fastannotate tests now pass Differential Revision: https://phab.mercurial-scm.org/D5615

fastannotate: adapt to buffer() going a way in Python 3 There's probably something more efficient I could do here, but I'm disinclined to spend much time on this at the moment. Differential Revision: https://phab.mercurial-scm.org/D5614

fastannotate: use pycompat.maplist instead of map Differential Revision: https://phab.mercurial-scm.org/D5613

fastannotate: slice strings to get single character Behaves identically on Python 3 and Python 2. Differential Revision: https://phab.mercurial-scm.org/D5612

fastannotate: fix isinstance checks to be against bytes instead of str Differential Revision: https://phab.mercurial-scm.org/D5611

absorb: add a pycompat.bytestr() to fix --edit-lines functionality on Python 3 Differential Revision: https://phab.mercurial-scm.org/D5610

remotefilelog: fix some bytes/str portability issues for Python 3 A few remotefilelog tests still fail on Python 3, but it's a much better story now. Differential Revision: https://phab.mercurial-scm.org/D5609

shallowutil: fsdecode the bytes group name before passing to os Differential Revision: https://phab.mercurial-scm.org/D5608

shallowutil: slice off a byte instead of subscripting This behaves identically on Python 2 and 3. Differential Revision: https://phab.mercurial-scm.org/D5607

remotefilelog: check against bytes type instead of buffer and coerce to bytes Fixes Python 3 compat here. Differential Revision: https://phab.mercurial-scm.org/D5606

remotefilelog: use list comprehension instead of filter for py3 portability Differential Revision: https://phab.mercurial-scm.org/D5605

tests: fix up uses of xrange in remotefilelog tests for py3 Differential Revision: https://phab.mercurial-scm.org/D5604

tests: add missing b prefixes in remotefilelog-getflogheads.py # skip-blame just b prefixes Differential Revision: https://phab.mercurial-scm.org/D5603

tests: make python oneliner portable to python 3 in remotefilelog test Differential Revision: https://phab.mercurial-scm.org/D5602

remotefilelog: implement __bool__ as well as __nonzero__ for py3 Differential Revision: https://phab.mercurial-scm.org/D5601

remotefilelog: fix logging in retry decorator This still fails with an error about no exception being available to re-raise, but so it goes. Differential Revision: https://phab.mercurial-scm.org/D5600

basepack: avoid 'rbe' mode in Python 3 Differential Revision: https://phab.mercurial-scm.org/D5598

remotefilelog: do file IO in terms of bytes Differential Revision: https://phab.mercurial-scm.org/D5597

rust-cpython: using MissingAncestors from Python code As precedently done with LazyAncestors on cpython.rs, we test for the presence of the 'rustext' module. incrementalmissingrevs() has two callers within the Mercurial core: `setdiscovery.partialdiscovery` and the `only()` revset. This move shows a significant discovery performance improvement in cases where the baseline is slow: using perfdiscovery on the PyPy repos, prepared with `contrib/discovery-helper <repo> 50 100`, we get averaged medians of 403ms with the Rust version vs 742ms without (about 45% better). But there are still indications that performance can be worse in cases the baseline is fast, possibly due to the conversion from Python to Rust and back becoming the bottleneck. We could measure this on mozilla-central in cases were the delta is just a few changesets. This requires confirmation, but if that's the reason, then an upcoming `partialdiscovery` fully in Rust should solve the problem. Differential Revision: https://phab.mercurial-scm.org/D5551

rust: MissingAncestors.basesheads() This new API method on `MissingAncestors` leverages directly the Rust implementation for relative heads of a set, and also lowers the cost of returning the results to Python in the context of discovery. These interchange costs can probably be further reduced by implementing the `partialdiscovery` class in Rust, but that will be investigated in the 5.0 development cycle. Differential Revision: https://phab.mercurial-scm.org/D5584

discovery: using the new basesheads() Our ultimate goal is to switch eventually to a Rust implementation, but this move actually seems to increase the performance in a pure Python build. What follows is a quick measurement done on PyPy on repos prepared with `contrib/discovery-helper.sh 50 100`. Before: ! wall 0.894384 comb 0.890000 user 0.890000 sys 0.000000 (best of 11) ! wall 0.971199 comb 0.970000 user 0.950000 sys 0.020000 (max of 11) ! wall 0.927993 comb 0.925455 user 0.919091 sys 0.006364 (avg of 11) ! wall 0.921619 comb 0.920000 user 0.910000 sys 0.010000 (median of 11) After: ! wall 0.614278 comb 0.610000 user 0.610000 sys 0.000000 (best of 14) ! wall 0.789459 comb 0.790000 user 0.770000 sys 0.020000 (max of 14) ! wall 0.722765 comb 0.720000 user 0.715714 sys 0.004286 (avg of 14) ! wall 0.734448 comb 0.720000 user 0.720000 sys 0.000000 (median of 14) Differential Revision: https://phab.mercurial-scm.org/D5583

ancestor: incrementalmissingancestors.basesheads() This new method will avoid the need to access the `bases` attribute directly in `setdiscovery`, and to prefilter `nullrev` before passing it to the `heads()` revset. Being a method, it can transparently be reimplemented in a Rust (or any native) version. Differential Revision: https://phab.mercurial-scm.org/D5582

rust-cpython: set conversion for MissingAncestors.bases() Also I hope that the separate `py_set()` helper will help transition to proper `PySet` support in `rust-cpython` Took the opportunity to replace explict for loop with iteration and collect(). Differential Revision: https://phab.mercurial-scm.org/D5581

rust: dagop.headrevs() Rust counterparts This introduces two Rust implementations for `mercurial.dagop.headrevs`. The algorithm is identical to the Python version. Depending on the caller, one or the other could be the most practical, or the most performant, by minimizing the amount of memory copy and allocations. Differential Revision: https://phab.mercurial-scm.org/D5580

rust: factorized testing Graphs it will useful to use these outside of `ancestors`, too. Differential Revision: https://phab.mercurial-scm.org/D5579

rust-cpython: moved generic conversion fn out of ancestors module This will allow to use it easily from other submodules Differential Revision: https://phab.mercurial-scm.org/D5578