tests: fix up uses of xrange in remotefilelog tests for py3 Differential Revision: https://phab.mercurial-scm.org/D5604

tests: add missing b prefixes in remotefilelog-getflogheads.py # skip-blame just b prefixes Differential Revision: https://phab.mercurial-scm.org/D5603

tests: make python oneliner portable to python 3 in remotefilelog test Differential Revision: https://phab.mercurial-scm.org/D5602

remotefilelog: implement __bool__ as well as __nonzero__ for py3 Differential Revision: https://phab.mercurial-scm.org/D5601

remotefilelog: fix logging in retry decorator This still fails with an error about no exception being available to re-raise, but so it goes. Differential Revision: https://phab.mercurial-scm.org/D5600

basepack: avoid 'rbe' mode in Python 3 Differential Revision: https://phab.mercurial-scm.org/D5598

remotefilelog: do file IO in terms of bytes Differential Revision: https://phab.mercurial-scm.org/D5597

rust-cpython: using MissingAncestors from Python code As precedently done with LazyAncestors on cpython.rs, we test for the presence of the 'rustext' module. incrementalmissingrevs() has two callers within the Mercurial core: `setdiscovery.partialdiscovery` and the `only()` revset. This move shows a significant discovery performance improvement in cases where the baseline is slow: using perfdiscovery on the PyPy repos, prepared with `contrib/discovery-helper <repo> 50 100`, we get averaged medians of 403ms with the Rust version vs 742ms without (about 45% better). But there are still indications that performance can be worse in cases the baseline is fast, possibly due to the conversion from Python to Rust and back becoming the bottleneck. We could measure this on mozilla-central in cases were the delta is just a few changesets. This requires confirmation, but if that's the reason, then an upcoming `partialdiscovery` fully in Rust should solve the problem. Differential Revision: https://phab.mercurial-scm.org/D5551

rust: MissingAncestors.basesheads() This new API method on `MissingAncestors` leverages directly the Rust implementation for relative heads of a set, and also lowers the cost of returning the results to Python in the context of discovery. These interchange costs can probably be further reduced by implementing the `partialdiscovery` class in Rust, but that will be investigated in the 5.0 development cycle. Differential Revision: https://phab.mercurial-scm.org/D5584

discovery: using the new basesheads() Our ultimate goal is to switch eventually to a Rust implementation, but this move actually seems to increase the performance in a pure Python build. What follows is a quick measurement done on PyPy on repos prepared with `contrib/discovery-helper.sh 50 100`. Before: ! wall 0.894384 comb 0.890000 user 0.890000 sys 0.000000 (best of 11) ! wall 0.971199 comb 0.970000 user 0.950000 sys 0.020000 (max of 11) ! wall 0.927993 comb 0.925455 user 0.919091 sys 0.006364 (avg of 11) ! wall 0.921619 comb 0.920000 user 0.910000 sys 0.010000 (median of 11) After: ! wall 0.614278 comb 0.610000 user 0.610000 sys 0.000000 (best of 14) ! wall 0.789459 comb 0.790000 user 0.770000 sys 0.020000 (max of 14) ! wall 0.722765 comb 0.720000 user 0.715714 sys 0.004286 (avg of 14) ! wall 0.734448 comb 0.720000 user 0.720000 sys 0.000000 (median of 14) Differential Revision: https://phab.mercurial-scm.org/D5583

ancestor: incrementalmissingancestors.basesheads() This new method will avoid the need to access the `bases` attribute directly in `setdiscovery`, and to prefilter `nullrev` before passing it to the `heads()` revset. Being a method, it can transparently be reimplemented in a Rust (or any native) version. Differential Revision: https://phab.mercurial-scm.org/D5582

rust-cpython: set conversion for MissingAncestors.bases() Also I hope that the separate `py_set()` helper will help transition to proper `PySet` support in `rust-cpython` Took the opportunity to replace explict for loop with iteration and collect(). Differential Revision: https://phab.mercurial-scm.org/D5581

rust: dagop.headrevs() Rust counterparts This introduces two Rust implementations for `mercurial.dagop.headrevs`. The algorithm is identical to the Python version. Depending on the caller, one or the other could be the most practical, or the most performant, by minimizing the amount of memory copy and allocations. Differential Revision: https://phab.mercurial-scm.org/D5580

rust: factorized testing Graphs it will useful to use these outside of `ancestors`, too. Differential Revision: https://phab.mercurial-scm.org/D5579

rust-cpython: moved generic conversion fn out of ancestors module This will allow to use it easily from other submodules Differential Revision: https://phab.mercurial-scm.org/D5578

revset: transparently forward _intlist argument in all case We took a safe approach for the first take, we can get bolder now.

narrow: reuse narrowspec.updateworkingcopy() when narrowing Similar to the previous patch for widening, but here we also need to teach updateworkingcopy() to forcefully delete files that are not recorded in the dirstate as clean. That should be safe because the narrowing command (e.g. `hg tracked --removeinclude`) has already checked that the working copy is clean. Differential Revision: https://phab.mercurial-scm.org/D5511

narrow: reuse narrowspec.updateworkingcopy() when widening The widening of the working copy we do after widening a repo is practically the same as we do in a repo share after the store narrowspec has been changed in a different share. Let's reuse the code for that that we now have in the narrowspec module. Differential Revision: https://phab.mercurial-scm.org/D5510

narrow: move copytonarrowspec() out of setnarrowpats() I think it was a mistake to write the working copy's narrowspec every time the store narrowspec is written. This starts separating those actions. Differential Revision: https://phab.mercurial-scm.org/D5509

narrow: drop now-unnecessary reassignment of repo attributes Differential Revision: https://phab.mercurial-scm.org/D5507

packaging: allow running packaging with custom uid+gid for CentOS rpmbuild in CentOS 7 has a bug causing rpmbuild to fail with "Bad owner/group" if spec or source files are owned by a different user: https://github.com/rpm-software-management/rpm/issues/2 This makes it very annoying to try and build the CentOS RPMs on CentOS with Docker. As an alternative, this change makes it possible to do so, using an environment variable. Differential Revision: https://phab.mercurial-scm.org/D5571

hg-docker: fix Python 3.4 compatibility (for CentOS 7) I realize Mercurial is not targetting Python 3.4 compatibility, but without this change, it's not even possible to build it on CentOS 7 (and I assume the same is true for RHEL 7). Differential Revision: https://phab.mercurial-scm.org/D5570

copies: use node.nullrev instead of literal -1 Differential Revision: https://phab.mercurial-scm.org/D5593

copies: use node.wdirrev instead of inventing another constant for it Differential Revision: https://phab.mercurial-scm.org/D5592

narrow: extract repo property for store narrowmatcher When a repo lock is released, we try to persist the manifest cache. That involves getting the narrowmatcher for the manifestlog. That should not fail if the store and working copy narrowspecs are out of sync. Without this patch, the later patches in this series will fail because of that. Differential Revision: https://phab.mercurial-scm.org/D5508

narrow: copy store narrowspec to working copy immediately We no longer need to delay it until the end of the transaction since we now restore a backup if the transaction aborts. Differential Revision: https://phab.mercurial-scm.org/D5506

narrow: include working copy narrowspec in transaction journal Now that we have separate narrowspecs for the store and the working copy, we need to include both in the transaction journal. Differential Revision: https://phab.mercurial-scm.org/D5505

narrow: make dirstateguard back up and restore working copy narrowspec instead We used to have only one narrowspec for the store and the working copy, but now that we have one narrowspec for each, it seems clear that the dirstateguard was supposed to back up and restore the narrowspec associated with the working copy, not the one associated with the store. clearbackup() (for the store narrowspec) is not needed because the presence of the file in localrepository._journalfiles() takes care of that. Differential Revision: https://phab.mercurial-scm.org/D5504

narrow: include journal.narrowspec in transaction journal We had missed this file before, which led to it lying around after the transaction completed. Differential Revision: https://phab.mercurial-scm.org/D5556

progress: deprecate ui.progress() It is now just a weird wrapper for ui.makeprogress(). Differential Revision: https://phab.mercurial-scm.org/D5531

context: use scmutil.matchfiles instead of matchmod.match(exact=True) Differential Revision: https://phab.mercurial-scm.org/D5591

histedit: fix call to _getgoal() by adding a byteskwargs() wrapper I also added some b-prefixes while I was here because I got confused and it seems silly to not just add them since it clarifies the whole change. Differential Revision: https://phab.mercurial-scm.org/D5585

revset: introduce an API that avoids `formatspec` input serialization Instead of having the data fully serialized, the input can be directly inserted in the tree at a later stage. Just using it for simple "%ld" case provide a significant boost. For example here are the impact on a sample discovery run between two pypy repositories with arbitrary differences (using hg perfdiscovery). $ hg perfdiscovery before: ! wall 0.700435 comb 0.710000 user 0.700000 sys 0.010000 (median of 15) after: ! wall 0.501305 comb 0.510000 user 0.490000 sys 0.020000 (median of 20)

revset: detect integer list on parsing Right now, using "%ld" with `repo.revs("…%ld…", somerevs)` is very inefficient, all items in `somerevs` will be serialized to ascii and then reparsed as integers. If `somerevs` contains just an handful of entry this is fine, however, when you get to thousands or hundreds of thousands of revisions this becomes very slow. To avoid this serialization we need to first detect this situation. The code involved in the whole process is quite complex so we start simple and focus on some "simple" but widespread cases. So far we only detect the situation and don't do anything special about it. The singled out will be serialized in `formatspec` in the same way as before.

revert: extract "%ld" formatting in a _formatintlist function We'll have to reuse this logic in different places.

revset: extract parsing logic out of formatspec We want to be able to perform better handling of some input when running revset (eg: `repo.revs("%ld", somerevs)`). The first step is to be able to access some of the parsed content before it gets substituted. There are many possible different substitutions, we'll add support for them gradually. In this changeset we support none, we just split some logic in a sub function as a preparatory step.

revset: enforce "%d" to be interpreted as literal revision number (API) (BC) Before this change, `formatspec("%d", x)` results in `"%d" % int(x)`. This seems simple and correct until you consider `nullrev`. In revset, a direct "-1" symbol is equivalent to `tip` not `nullrev`. This is a subtle error that went undetected for a while. Wrapping the revision number inside 'rev()' remove the ambiguity, preserving nullrev value passed to formatspec. It got caught by the rebase code, were the following wrongly returned `[1]`: repo.revs("children(%d) and ancestors(%ld)", 0, [nullrev]) This is flagged as API, because `%d` can be used for non-revision integer argument of revset function. We probably need to introduce a new '%…' substitution to allow literal integer (maybe `%i`). However, the `%d` usage is currently widespread for revision number so it is important to fix this issue for `%d`. This choice is reinforced by the fact _intlist is implemented as revisions only. Restricting `%d` to revision only makes things more consistent. This bug can become especially tricky since `_intlist` recognize `nullrev` right. So `revs('%ld', [-1, 0])` → select `[nullrev, 0]` but `revs('%ld', [-1])` is simplified and treated as `%d` selecting `[tip]`. Another side effect is that "%d" of an unknown revision simply match nothing. It was previously raising and error. This is consistent with what "%ld" (and `_intlist`) is doing, so it seems like a good move.

revset: remove the last usage of "%d" for a non-revision entry In order to fix an important bug, we are about to narrow the semantic of "%d" in revset. This is one of the few exceptions that we need to get rid of before being able to fix the bug. See the later semantic narrowing changeset for full rationale on the semantic change.

revset: remove a rare usage of "%d" for a non-revision item In order to fix an important bug, we are about to narrow the semantic of "%d" in revset. This is one of the few exceptions that we need to get rid of before being able to fix the bug. See the later semantic narrowing changeset for full rationale on the semantic change.

merge with stable

tests: replace mockmakedate function in test-amend.t This is a follow up patch for D5554. This replaces mockmakedate function by `tests/mockmakedate.py`. Differential Revision: https://phab.mercurial-scm.org/D5577

histedit: add rewrite.update-timestamp support to fold and mess This adds the config option to update time to current in histedit fold and mess options. Setting rewrite.update-timestamp option to `True` will update the timestamp to current time. This also adds `tests/mockmakedate.py` for supplying testable values in case current time is invoked in the tests. Differential Revision: https://phab.mercurial-scm.org/D5554

histedit: remove trailing space from warning message

histedit: fix weird indent of i18n text

progress: specify updatebar() function by constructor argument This makes it easy for ui extensions to intercept progress messages. It also seems slightly nicer in that scmutil.progress doesn't touch ui internals.

progress: change _updatebar() to take parameters as arguments I want to move updatebar() back to ui. See the next patch for why.

archival: construct progress helper through ui.makeprogress() No idea why we didn't.

help: document rewrite.backup-bundle option

repair: move ui.history-editing-backup to [rewrite] section Since we have the "rewrite" section for general history-editing options, the backup option should be there.

revlog: use separate variables to track version flags It wasn't obvious to me that "versionflags" is used both to define the default version+features value for new revlogs and to track the value read from a revlog. We rename the former use and add explicit assignment of "versionflags" later to differentiate between the two. Differential Revision: https://phab.mercurial-scm.org/D5564

revlog: inline opener options logic into _loadindex() We always call _loadindex() during __init__. But we also call _loadindex() as part of censorrevision(). Before, when reloading the index during censorrevision(), we would lose the configured mmapindexthreshold setting from the opener. By inlining the logic in _loadindex(), we ensure that opener options are always respected when loading the index. Differential Revision: https://phab.mercurial-scm.org/D5563

revlog: store mmaplargeindex as an instance attribute This makes it more consistent with everything else. It also makes it possible to move the opener/features processing into _loadindex(). Differential Revision: https://phab.mercurial-scm.org/D5562

revlog: always enable generaldelta on version 2 revlogs This commit starts the process of diverging version 2 revlogs from version 1 revlogs. generaldelta is a useful feature and has been enabled by default for ages. I can't think of a good reason why the feature should be disabled. Yes, it is true changelogs today don't have generaldelta enabled. But that's because they don't have delta chains enabled, so generaldelta makes no sense there. This commit makes generaldelta always enabled on version 2 revlogs. As part of this, one-off code in changelog.py mucking with revlog.version had to be made conditional on the revlog version, as we don't want to change revlog feature flags on version 2 revlogs. The fact this code exists is horrible and stems from revlog options being shared by the opener. We probably want a better API here. But that can wait for another patch. Differential Revision: https://phab.mercurial-scm.org/D5561

revlog: rename v to versionflags Single letter variables are harder to read. Differential Revision: https://phab.mercurial-scm.org/D5560

revlog: always process opener options I'm not sure when ``opener.options`` would ever be explicitly set to None. It is definitely not possible to construct a repo this way because ``localrepo.resolvestorevfsoptions()`` always returns a dict and ``localrepo.makelocalrepository()`` always sets ``opener.options`` to this value. Because we always execute this code now, if options are empty we defaulted to creating version 0 revlogs. So we had to change the code slightly to fall back to the default revlog version and flags. As astute reader will note that it is not possible to create version 0 revlogs now. However, I don't think it was possible before, as this required ``opener.options`` being unset, which I don't think was possible. I suspect this means our test coverage for version 0 revlog repositories is possibly non-existent! Since I don't see a config option to disable revlog v1, I'm not even sure if we had a way to create new repos with version 0 revlogs! Who knows. Differential Revision: https://phab.mercurial-scm.org/D5559

internals: minor rewriting of revlogs documentation In preparation for formalizing revlog version 2. Differential Revision: https://phab.mercurial-scm.org/D5558

tests: make HGCATAPULTSERVERPIPE imply HGTESTCATAPULTSERVERPIPE I had attempted to do this before, but missed this case. This makes it so that one can do the following to get catapult traces that include both the .t test name (and non-hg commands run by that .t test) *and* the hg-internal tracing, in one trace: HGCATAPULTSERVERPIPE=/tmp/catapult.pipe run-tests.py <args> Without this change, we need to specify both `HG{,TEST}CATAPULTSERVERPIPE`; if we specify just the TEST one, we only get the .t tests (no hg-internals), which is working as intended. If we specify the non-TEST one, we only get the hg-internals (not the rest of the .t test), which was not intended. If you want to restore the previous behavior (just hg internals, not the stuff from the .t tests), run like: HGTESTCATAPULTSERVERPIPE=/dev/null \ HGCATAPULTSERVERPIPE=/tmp/catapult.pipe \ run-tests.py <args> Differential Revision: https://phab.mercurial-scm.org/D5569

tests: add test for warning on histedit with tagged commits This is a follow-up patch to 7b7e081f8954 (D5494). Differential Revision: https://phab.mercurial-scm.org/D5552

extdiff: move external tool command line building into separate function

test: explicit a rebase source in test-rebase-collapse.t At the time of the rebase, the current revision is `null` so this rebase should fail. However, a current bug in formatspec makes it pass, we fix rebase call of the "affected" test to express its actual intent before we fix the actual bug in a later changeset.