narrow: move copytonarrowspec() out of setnarrowpats() I think it was a mistake to write the working copy's narrowspec every time the store narrowspec is written. This starts separating those actions. Differential Revision: https://phab.mercurial-scm.org/D5509

narrow: drop now-unnecessary reassignment of repo attributes Differential Revision: https://phab.mercurial-scm.org/D5507

packaging: allow running packaging with custom uid+gid for CentOS rpmbuild in CentOS 7 has a bug causing rpmbuild to fail with "Bad owner/group" if spec or source files are owned by a different user: https://github.com/rpm-software-management/rpm/issues/2 This makes it very annoying to try and build the CentOS RPMs on CentOS with Docker. As an alternative, this change makes it possible to do so, using an environment variable. Differential Revision: https://phab.mercurial-scm.org/D5571

hg-docker: fix Python 3.4 compatibility (for CentOS 7) I realize Mercurial is not targetting Python 3.4 compatibility, but without this change, it's not even possible to build it on CentOS 7 (and I assume the same is true for RHEL 7). Differential Revision: https://phab.mercurial-scm.org/D5570

copies: use node.nullrev instead of literal -1 Differential Revision: https://phab.mercurial-scm.org/D5593

copies: use node.wdirrev instead of inventing another constant for it Differential Revision: https://phab.mercurial-scm.org/D5592

narrow: extract repo property for store narrowmatcher When a repo lock is released, we try to persist the manifest cache. That involves getting the narrowmatcher for the manifestlog. That should not fail if the store and working copy narrowspecs are out of sync. Without this patch, the later patches in this series will fail because of that. Differential Revision: https://phab.mercurial-scm.org/D5508

narrow: copy store narrowspec to working copy immediately We no longer need to delay it until the end of the transaction since we now restore a backup if the transaction aborts. Differential Revision: https://phab.mercurial-scm.org/D5506

narrow: include working copy narrowspec in transaction journal Now that we have separate narrowspecs for the store and the working copy, we need to include both in the transaction journal. Differential Revision: https://phab.mercurial-scm.org/D5505

narrow: make dirstateguard back up and restore working copy narrowspec instead We used to have only one narrowspec for the store and the working copy, but now that we have one narrowspec for each, it seems clear that the dirstateguard was supposed to back up and restore the narrowspec associated with the working copy, not the one associated with the store. clearbackup() (for the store narrowspec) is not needed because the presence of the file in localrepository._journalfiles() takes care of that. Differential Revision: https://phab.mercurial-scm.org/D5504

narrow: include journal.narrowspec in transaction journal We had missed this file before, which led to it lying around after the transaction completed. Differential Revision: https://phab.mercurial-scm.org/D5556

progress: deprecate ui.progress() It is now just a weird wrapper for ui.makeprogress(). Differential Revision: https://phab.mercurial-scm.org/D5531

context: use scmutil.matchfiles instead of matchmod.match(exact=True) Differential Revision: https://phab.mercurial-scm.org/D5591

histedit: fix call to _getgoal() by adding a byteskwargs() wrapper I also added some b-prefixes while I was here because I got confused and it seems silly to not just add them since it clarifies the whole change. Differential Revision: https://phab.mercurial-scm.org/D5585

revset: introduce an API that avoids `formatspec` input serialization Instead of having the data fully serialized, the input can be directly inserted in the tree at a later stage. Just using it for simple "%ld" case provide a significant boost. For example here are the impact on a sample discovery run between two pypy repositories with arbitrary differences (using hg perfdiscovery). $ hg perfdiscovery before: ! wall 0.700435 comb 0.710000 user 0.700000 sys 0.010000 (median of 15) after: ! wall 0.501305 comb 0.510000 user 0.490000 sys 0.020000 (median of 20)

revset: detect integer list on parsing Right now, using "%ld" with `repo.revs("…%ld…", somerevs)` is very inefficient, all items in `somerevs` will be serialized to ascii and then reparsed as integers. If `somerevs` contains just an handful of entry this is fine, however, when you get to thousands or hundreds of thousands of revisions this becomes very slow. To avoid this serialization we need to first detect this situation. The code involved in the whole process is quite complex so we start simple and focus on some "simple" but widespread cases. So far we only detect the situation and don't do anything special about it. The singled out will be serialized in `formatspec` in the same way as before.

revert: extract "%ld" formatting in a _formatintlist function We'll have to reuse this logic in different places.

revset: extract parsing logic out of formatspec We want to be able to perform better handling of some input when running revset (eg: `repo.revs("%ld", somerevs)`). The first step is to be able to access some of the parsed content before it gets substituted. There are many possible different substitutions, we'll add support for them gradually. In this changeset we support none, we just split some logic in a sub function as a preparatory step.

revset: enforce "%d" to be interpreted as literal revision number (API) (BC) Before this change, `formatspec("%d", x)` results in `"%d" % int(x)`. This seems simple and correct until you consider `nullrev`. In revset, a direct "-1" symbol is equivalent to `tip` not `nullrev`. This is a subtle error that went undetected for a while. Wrapping the revision number inside 'rev()' remove the ambiguity, preserving nullrev value passed to formatspec. It got caught by the rebase code, were the following wrongly returned `[1]`: repo.revs("children(%d) and ancestors(%ld)", 0, [nullrev]) This is flagged as API, because `%d` can be used for non-revision integer argument of revset function. We probably need to introduce a new '%…' substitution to allow literal integer (maybe `%i`). However, the `%d` usage is currently widespread for revision number so it is important to fix this issue for `%d`. This choice is reinforced by the fact _intlist is implemented as revisions only. Restricting `%d` to revision only makes things more consistent. This bug can become especially tricky since `_intlist` recognize `nullrev` right. So `revs('%ld', [-1, 0])` → select `[nullrev, 0]` but `revs('%ld', [-1])` is simplified and treated as `%d` selecting `[tip]`. Another side effect is that "%d" of an unknown revision simply match nothing. It was previously raising and error. This is consistent with what "%ld" (and `_intlist`) is doing, so it seems like a good move.

revset: remove the last usage of "%d" for a non-revision entry In order to fix an important bug, we are about to narrow the semantic of "%d" in revset. This is one of the few exceptions that we need to get rid of before being able to fix the bug. See the later semantic narrowing changeset for full rationale on the semantic change.

revset: remove a rare usage of "%d" for a non-revision item In order to fix an important bug, we are about to narrow the semantic of "%d" in revset. This is one of the few exceptions that we need to get rid of before being able to fix the bug. See the later semantic narrowing changeset for full rationale on the semantic change.

merge with stable

tests: replace mockmakedate function in test-amend.t This is a follow up patch for D5554. This replaces mockmakedate function by `tests/mockmakedate.py`. Differential Revision: https://phab.mercurial-scm.org/D5577

histedit: add rewrite.update-timestamp support to fold and mess This adds the config option to update time to current in histedit fold and mess options. Setting rewrite.update-timestamp option to `True` will update the timestamp to current time. This also adds `tests/mockmakedate.py` for supplying testable values in case current time is invoked in the tests. Differential Revision: https://phab.mercurial-scm.org/D5554

histedit: remove trailing space from warning message

histedit: fix weird indent of i18n text

progress: specify updatebar() function by constructor argument This makes it easy for ui extensions to intercept progress messages. It also seems slightly nicer in that scmutil.progress doesn't touch ui internals.

progress: change _updatebar() to take parameters as arguments I want to move updatebar() back to ui. See the next patch for why.

archival: construct progress helper through ui.makeprogress() No idea why we didn't.

help: document rewrite.backup-bundle option

repair: move ui.history-editing-backup to [rewrite] section Since we have the "rewrite" section for general history-editing options, the backup option should be there.

revlog: use separate variables to track version flags It wasn't obvious to me that "versionflags" is used both to define the default version+features value for new revlogs and to track the value read from a revlog. We rename the former use and add explicit assignment of "versionflags" later to differentiate between the two. Differential Revision: https://phab.mercurial-scm.org/D5564

revlog: inline opener options logic into _loadindex() We always call _loadindex() during __init__. But we also call _loadindex() as part of censorrevision(). Before, when reloading the index during censorrevision(), we would lose the configured mmapindexthreshold setting from the opener. By inlining the logic in _loadindex(), we ensure that opener options are always respected when loading the index. Differential Revision: https://phab.mercurial-scm.org/D5563

revlog: store mmaplargeindex as an instance attribute This makes it more consistent with everything else. It also makes it possible to move the opener/features processing into _loadindex(). Differential Revision: https://phab.mercurial-scm.org/D5562

revlog: always enable generaldelta on version 2 revlogs This commit starts the process of diverging version 2 revlogs from version 1 revlogs. generaldelta is a useful feature and has been enabled by default for ages. I can't think of a good reason why the feature should be disabled. Yes, it is true changelogs today don't have generaldelta enabled. But that's because they don't have delta chains enabled, so generaldelta makes no sense there. This commit makes generaldelta always enabled on version 2 revlogs. As part of this, one-off code in changelog.py mucking with revlog.version had to be made conditional on the revlog version, as we don't want to change revlog feature flags on version 2 revlogs. The fact this code exists is horrible and stems from revlog options being shared by the opener. We probably want a better API here. But that can wait for another patch. Differential Revision: https://phab.mercurial-scm.org/D5561

revlog: rename v to versionflags Single letter variables are harder to read. Differential Revision: https://phab.mercurial-scm.org/D5560

revlog: always process opener options I'm not sure when ``opener.options`` would ever be explicitly set to None. It is definitely not possible to construct a repo this way because ``localrepo.resolvestorevfsoptions()`` always returns a dict and ``localrepo.makelocalrepository()`` always sets ``opener.options`` to this value. Because we always execute this code now, if options are empty we defaulted to creating version 0 revlogs. So we had to change the code slightly to fall back to the default revlog version and flags. As astute reader will note that it is not possible to create version 0 revlogs now. However, I don't think it was possible before, as this required ``opener.options`` being unset, which I don't think was possible. I suspect this means our test coverage for version 0 revlog repositories is possibly non-existent! Since I don't see a config option to disable revlog v1, I'm not even sure if we had a way to create new repos with version 0 revlogs! Who knows. Differential Revision: https://phab.mercurial-scm.org/D5559

internals: minor rewriting of revlogs documentation In preparation for formalizing revlog version 2. Differential Revision: https://phab.mercurial-scm.org/D5558

tests: make HGCATAPULTSERVERPIPE imply HGTESTCATAPULTSERVERPIPE I had attempted to do this before, but missed this case. This makes it so that one can do the following to get catapult traces that include both the .t test name (and non-hg commands run by that .t test) *and* the hg-internal tracing, in one trace: HGCATAPULTSERVERPIPE=/tmp/catapult.pipe run-tests.py <args> Without this change, we need to specify both `HG{,TEST}CATAPULTSERVERPIPE`; if we specify just the TEST one, we only get the .t tests (no hg-internals), which is working as intended. If we specify the non-TEST one, we only get the hg-internals (not the rest of the .t test), which was not intended. If you want to restore the previous behavior (just hg internals, not the stuff from the .t tests), run like: HGTESTCATAPULTSERVERPIPE=/dev/null \ HGCATAPULTSERVERPIPE=/tmp/catapult.pipe \ run-tests.py <args> Differential Revision: https://phab.mercurial-scm.org/D5569

tests: add test for warning on histedit with tagged commits This is a follow-up patch to 7b7e081f8954 (D5494). Differential Revision: https://phab.mercurial-scm.org/D5552

extdiff: move external tool command line building into separate function

test: explicit a rebase source in test-rebase-collapse.t At the time of the rebase, the current revision is `null` so this rebase should fail. However, a current bug in formatspec makes it pass, we fix rebase call of the "affected" test to express its actual intent before we fix the actual bug in a later changeset.

tags: join string that's unnecessarily split across lines Differential Revision: https://phab.mercurial-scm.org/D5574

dispatch: add newline after ui.log "ui blocked ms" message Differential Revision: https://phab.mercurial-scm.org/D5573

remotefilelog: fix {file_copies} template keyword remotefilelog overrides templatekw.getrenamedfn(). I changed the return type of that function in ec37df9042f9 (templatekw: make getrenamed() return only filename, not nodeid, 2018-03-26). So we need to change remotefilelog too. The function is used by the {file_copies} template keyword and by `hg log --copies`. Differential Revision: https://phab.mercurial-scm.org/D5576

tests: demonstrate broken {file_copies} template with remotefilelog Differential Revision: https://phab.mercurial-scm.org/D5575

contrib: update window title when possible in perf-revlog-write-plot.py This is useful when comparing multiple graphs.

histedit: add newline after ui.log "# acttions to histedit" message Differential Revision: https://phab.mercurial-scm.org/D5572

rust-cpython: bindings for MissingAncestors The exposition is rather straightforward, except for the remove_ancestors_from() method, which forces us to an inefficient conversion between Python sets and Rust HashSets. Two alternatives are proposed in comments: - changing the inner API to "emit" the revision numbers to discard this would be a substantial change, and it would be better only in the cases where there are more to retain than to discard - mutating the Python set directly: this would force us to define an abstract `RevisionSet` trait, and implement it both for plain `HashSet` and for a struct enclosing a Python set with the GIL marker `Python<'p>`, also a non trivial effort. The main (and seemingly only) caller of this method being `mercurial.setdiscovery`, which is currently undergoing serious refactoring, it's not clear whether these improvements would be worth the effort right now, so we're leaving it as-is. Also, in `get_bases()` (will also be used by `setdiscovery`), we'd prefer to build a Python set directly, but we resort to returning a tuple, waiting to hear back from our PR onto rust-cpython about that Differential Revision: https://phab.mercurial-scm.org/D5550

rust-cpython: generalised conversion function Because `hg::ancestors::MissingAncestors` has methods taking some `HashSet<Revision>` besides `impl IntoIterator<Item = Revision>` as parameters we'll need the more generic `rev_pyiter_collect()` function to also build these Differential Revision: https://phab.mercurial-scm.org/D5549

rust-cpython: style consistency leftovers In particular, during review of `LazyAncestors` bindings, most `match` statements for error conversion have been replaced by higher level methods of `Result` and most personal insecurity comments have been removed. This makes it more systematic. Differential Revision: https://phab.mercurial-scm.org/D5548

rust-cpython: consistency in use of hg-core constructs While not everybody likes the `CoreLazy` and `CoreIterator` aliases, it's better not to mix them with direct references. Note: it's quite possible in the future that these would stop being exposed at the top of the `hg` crate Differential Revision: https://phab.mercurial-scm.org/D5547

rust-cpython: rustdoc improvements By default, `cargo doc` builds the documentation for public constructs only, so we make public those that can. Since `cindex` is not safe, we keep it private. Unfortunately, the macro syntax of rust-cpython doesn't allow us to document the classes directly, so we resort to do that at the module level. Differential Revision: https://phab.mercurial-scm.org/D5546

fuzz: stop building Python in the Makefile $OUT/sanpy is now built by the build.sh in oss-fuzz, which simplifies the setup here considerably. Differential Revision: https://phab.mercurial-scm.org/D5553

py3: add a test to whitelist caught by buildbot Differential Revision: https://phab.mercurial-scm.org/D5555

tests: de-flake test-narrow-share.t by making dirstate predictable test-narrow-share.t was sometimes (~0.5% on my machine) failing like this: @@ -61,7 +61,7 @@ A d3/g $ hg -R main debugdirstate --no-dates n 644 2 set d1/f - n 644 2 unset d3/f + n 644 2 set d3/f a 0 -1 unset d3/g n 644 2 set d5/f n 644 2 set d7/f The timestamp for d3/f would get set if it was determined at some point that it was clean. That check is usually done when the user runs `hg st`. We don't do that before the failure in the test case, but it happens at the end of the `hg clone` call. So if the file system's time happens to roll over after the clone's working copy has been written, but before its (final) dirstate has been written, we can end up with a set timestamp there. This patch makes it consistent by sleeping for 2 seconds so the timestamp gets reliably set. Differential Revision: https://phab.mercurial-scm.org/D5568

convert: don't drop commits that are empty in the source when using --filemap I ran into this when using `hg lfconvert --to-normal` (which uses the filemap class internally), and saw that commits with nothing but a branch change were dropped. We could put in an option that only lfconvert uses internally. But silently dropping anything other than a commit where all changes were excluded seems unintended. For example, there's a message in mercurial_sink.putcommit() if it drops an empty commit. (And the reason that isn't kicking in here is because lfconvert isn't passing --filemap, so the self.filemapmode conditional there is always False.) The naive change of `return not files` broke test-convert-filemap.t, so this is a little more elaborate than needed for converting from largefiles.

convert: add missing numcommits() override to hg sources Otherwise, the progressbar in converter.walktree() gets None as its total during the initial scan. (Though that seems harmless, and there are other foreign vcs sources without this.)

testrunner: make `-j100 --runs-per-test=100 test-foo.t` use 100 jobs It would use only 1 job since 1039404c5e1d (run-tests: print number of tests and parallel process count, 2018-10-13), but I'm pretty sure that was unintentional. Differential Revision: https://phab.mercurial-scm.org/D5557

narrow: use merge.ACTION_GET instead of duplicating 'g' constant As suggested by Yuya. Differential Revision: https://phab.mercurial-scm.org/D5512