fileset: combine union of basic patterns into single matcher This appears to improve query performance in a big repository than I thought. Writing less Python in a hot loop, faster computation we gain. $ hg files --cwd mozilla-central --time 'set:a* + b* + c* + d* + e*' (orig) time: real 0.670 secs (user 0.640+0.000 sys 0.030+0.000) (new) time: real 0.210 secs (user 0.180+0.000 sys 0.020+0.000)

fileset: reorder 'or' expression by weight

fileset: introduce weight constants for readability These constants are defined in the filesetlang module since it's the bottommost module depending on WEIGHT_CHECK_FILENAME, and extensions will be likely to import it to process function arguments. Credit for the naming goes to Augie Fackler.

sparse: use named parameters in i18n strings This should give more hints about what the %s means, and allow reordering.

sparse: do not include operator in i18n strings

fix: compute changed lines lazily to make whole-file fixer tools faster Differential Revision: https://phab.mercurial-scm.org/D4100

resolve: support commands.resolve.confirm option with --mark flag Now, commands.resolve.confirm config option also respect --mark; and confirm to mark all 'unresolved' files as 'resolved'. It will confirm only when you don't pass any pats i.e 'hg resolve -m', because when no file pats are passed then --mark's default functionality is to mark all unresolved files. And if user has given file pats then I think there is no need to confirm. Differential Revision: https://phab.mercurial-scm.org/D4101

resolve: add commands.resolve.mark-check=abort to tweakdefaults Differential Revision: https://phab.mercurial-scm.org/D4072

resolve: graduate resolve.mark-check from experimental, add docs Since this hasn't been in a release yet, I'm not bothering to add an alias for the experimental name of the config variable. Differential Revision: https://phab.mercurial-scm.org/D4071

shortest: use 'x' prefix to disambiguate from revnum if configured Differential Revision: https://phab.mercurial-scm.org/D4042

revisions: allow "x123" to refer to nodeid prefix "123" When resolving "123" to a revision, we try to interpret it as revnum before we try to interpret it as a nodeid hex prefix. This can lead to the shortest valid prefix being longer than necessary. This patch lets us write such nodeids in a shorter form by prefixing them with "x" instead of adding more hex digits until they're longer than the longest decimal revnum. On my hg repo with almost 69k revisions, turning this feature on saves on average 0.4% on the average nodeid length. That clearly doesn't justify this patch. However, it becomes more usefule when combined with the earlier patches in this series that let you disambiguate nodeid prefixes within a configured revset. Note that we attempt to resolve symbols as nodeid prefixes after we've exhausted all other posibilities, so this is a backwards compatible change (only queries that would previously fail may now succeed). I've still hidden this feature behind an experiemntal config option so we can roll it back if needed. Differential Revision: https://phab.mercurial-scm.org/D4041

shortest: make isrev() a top-level function I'm going to add another caller in the next patch. Differential Revision: https://phab.mercurial-scm.org/D4040

shortest: cache disambiguation revset This makes it actually useful. In compared the time in my hg repo with 69.6k revisions and with a disambiguation revset of "not public()" that matches 563 visible revisions. I ran "time hg log -T '{shortest(node1,)}' -r 0:1000" (no revisions within the revset in that revision range). Before this patch, it took 57s and after it took 0.7s. Differential Revision: https://phab.mercurial-scm.org/D4039

index: don't allow index[len(index)] to mean nullid Now everything else has been cleaned up and we can drop support for getting the nullid from the end of the index (from *past* the end actually, since we reduced the length in the previous patch). Differential Revision: https://phab.mercurial-scm.org/D4023

index: don't include nullid in len() I suspect the reason the nullid is in the index in the last position is that it lets index[i] for regular revision number, even when index was just a regular Python list. An alternative solution would have been to reserve revision number 0 for the null revision. I don't know why that wasn't done. Now that we have classes backing the index, we can easily make index[-1] get the nullid without having to put it last in the list and including it in the len(). This patch just hides the nullid -- it will still be accessible at index[len(index)]. I realize that this will be annoying when checking out across this commit for debugging (including bisection). Differential Revision: https://phab.mercurial-scm.org/D4022

index: replace insert(-1, e) method by append(e) method I want to make index[len(index) - 1] be the tip revision, not null revision as it is today. insert(-1, e) will then make little sense. Since insert() currently requires the first argument to be -1, it seems simpler to replace it by a method that allows insertion only at the end. Note that revlogoldindex already has this method (by virtue of extending list). Differential Revision: https://phab.mercurial-scm.org/D4021

pure: create type for revlog v0 index The new type takes care of hiding the nullid entry. This type helps us transition away from including that entry at index[len(index)-1]. Differential Revision: https://phab.mercurial-scm.org/D4020

index: drop support for negative indexes into the index I want index[i] to work for any valid revnum including -1 (as it already does), and I also want len(index) to be the number of revisions in the index (not counting the null revision), so it cannot also support negative revision numbers other than -1 for nullid. I didn't bother removing support for it from revlog v0. Differential Revision: https://phab.mercurial-scm.org/D4019

index: handle index[-1] as nullid more explicitly I find it more intuitive to check if "pos == -1" than to first add the index length (which includes one extra item for the nullid) and compare that to "length - 1". However, because test-parseindex2.py compares the whole index (up to len(index)-1), we need to also preserve that other check for a little while more. I'll remove it soon. Differential Revision: https://phab.mercurial-scm.org/D4018

index: store nullrev as -1 in nodetree Nothing important, it just seems more natural to not map nullrev to INT_MAX. We just need to change the revision encoding a little to make space for the -1. Differential Revision: https://phab.mercurial-scm.org/D4005

index: create function for deleting node from nodetree A node is marked deleted by nt_insert(self, node, -1), but "-1" could mean so many things (notably nullrev), so let's hide it in a function. The function could possibly also remove nodes from the tree in the future (it currently only marks them deleted). Differential Revision: https://phab.mercurial-scm.org/D4004

revlog: remove some knowledge of sentinel nullid in index I think the "-2" to mean "last position in index, not counting the null revision at the end" is an implementation detail of the index that we should avoid spreading knowledge of. I hope we can even remove support for index[-2]. Differential Revision: https://phab.mercurial-scm.org/D4016

scmutil: make shortest() respect disambiguation revset The previous patch would let you use a shorter prefix if the prefix is unique within a configured revset. However, that's not very useful if there's no simple way of knowing what that shorter prefix is. This patch adapts the shortest() template function to use the shorter prefixes for nodes in the configured revset. This is currently extremely slow, because it calculates the revset for each call to shortest(). To make this faster, the next patch will start caching the revset instance. Ideally we'd cache a prefix tree instance instead. Differential Revision: https://phab.mercurial-scm.org/D4038

lookup: add option to disambiguate prefix within revset When resolving a nodeid prefix that is not unique within the repo and the user has configured a revset that they want to disambiguate within, we now try to look up within that revset before we fail. If there is a unique match within the revset, we use that. This is of course most effective at allowing a short prefix if the revset contains few nodes. For most of our internal users at Google, "not public()" is sufficiently small that a hex digit or two is enough. The implementation is currently pretty slow, but good enough for small revsets (which is the expected use case). The scan in the revset is linear. We may want to use a prefix tree if we want to allow users to use a larger revset. Credit for the idea goes to Kyle Lippincott. Differential Revision: https://phab.mercurial-scm.org/D4037

revlog: use specialized exception for ambiguous prefix lookup It's useful to be able to catch a specific exception for this case. We'll use it soon. Differential Revision: https://phab.mercurial-scm.org/D4036

narrowspec: remove the unused _parsestoredpatterns() function Previous patch started using sparse.parseconfig() to read narrowspec and now we don't need this function. Differential Revision: https://phab.mercurial-scm.org/D4058

narrowspec: use sparse.parseconfig() to parse narrowspec file (BC) This also make narrow files use 'include' and 'exclude' instead of plural forms which are 'includes' and 'excludes'. This is BC because existing narrowspecs have to replace excludes, includes words with exclude and include i.e. their singular versions. Differential Revision: https://phab.mercurial-scm.org/D4057

sparse: add an action argument to parseconfig() This will help us in reusing this function to parse narrow config files and unfiying the config file parsing logic. Differential Revision: https://phab.mercurial-scm.org/D4056

tests: avoid unnecessarily looking inside .hg/ in test-narrow* Many of the narrow tests were looking at .hg/narrowspec. I think most of them were written before we added `hg tracked`, but now that we have that command, we should fix the tests. There were also a few instances of tests looking at .hg/dirstate. I fixed those to use `hg files` instead. Differential Revision: https://phab.mercurial-scm.org/D4093

narrow: extract part of narrowspec backup to core Just a little preparation for the next patch. Differential Revision: https://phab.mercurial-scm.org/D4095

narrow: move requirement constant from changegroup to repository As suggested by Gregory Szorc. Differential Revision: https://phab.mercurial-scm.org/D4094

merge with stable

fileset: optimize 'x and not y' to 'x - y' 'x - y' is first rewritten to 'x and not y' so that x and y are reordered by weight.

fileset: reorder 'and' expression to evaluate basic patterns first Timing of a crafted example (when disk cache is warm): $ hg files set:'binary() and path:contrib' (orig) time: real 0.140 secs (user 0.120+0.000 sys 0.020+0.000) (new) time: real 0.040 secs (user 0.030+0.000 sys 0.010+0.000)

fileset: roughly adjust weights of functions ... in order to move status predicates far away from basic patterns. I don't know if each weight is appropriate, but it should be good enough as a start.

fileset: add stub for weight-based optimization The main purpose of this change is to group basic patterns, which can be mapped to a plain matcher. I'm not so interested in a weight of each function.

fileset: reject 'negate' node early while transforming parsed tree That's how a 'negate' node is processed in revset.

fileset: drop 'group' node from tree to be evaluated This helps rewriting a parsed tree.

fileset: add phase to transform parsed tree This isn't strictly necessary, but I decided to just follow the strategy of the revset parsing.

py3: whitelist another 3 tests caught by the ratchet Differential Revision: https://phab.mercurial-scm.org/D4069

testrunner: allow multiple #testcases This lets you have multiple #testcases declarations and they're taken to be different dimensions of the test, so their cross product becomes the total set of test cases. For example: #testcases obsstore-on obsstore-off #testcases manifest-flat manifest-tree $ hg rebase ... ... #if obsstore-on $ hg log ... #endif Note that this is an excellent way to slow down the test suite, so use it with care. The feature is implemented by replacing most of the "case" variables that were strings before by an array of strings with each item a different dimension of the test case. The file names are created by joining the dimensions by "#" (e.g. test-foo.t#obsstore-on#manifest-flat). Differential Revision: https://phab.mercurial-scm.org/D4049

testrunner: use "#" for "test cases" suffix in .err filename too This seems like a natural follow-up to b865bba56db1 (run-tests: update the test case name format, 2018-05-13). Differential Revision: https://phab.mercurial-scm.org/D4052

resolve: add confirm config option This config setting gives a functionality to confirm before it re-merge all unresolved files. If this config is enabled, when you run 'hg resolve --all' it will prompt with a msg "re-merge all unresolved files (yn)?" To enable this functionality: [commands] resolve.confirm = True Differential Revision: https://phab.mercurial-scm.org/D3988

revlog: remove side effect from failed nt_init() If nt_init() successfully allocates memory for the node tree but then runs out of memory while trying to insert nullid into it, it will leave the node tree pointer set on the index object. That means that future node tree operations will think that the tree is properly initialized. It seems very unlikely to make a difference in practice (if nt_init() runs out of memory, not much else will probably work), but since I spotted it, I figured I might as well fix it. Differential Revision: https://phab.mercurial-scm.org/D4028

revlog: remove micro-optimization for looking up only nullid index_find_node() would call nt_find() before initializing the node tree. nt_find() would then return -2 unless the requested revision was the null revision. I can't imagine what scenario that is optimizing for, and doing the initialization earlier makes the code simpler and easier to follow, so that's what this patch does. Differential Revision: https://phab.mercurial-scm.org/D4027

revlog: remove unnecessary output parameter from node_check() The "nodelen" output parameter is always set to 20 if the function returns successfully. Differential Revision: https://phab.mercurial-scm.org/D4026

narrow: move status-filtering to core and to ctx One of my recent changes from repo.status(ctx1, ctx2) to ctx1.status(ctx2) broke some of our Google-internal tests. The problem turned out to be that the narrow extension was overriding repo.status() to make it filter out paths outside the narrowspec. When I changed to ctx1.status(ctx2), then that filtering obviously got lost. ctx.status() seems like a better method to do the filtering in, so this patch moves the filtering into that method, thereby also moving it out of the extension and into core. Differential Revision: https://phab.mercurial-scm.org/D4068

amend: support "history-editing-backup" config option Now, amend is in the list of those history editing commands which support `history-editing-backup` config option. If you don't want to store any backup then just use this config. [ui] hisotry-editing-backup = False Current status of list of history editing commands which support this config: 1. histedit 2. rebase 3. amend Differential Revision: https://phab.mercurial-scm.org/D3968

rebase: move "backup" flag to rebaseruntime It was getting messy to populate "backup" flag to calls of `_finishrebase()` and `_prepareabortorcontinue`, so made some changes to move "backup" flag to rbsrt. Differential Revision: https://phab.mercurial-scm.org/D4055

i18n: format warning of hggettext in standard compiler error style Now, hggettext specific warning messages are formatted in: FILENAME:LINENO:MESSAGE This allows editors to jump into corresponded line easily.

i18n: omit redundant translatable synopsis text to avoid xgettext warning This empty translatable synopsis text causes xgettext warning below: Empty msgid. It is reserved by GNU gettext: gettext("") returns the header entry with meta information, not the empty string. This "synopsis" argument of @command annotation is omitable.

i18n: avoid substitution of PYFILES at runtime for readability of output This substitution decreases readability of "make update-pot" output, because PYFILES consists of many files. This patch makes "make update-pot" show "find mercurial hgext doc -name '*.py'" instead of many *.py files at runtime.

util: make new timedcmstats class Python 3 compatible

fix: add a monkey-patchable point after all new revisions have been committed Differential Revision: https://phab.mercurial-scm.org/D4048

help: add quotes to a few commands we point to I didn't know that 'hg help "revsets.x or y"' was valid syntax, so the quoting is extra useful here to make it clear that that is an actual command. Differential Revision: https://phab.mercurial-scm.org/D4059

linelog: port to Python 3 Differential Revision: https://phab.mercurial-scm.org/D4051

exchange: move simple narrow changegroup generation from extension The narrow extension completely replaced the function generating the changegroup part when a narrow changegroup was requested. Previous commits have taught the in-core changegroup code how to filter files based on a matcher. This commit teaches the in-core bundle2 part generation code to construct a matcher based on arguments. It will also emit a bundle2 part describing the narrow spec. I believe the changegroup part generation code in the narrow extension is now limited to ellipsis serving mode. i.e. core is now capable of narrow changegroup generation when ellipsis mode is disabled. Differential Revision: https://phab.mercurial-scm.org/D4014

exchange: make narrow ACL presence imply narrow=True And refactor the logic for determining when to invoke our custom changegroup part function so it is only conditional on narrow being set. This makes it more obvious under which conditions we should invoke the custom implementation. Also use raw strings so **kwargs works on Python 3. Differential Revision: https://phab.mercurial-scm.org/D4013

changegroup: inline prune() logic from narrow prune() needs to ellide manifests that aren't part of the narrow matcher. The code is violating storage abstractions, so a comment has been added. Keep in mind the impetus for moving this code to core is so changegroup code can be refactored to be storage agnostic. Differential Revision: https://phab.mercurial-scm.org/D4012

fileset: extract language processing part to new module (API) I'll add a couple more functions that work on parsed tree. % wc -l mercurial/fileset*.py 559 mercurial/fileset.py 135 mercurial/filesetlang.py 694 total