fileset: move copy constructor of matchctx near __init__

fileset: build status according to 'withstatus' hint _switchcallers is no longer needed since 'withstatus' node is reinserted for arguments of functions like revs(). New matchctx instance is created per 'withstatus' to make sure that status tuple is available only for children of the 'withstatus' node.

fileset: insert hints where status should be computed This will allow us to compute status against a narrowed set of files. For example, "path:build/ & (unknown() + missing())" is rewritten as "path:build/ & <withstatus>(unknown() + missing(), 'unknown missing')", and the status call can be narrowed by the left-hand-side matcher, "path:build/". mctx.buildstatus() calls will be solely processed by getmatchwithstatus().

fileset: move buildstatus() to matchctx method In future patches, file status will be computed while evaluating a parsed tree. This patch provides a matchctx interface to build status.

fileset: keep basectx by matchctx

fileset: pass in basectx to _buildstatus() I'll make matchctx remember both ctx and basectx so that file status between them can be computed later. This prepares for the change.

resolve: update commands.resolve.confirm help text Included --mark and --unmark in the help text of resolve.confirm.config. Differential Revision: https://phab.mercurial-scm.org/D4103

resolve: support confirm config option with --unmark flag Now, commands.resolve.confirm also respect --unmark option; and confirm to unresolve all resolved files. It will confirm only when no files pats are passed (same as --mark), because when no pats are passed the default is to mark resolved files as unresolved. And if user has passed file pats then I think there is no need to confirm for that. Differential Revision: https://phab.mercurial-scm.org/D4102

resolve: correct behavior of mark-check=none to match docs Differential Revision: https://phab.mercurial-scm.org/D4121

narrow: move .hg/narrowspec to .hg/store/narrowspec (BC) The narrowspec is more closely related to the store than to the working copy. For example, if the narrowspec changes, the set of revlogs also needs to change (the working copy may change, but that depends on which commit is checked out). Also, when using the share extension, the narrowspec needs to be shared along with the store. This patch therefore moves the narrowspec into the store/ directory. This is clearly a breaking change, but I haven't bothered trying to fall back to reading the narrowspec from the old location (.hg/), because there are very few users of narrow out there. (We'll add a temporary hack to our Google-internal extension to handle the migration.) Differential Revision: https://phab.mercurial-scm.org/D4099

narrow: drop checkambig=True when restoring backup IIUC, checkambig is about updating timestamps of the file while renaming. That's important for the dirstate, but we never check the timestamp of the narrowspec file. We can therefore avoid checking passing checkambig=True. Differential Revision: https://phab.mercurial-scm.org/D4098

narrow: remove a repo file-cache invalidation It's unclear why this was needed. All tests pass without it. I asked Kyle Lippincott (who added the check) and he also doesn't remember what it was for. Differential Revision: https://phab.mercurial-scm.org/D4097

narrow: call narrowspec.{save,restore,clear}backup directly I want to move .hg/narrowspec to .hg/store/narrowspec and we need to decouple the narrowspec update from the dirstate update for that. This patch lets the callers call the narrowspec backup functions directly, in addition to the dirstate backup functions. The narrowspec methods are made to check if narrowing is enabled. For that, a repo instance was needed, which all the callers luckily already had available. Differential Revision: https://phab.mercurial-scm.org/D4096

index: don't add 1 to length variables A lot of "+ 1" and "-1" were mechanically added to ease the transition in 781b2720d2ac (index: don't include nullid in len(), 2018-07-20). Let's clean it up now. Differential Revision: https://phab.mercurial-scm.org/D4106

index: drop support for nullid at position len(index) in index_node I think no callers exist since at least a3dacabd476b (index: don't allow index[len(index)] to mean nullid, 2018-07-20). Differential Revision: https://phab.mercurial-scm.org/D4105

index: return False for "len(index) in index" Since we no longer accept index[len(index)], we should clearly make "len(index) in index" return False. This should have been part of a3dacabd476b (index: don't allow index[len(index)] to mean nullid, 2018-07-20) Differential Revision: https://phab.mercurial-scm.org/D4104

fileset: combine union of basic patterns into single matcher This appears to improve query performance in a big repository than I thought. Writing less Python in a hot loop, faster computation we gain. $ hg files --cwd mozilla-central --time 'set:a* + b* + c* + d* + e*' (orig) time: real 0.670 secs (user 0.640+0.000 sys 0.030+0.000) (new) time: real 0.210 secs (user 0.180+0.000 sys 0.020+0.000)

fileset: reorder 'or' expression by weight

fileset: introduce weight constants for readability These constants are defined in the filesetlang module since it's the bottommost module depending on WEIGHT_CHECK_FILENAME, and extensions will be likely to import it to process function arguments. Credit for the naming goes to Augie Fackler.

sparse: use named parameters in i18n strings This should give more hints about what the %s means, and allow reordering.

sparse: do not include operator in i18n strings

fix: compute changed lines lazily to make whole-file fixer tools faster Differential Revision: https://phab.mercurial-scm.org/D4100

resolve: support commands.resolve.confirm option with --mark flag Now, commands.resolve.confirm config option also respect --mark; and confirm to mark all 'unresolved' files as 'resolved'. It will confirm only when you don't pass any pats i.e 'hg resolve -m', because when no file pats are passed then --mark's default functionality is to mark all unresolved files. And if user has given file pats then I think there is no need to confirm. Differential Revision: https://phab.mercurial-scm.org/D4101

resolve: add commands.resolve.mark-check=abort to tweakdefaults Differential Revision: https://phab.mercurial-scm.org/D4072

resolve: graduate resolve.mark-check from experimental, add docs Since this hasn't been in a release yet, I'm not bothering to add an alias for the experimental name of the config variable. Differential Revision: https://phab.mercurial-scm.org/D4071

shortest: use 'x' prefix to disambiguate from revnum if configured Differential Revision: https://phab.mercurial-scm.org/D4042

revisions: allow "x123" to refer to nodeid prefix "123" When resolving "123" to a revision, we try to interpret it as revnum before we try to interpret it as a nodeid hex prefix. This can lead to the shortest valid prefix being longer than necessary. This patch lets us write such nodeids in a shorter form by prefixing them with "x" instead of adding more hex digits until they're longer than the longest decimal revnum. On my hg repo with almost 69k revisions, turning this feature on saves on average 0.4% on the average nodeid length. That clearly doesn't justify this patch. However, it becomes more usefule when combined with the earlier patches in this series that let you disambiguate nodeid prefixes within a configured revset. Note that we attempt to resolve symbols as nodeid prefixes after we've exhausted all other posibilities, so this is a backwards compatible change (only queries that would previously fail may now succeed). I've still hidden this feature behind an experiemntal config option so we can roll it back if needed. Differential Revision: https://phab.mercurial-scm.org/D4041

shortest: make isrev() a top-level function I'm going to add another caller in the next patch. Differential Revision: https://phab.mercurial-scm.org/D4040

shortest: cache disambiguation revset This makes it actually useful. In compared the time in my hg repo with 69.6k revisions and with a disambiguation revset of "not public()" that matches 563 visible revisions. I ran "time hg log -T '{shortest(node1,)}' -r 0:1000" (no revisions within the revset in that revision range). Before this patch, it took 57s and after it took 0.7s. Differential Revision: https://phab.mercurial-scm.org/D4039

index: don't allow index[len(index)] to mean nullid Now everything else has been cleaned up and we can drop support for getting the nullid from the end of the index (from *past* the end actually, since we reduced the length in the previous patch). Differential Revision: https://phab.mercurial-scm.org/D4023

index: don't include nullid in len() I suspect the reason the nullid is in the index in the last position is that it lets index[i] for regular revision number, even when index was just a regular Python list. An alternative solution would have been to reserve revision number 0 for the null revision. I don't know why that wasn't done. Now that we have classes backing the index, we can easily make index[-1] get the nullid without having to put it last in the list and including it in the len(). This patch just hides the nullid -- it will still be accessible at index[len(index)]. I realize that this will be annoying when checking out across this commit for debugging (including bisection). Differential Revision: https://phab.mercurial-scm.org/D4022

index: replace insert(-1, e) method by append(e) method I want to make index[len(index) - 1] be the tip revision, not null revision as it is today. insert(-1, e) will then make little sense. Since insert() currently requires the first argument to be -1, it seems simpler to replace it by a method that allows insertion only at the end. Note that revlogoldindex already has this method (by virtue of extending list). Differential Revision: https://phab.mercurial-scm.org/D4021

pure: create type for revlog v0 index The new type takes care of hiding the nullid entry. This type helps us transition away from including that entry at index[len(index)-1]. Differential Revision: https://phab.mercurial-scm.org/D4020

index: drop support for negative indexes into the index I want index[i] to work for any valid revnum including -1 (as it already does), and I also want len(index) to be the number of revisions in the index (not counting the null revision), so it cannot also support negative revision numbers other than -1 for nullid. I didn't bother removing support for it from revlog v0. Differential Revision: https://phab.mercurial-scm.org/D4019

index: handle index[-1] as nullid more explicitly I find it more intuitive to check if "pos == -1" than to first add the index length (which includes one extra item for the nullid) and compare that to "length - 1". However, because test-parseindex2.py compares the whole index (up to len(index)-1), we need to also preserve that other check for a little while more. I'll remove it soon. Differential Revision: https://phab.mercurial-scm.org/D4018

index: store nullrev as -1 in nodetree Nothing important, it just seems more natural to not map nullrev to INT_MAX. We just need to change the revision encoding a little to make space for the -1. Differential Revision: https://phab.mercurial-scm.org/D4005

index: create function for deleting node from nodetree A node is marked deleted by nt_insert(self, node, -1), but "-1" could mean so many things (notably nullrev), so let's hide it in a function. The function could possibly also remove nodes from the tree in the future (it currently only marks them deleted). Differential Revision: https://phab.mercurial-scm.org/D4004

revlog: remove some knowledge of sentinel nullid in index I think the "-2" to mean "last position in index, not counting the null revision at the end" is an implementation detail of the index that we should avoid spreading knowledge of. I hope we can even remove support for index[-2]. Differential Revision: https://phab.mercurial-scm.org/D4016

scmutil: make shortest() respect disambiguation revset The previous patch would let you use a shorter prefix if the prefix is unique within a configured revset. However, that's not very useful if there's no simple way of knowing what that shorter prefix is. This patch adapts the shortest() template function to use the shorter prefixes for nodes in the configured revset. This is currently extremely slow, because it calculates the revset for each call to shortest(). To make this faster, the next patch will start caching the revset instance. Ideally we'd cache a prefix tree instance instead. Differential Revision: https://phab.mercurial-scm.org/D4038

lookup: add option to disambiguate prefix within revset When resolving a nodeid prefix that is not unique within the repo and the user has configured a revset that they want to disambiguate within, we now try to look up within that revset before we fail. If there is a unique match within the revset, we use that. This is of course most effective at allowing a short prefix if the revset contains few nodes. For most of our internal users at Google, "not public()" is sufficiently small that a hex digit or two is enough. The implementation is currently pretty slow, but good enough for small revsets (which is the expected use case). The scan in the revset is linear. We may want to use a prefix tree if we want to allow users to use a larger revset. Credit for the idea goes to Kyle Lippincott. Differential Revision: https://phab.mercurial-scm.org/D4037

revlog: use specialized exception for ambiguous prefix lookup It's useful to be able to catch a specific exception for this case. We'll use it soon. Differential Revision: https://phab.mercurial-scm.org/D4036

narrowspec: remove the unused _parsestoredpatterns() function Previous patch started using sparse.parseconfig() to read narrowspec and now we don't need this function. Differential Revision: https://phab.mercurial-scm.org/D4058

narrowspec: use sparse.parseconfig() to parse narrowspec file (BC) This also make narrow files use 'include' and 'exclude' instead of plural forms which are 'includes' and 'excludes'. This is BC because existing narrowspecs have to replace excludes, includes words with exclude and include i.e. their singular versions. Differential Revision: https://phab.mercurial-scm.org/D4057

sparse: add an action argument to parseconfig() This will help us in reusing this function to parse narrow config files and unfiying the config file parsing logic. Differential Revision: https://phab.mercurial-scm.org/D4056

tests: avoid unnecessarily looking inside .hg/ in test-narrow* Many of the narrow tests were looking at .hg/narrowspec. I think most of them were written before we added `hg tracked`, but now that we have that command, we should fix the tests. There were also a few instances of tests looking at .hg/dirstate. I fixed those to use `hg files` instead. Differential Revision: https://phab.mercurial-scm.org/D4093

narrow: extract part of narrowspec backup to core Just a little preparation for the next patch. Differential Revision: https://phab.mercurial-scm.org/D4095

narrow: move requirement constant from changegroup to repository As suggested by Gregory Szorc. Differential Revision: https://phab.mercurial-scm.org/D4094

merge with stable

tests: update test expectations in pre-2.7.9 branch of this test As far as I can tell this is the only spot that got overlooked. Rough test methodology (run inside an hg working copy): docker run --rm -v `pwd`:/hg -it debian:wheezy bash apt-get update apt-get install python python-dev build-essential unzip mercurial \ wget libbz2-dev make testpy-2.7.8 You could try and use the system Python, but it's 2.7.3 and has lots of issues for hg these days that are not worth fixing. Differential Revision: https://phab.mercurial-scm.org/D4070

fileset: optimize 'x and not y' to 'x - y' 'x - y' is first rewritten to 'x and not y' so that x and y are reordered by weight.

fileset: reorder 'and' expression to evaluate basic patterns first Timing of a crafted example (when disk cache is warm): $ hg files set:'binary() and path:contrib' (orig) time: real 0.140 secs (user 0.120+0.000 sys 0.020+0.000) (new) time: real 0.040 secs (user 0.030+0.000 sys 0.010+0.000)

fileset: roughly adjust weights of functions ... in order to move status predicates far away from basic patterns. I don't know if each weight is appropriate, but it should be good enough as a start.

fileset: add stub for weight-based optimization The main purpose of this change is to group basic patterns, which can be mapped to a plain matcher. I'm not so interested in a weight of each function.

fileset: reject 'negate' node early while transforming parsed tree That's how a 'negate' node is processed in revset.

fileset: drop 'group' node from tree to be evaluated This helps rewriting a parsed tree.

fileset: add phase to transform parsed tree This isn't strictly necessary, but I decided to just follow the strategy of the revset parsing.