revisions: allow "x123" to refer to nodeid prefix "123" When resolving "123" to a revision, we try to interpret it as revnum before we try to interpret it as a nodeid hex prefix. This can lead to the shortest valid prefix being longer than necessary. This patch lets us write such nodeids in a shorter form by prefixing them with "x" instead of adding more hex digits until they're longer than the longest decimal revnum. On my hg repo with almost 69k revisions, turning this feature on saves on average 0.4% on the average nodeid length. That clearly doesn't justify this patch. However, it becomes more usefule when combined with the earlier patches in this series that let you disambiguate nodeid prefixes within a configured revset. Note that we attempt to resolve symbols as nodeid prefixes after we've exhausted all other posibilities, so this is a backwards compatible change (only queries that would previously fail may now succeed). I've still hidden this feature behind an experiemntal config option so we can roll it back if needed. Differential Revision: https://phab.mercurial-scm.org/D4041

shortest: make isrev() a top-level function I'm going to add another caller in the next patch. Differential Revision: https://phab.mercurial-scm.org/D4040

shortest: cache disambiguation revset This makes it actually useful. In compared the time in my hg repo with 69.6k revisions and with a disambiguation revset of "not public()" that matches 563 visible revisions. I ran "time hg log -T '{shortest(node1,)}' -r 0:1000" (no revisions within the revset in that revision range). Before this patch, it took 57s and after it took 0.7s. Differential Revision: https://phab.mercurial-scm.org/D4039

index: don't allow index[len(index)] to mean nullid Now everything else has been cleaned up and we can drop support for getting the nullid from the end of the index (from *past* the end actually, since we reduced the length in the previous patch). Differential Revision: https://phab.mercurial-scm.org/D4023

index: don't include nullid in len() I suspect the reason the nullid is in the index in the last position is that it lets index[i] for regular revision number, even when index was just a regular Python list. An alternative solution would have been to reserve revision number 0 for the null revision. I don't know why that wasn't done. Now that we have classes backing the index, we can easily make index[-1] get the nullid without having to put it last in the list and including it in the len(). This patch just hides the nullid -- it will still be accessible at index[len(index)]. I realize that this will be annoying when checking out across this commit for debugging (including bisection). Differential Revision: https://phab.mercurial-scm.org/D4022

index: replace insert(-1, e) method by append(e) method I want to make index[len(index) - 1] be the tip revision, not null revision as it is today. insert(-1, e) will then make little sense. Since insert() currently requires the first argument to be -1, it seems simpler to replace it by a method that allows insertion only at the end. Note that revlogoldindex already has this method (by virtue of extending list). Differential Revision: https://phab.mercurial-scm.org/D4021

pure: create type for revlog v0 index The new type takes care of hiding the nullid entry. This type helps us transition away from including that entry at index[len(index)-1]. Differential Revision: https://phab.mercurial-scm.org/D4020

index: drop support for negative indexes into the index I want index[i] to work for any valid revnum including -1 (as it already does), and I also want len(index) to be the number of revisions in the index (not counting the null revision), so it cannot also support negative revision numbers other than -1 for nullid. I didn't bother removing support for it from revlog v0. Differential Revision: https://phab.mercurial-scm.org/D4019

index: handle index[-1] as nullid more explicitly I find it more intuitive to check if "pos == -1" than to first add the index length (which includes one extra item for the nullid) and compare that to "length - 1". However, because test-parseindex2.py compares the whole index (up to len(index)-1), we need to also preserve that other check for a little while more. I'll remove it soon. Differential Revision: https://phab.mercurial-scm.org/D4018

index: store nullrev as -1 in nodetree Nothing important, it just seems more natural to not map nullrev to INT_MAX. We just need to change the revision encoding a little to make space for the -1. Differential Revision: https://phab.mercurial-scm.org/D4005

index: create function for deleting node from nodetree A node is marked deleted by nt_insert(self, node, -1), but "-1" could mean so many things (notably nullrev), so let's hide it in a function. The function could possibly also remove nodes from the tree in the future (it currently only marks them deleted). Differential Revision: https://phab.mercurial-scm.org/D4004

revlog: remove some knowledge of sentinel nullid in index I think the "-2" to mean "last position in index, not counting the null revision at the end" is an implementation detail of the index that we should avoid spreading knowledge of. I hope we can even remove support for index[-2]. Differential Revision: https://phab.mercurial-scm.org/D4016

scmutil: make shortest() respect disambiguation revset The previous patch would let you use a shorter prefix if the prefix is unique within a configured revset. However, that's not very useful if there's no simple way of knowing what that shorter prefix is. This patch adapts the shortest() template function to use the shorter prefixes for nodes in the configured revset. This is currently extremely slow, because it calculates the revset for each call to shortest(). To make this faster, the next patch will start caching the revset instance. Ideally we'd cache a prefix tree instance instead. Differential Revision: https://phab.mercurial-scm.org/D4038

lookup: add option to disambiguate prefix within revset When resolving a nodeid prefix that is not unique within the repo and the user has configured a revset that they want to disambiguate within, we now try to look up within that revset before we fail. If there is a unique match within the revset, we use that. This is of course most effective at allowing a short prefix if the revset contains few nodes. For most of our internal users at Google, "not public()" is sufficiently small that a hex digit or two is enough. The implementation is currently pretty slow, but good enough for small revsets (which is the expected use case). The scan in the revset is linear. We may want to use a prefix tree if we want to allow users to use a larger revset. Credit for the idea goes to Kyle Lippincott. Differential Revision: https://phab.mercurial-scm.org/D4037

revlog: use specialized exception for ambiguous prefix lookup It's useful to be able to catch a specific exception for this case. We'll use it soon. Differential Revision: https://phab.mercurial-scm.org/D4036

narrowspec: remove the unused _parsestoredpatterns() function Previous patch started using sparse.parseconfig() to read narrowspec and now we don't need this function. Differential Revision: https://phab.mercurial-scm.org/D4058

narrowspec: use sparse.parseconfig() to parse narrowspec file (BC) This also make narrow files use 'include' and 'exclude' instead of plural forms which are 'includes' and 'excludes'. This is BC because existing narrowspecs have to replace excludes, includes words with exclude and include i.e. their singular versions. Differential Revision: https://phab.mercurial-scm.org/D4057

sparse: add an action argument to parseconfig() This will help us in reusing this function to parse narrow config files and unfiying the config file parsing logic. Differential Revision: https://phab.mercurial-scm.org/D4056

tests: avoid unnecessarily looking inside .hg/ in test-narrow* Many of the narrow tests were looking at .hg/narrowspec. I think most of them were written before we added `hg tracked`, but now that we have that command, we should fix the tests. There were also a few instances of tests looking at .hg/dirstate. I fixed those to use `hg files` instead. Differential Revision: https://phab.mercurial-scm.org/D4093

narrow: extract part of narrowspec backup to core Just a little preparation for the next patch. Differential Revision: https://phab.mercurial-scm.org/D4095

narrow: move requirement constant from changegroup to repository As suggested by Gregory Szorc. Differential Revision: https://phab.mercurial-scm.org/D4094

merge with stable

fileset: optimize 'x and not y' to 'x - y' 'x - y' is first rewritten to 'x and not y' so that x and y are reordered by weight.

fileset: reorder 'and' expression to evaluate basic patterns first Timing of a crafted example (when disk cache is warm): $ hg files set:'binary() and path:contrib' (orig) time: real 0.140 secs (user 0.120+0.000 sys 0.020+0.000) (new) time: real 0.040 secs (user 0.030+0.000 sys 0.010+0.000)

fileset: roughly adjust weights of functions ... in order to move status predicates far away from basic patterns. I don't know if each weight is appropriate, but it should be good enough as a start.

fileset: add stub for weight-based optimization The main purpose of this change is to group basic patterns, which can be mapped to a plain matcher. I'm not so interested in a weight of each function.

fileset: reject 'negate' node early while transforming parsed tree That's how a 'negate' node is processed in revset.

fileset: drop 'group' node from tree to be evaluated This helps rewriting a parsed tree.

fileset: add phase to transform parsed tree This isn't strictly necessary, but I decided to just follow the strategy of the revset parsing.

py3: whitelist another 3 tests caught by the ratchet Differential Revision: https://phab.mercurial-scm.org/D4069