linelog: add replacelines_vec for fastannotate # no-check-commit because we're conforming to an existing interface Differential Revision: https://phab.mercurial-scm.org/D3993

absorb: drop wrapper around the amend command We can reinstate this later if we want. Differential Revision: https://phab.mercurial-scm.org/D3992

absorb: note some TODOs from the code review Differential Revision: https://phab.mercurial-scm.org/D4047

absorb: use ui.debug() instead of open-coding it Differential Revision: https://phab.mercurial-scm.org/D4046

absorb: use pycompat to get xrange Differential Revision: https://phab.mercurial-scm.org/D4045

absorb: use set literal to avoid intermediate list Differential Revision: https://phab.mercurial-scm.org/D4044

absorb: avoid mutable default arg Differential Revision: https://phab.mercurial-scm.org/D4043

absorb: import extension from Facebook's hg-experimental absorb is a wicked-fast command to use blame information to automatically amend edits to the correct draft revision. Originally written by Jun Wu, this import is hgext3rd/absorb/__init__.py with: * the `testedwith` value changed * the linelog import updated * some missing configitems registered * some imports reordered per check-code.py * some missing __future__ imports added per check-code.py Differential Revision: https://phab.mercurial-scm.org/D3991

resolve: organize 'if confirm' conditionals Differential Revision: https://phab.mercurial-scm.org/D4123

index: pass only nodetree to nt_new() The function now only depends on the nodetree, not the index. Differential Revision: https://phab.mercurial-scm.org/D4111

index: drop now-redundant "nt" prefix of fields in nodetree struct Differential Revision: https://phab.mercurial-scm.org/D4110

index: move more fields onto nodetree type The fields moves are the ones that are not related to how the nodetree is used in the index and that will make sense for the new nodetree instance for a subset of the index that I'll add later. Differential Revision: https://phab.mercurial-scm.org/D4109

index: extract a type for the nodetree This is a first step towards exposing the nodetree as a Python type. Differential Revision: https://phab.mercurial-scm.org/D4108

index: make "nt_*" functions work on an initialized nodetree I want to be able to reuse these functions with another nodetree instance later (for disambiguating node prefix within a revset). That other nodetree instance won't want to be fully populated from the index, so this commit moves that part to the callers. Differential Revision: https://phab.mercurial-scm.org/D4107

changegroup: inline _packellipsischangegroup It now does nothing special. The logic is simple enough to inline in the 2 callers in narrow that need it. The changegroup generation APIs could probably be a bit simpler. But that's for another time. Differential Revision: https://phab.mercurial-scm.org/D4092

changegroup: move fullnodes into cgpacker And with this change, the narrow packer no longer defines any addition attributes on packer instances! Differential Revision: https://phab.mercurial-scm.org/D4091

changegroup: specify ellipses mode explicitly Currently, code throughout changegroup relies on the presence of self._full_nodes to enable ellipses mode. This is a very tenuous check. And the check may be wrong once we move _full_nodes into cgpacker. Let's capture the enabling of ellipses mode explicitly as a constructor argument and as an instance variable. We could probably derive ellipses mode by presence of other variables. But for now, this explicit approach seems simplest since it is most compatible with existing code. Differential Revision: https://phab.mercurial-scm.org/D4090

changegroup: pass ellipsis roots into cgpacker constructor And rename the internal variable to conform with naming conventions. Differential Revision: https://phab.mercurial-scm.org/D4089

changegroup: move revision maps to cgpacker And remove the underscores so the variables conform to our naming convention. The logic in _close() should be the only thing warranting scrutiny during review. Differential Revision: https://phab.mercurial-scm.org/D4088

changegroup: move changelogdone into cgpacker Looking at what it is used for, it feels like there is a better way to implement all this. So recording a TODO to track that. Differential Revision: https://phab.mercurial-scm.org/D4087

changegroup: declare shallow flag in constructor Thus begins the process of better formalizing ellipses and shallow changegroup generation mode so it is tracked by cgpacker at construction time instead of bolted on after the fact by a wrapper function. Differential Revision: https://phab.mercurial-scm.org/D4086

changegroup: make some packer attributes private These methods and attributes are low level and should not be called or outside outside of instances. Indicate as such through naming. Differential Revision: https://phab.mercurial-scm.org/D4085

changegroup: rename cg1packer to cgpacker There is now only a single class. We don't need to encode the version in its name since the version is a lie. Differential Revision: https://phab.mercurial-scm.org/D4084

changegroup: control delta parent behavior via constructor The last remaining override on cg2packer related to parent delta computation. We pass a parameter to the constructor to control whether to delta against the previous revision and we inline all parent delta logic into a single function. With this change, cg2packer is empty, so it has been deleted. Differential Revision: https://phab.mercurial-scm.org/D4083

changegroup: control reordering via constructor argument cg2packer.__init__ exists just to override self._reorder. Let's parameterize this behavior via an argument to the parent's __init__. The logic for self._reorder is kinda wonky. None is used as a special value and the value should be None in some situations. It is probably worth rewriting this logic to make behavior more explicit. This will likely happen as part of future work to control the delta generation process that I have planned. Differential Revision: https://phab.mercurial-scm.org/D4082

changegroup: consolidate tree manifests sending into cg1packer Previously, we overrode a method to control how manifests were serialized. This method was redefined on cg3packer to send tree manifests. This commit moves the tree manifests sending variation to cg1packer and teaches the cgpacker constructor to control which version to use. After these changes, cg3packer was empty. So it has been removed. Differential Revision: https://phab.mercurial-scm.org/D4081

changegroup: pass end of manifests marker into constructor cg3 inserts a custom marker in the stream once all manifests have been transferred. This is currently abstracted out by overriding a method. Let's pass the end of manifests marker in as an argument to avoid the extra method. Differential Revision: https://phab.mercurial-scm.org/D4080

changegroup: pass function to build delta header into constructor Previously, the delta header struct format was defined on each class and each class had a separate function for building the delta header. We replace both of these with an argument to __init__ containing a callable that can format a delta header given a revisiondelta instance. Differential Revision: https://phab.mercurial-scm.org/D4079

changegroup: make delta header struct formatters actual structs Why we weren't using compiled Struct instances, I don't know. They make code simpler. In theory they are faster. Although I don't believe I was able to measure any meaningful change. That could be because this code is often dominated by compression, deltafication, and function call overhead. Differential Revision: https://phab.mercurial-scm.org/D4078

changegroup: pass version into constructor Currently, the version is an attribute on each class. Passing the argument into the constructor gets us one step closer to eliminating cg2packer and cg3packer. Differential Revision: https://phab.mercurial-scm.org/D4077

changegroup: define functions for creating changegroup packers Currently, we have 3 classes for changegroup generation. Each class handles a specific changegroup format. And each subsequent version's class inherits from the previous one. The interface for the classes is not very well defined and a lot of version-specific behavior is behind overloaded functions. This approach adds complexity and makes changegroup generation difficult to reason about. Upcoming commits will be consolidating these 3 classes so differences between changegroup versions and changegroup generation are controlled by parameters to a single constructor / type rather than by overriding class attributes via inheritance. We begin this process by building dedicated functions for creating each changegroup packer instance. Currently they just call the constructor on the appropriate class. This will soon change. Differential Revision: https://phab.mercurial-scm.org/D4076

changegroup: capture revision delta in a data structure The current changegroup generation code is tightly coupled to the revlog API. This tight coupling makes it difficult to implement alternate storage backends without requiring a large surface area of the revlog API to be exposed. This is not desirable. In order to support changegroup generation with non-revlog storage, we'll need to abstract the concept of delta generation. This commit is the first step down that road. We introduce a data structure for representing a delta in a changegroup. The API still leaves a lot to be desired. But at least we now have separation between data and actions performed on it. As part of this, we tweak behavior slightly: we no longer concatenate the delta prefix with the metadata header. Instead, we track and emit the prefix as a separate chunk. This shouldn't have any meaningful impact since all the chunks just get sent to the wire, the compressor, etc. Because we're introducing a new object, this does add some overhead to changegroup execution. `hg perfchangegroupchangelog` on my clone of the Mercurial repo (~40,000 visible revisions in the changelog) slows down a bit: ! wall 1.268600 comb 1.270000 user 1.270000 sys 0.000000 (best of 8) ! wall 1.419479 comb 1.410000 user 1.410000 sys 0.000000 (best of 8) With for `hg bundle -t none-v2 -a /dev/null`: before: real 6.610 secs (user 6.460+0.000 sys 0.140+0.000) after: real 7.210 secs (user 7.060+0.000 sys 0.140+0.000) I plan to claw back this regression in future commits. And I may even do away with this data structure once the refactor is complete. For now, it makes things easier to comprehend. Differential Revision: https://phab.mercurial-scm.org/D4075

changegroup: inline ellipsisdata() There's only one caller of it. I don't think it needs to exist as a standalone function. Differential Revision: https://phab.mercurial-scm.org/D4074

changegroup: rename "revlog" variables "revlog" shadows the module import. But more importantly, changegroup generation should be storage agnostic and not assume the existence of revlogs. Let's rename the thing providing revision storage to "store" to reflect this ideal property. Differential Revision: https://phab.mercurial-scm.org/D4073

changegroup: move generate() modifications from narrow Narrow had a custom version of generate() that was essentially a copy of generate() with inline additions to facilitate ellipses serving. This commit inlines those modifications into generate(). Differential Revision: https://phab.mercurial-scm.org/D4067

changegroup: move generatefiles() from narrow The code is a bit ugly in that it overrides the linknodes function that is passed in as a function. I'd like to think that the caller of generatefiles() would pass in the appropriate function. We can clean this up later. Differential Revision: https://phab.mercurial-scm.org/D4066

changegroup: move _sortgroup() from narrow Differential Revision: https://phab.mercurial-scm.org/D4065

changegroup: move close() from narrow More of the same. Differential Revision: https://phab.mercurial-scm.org/D4064

changegroup: move revchunk() from narrow The monkeypatched revchunk for ellipses serving is a completely independent implementation. We model it as such in the changegroup code. revchunk() is now a simple proxy function. Again, I wish we had better APIs here. Especially since this narrow code is part of cg1packer and cg1packer can't be used with narrow. Class inheritance is wonky. And I will definitely be making changes to changegroup code for delta generation. As part of the code move, `node.nullrev` was replaced by `nullrev`. And a reference to `orig` was replaced to call `self._revchunknormal` directly. Differential Revision: https://phab.mercurial-scm.org/D4063

changegroup: move deltaparent() from narrow I'm not keen on performing the attribute sniff to test for presence of ellipses mode: I'd rather we use a separate packer instance that was ellipses mode specific. But I've tried to formalize a better API without narrow in core and I can't make sense of all the monkeypatching. My goal is to inline as much of the monkeypatching as possible then refactor the changegroup generation API. We add this code to the cg2packer because narrow doesn't work with cg1. Differential Revision: https://phab.mercurial-scm.org/D4062

changegroup: move _packellipsischangegroup() from narrow The behavior here is not ideal, as the function constructs a packer then adds attributes to it. This will be cleaned up in subsequent commits. Moving this code is necessary to move the remainder of the bundle2-level changegroup part generation code into core. Differential Revision: https://phab.mercurial-scm.org/D4061

changegroup: move ellipsisdata() from narrow This is a pretty straightforward copy of the function. Differential Revision: https://phab.mercurial-scm.org/D4060

fileset: narrow status computation by left-hand-side of 'and' node Timing with warm disk cache: $ hg status --cwd mozilla-central 'set:path:build/ and unknown()' --time (orig) time: real 1.970 secs (user 1.560+0.000 sys 0.410+0.000) (new) time: real 0.330 secs (user 0.310+0.000 sys 0.020+0.000)

fileset: move copy constructor of matchctx near __init__

fileset: build status according to 'withstatus' hint _switchcallers is no longer needed since 'withstatus' node is reinserted for arguments of functions like revs(). New matchctx instance is created per 'withstatus' to make sure that status tuple is available only for children of the 'withstatus' node.

fileset: insert hints where status should be computed This will allow us to compute status against a narrowed set of files. For example, "path:build/ & (unknown() + missing())" is rewritten as "path:build/ & <withstatus>(unknown() + missing(), 'unknown missing')", and the status call can be narrowed by the left-hand-side matcher, "path:build/". mctx.buildstatus() calls will be solely processed by getmatchwithstatus().

fileset: move buildstatus() to matchctx method In future patches, file status will be computed while evaluating a parsed tree. This patch provides a matchctx interface to build status.

fileset: keep basectx by matchctx

fileset: pass in basectx to _buildstatus() I'll make matchctx remember both ctx and basectx so that file status between them can be computed later. This prepares for the change.

resolve: update commands.resolve.confirm help text Included --mark and --unmark in the help text of resolve.confirm.config. Differential Revision: https://phab.mercurial-scm.org/D4103

resolve: support confirm config option with --unmark flag Now, commands.resolve.confirm also respect --unmark option; and confirm to unresolve all resolved files. It will confirm only when no files pats are passed (same as --mark), because when no pats are passed the default is to mark resolved files as unresolved. And if user has passed file pats then I think there is no need to confirm for that. Differential Revision: https://phab.mercurial-scm.org/D4102

resolve: correct behavior of mark-check=none to match docs Differential Revision: https://phab.mercurial-scm.org/D4121

narrow: move .hg/narrowspec to .hg/store/narrowspec (BC) The narrowspec is more closely related to the store than to the working copy. For example, if the narrowspec changes, the set of revlogs also needs to change (the working copy may change, but that depends on which commit is checked out). Also, when using the share extension, the narrowspec needs to be shared along with the store. This patch therefore moves the narrowspec into the store/ directory. This is clearly a breaking change, but I haven't bothered trying to fall back to reading the narrowspec from the old location (.hg/), because there are very few users of narrow out there. (We'll add a temporary hack to our Google-internal extension to handle the migration.) Differential Revision: https://phab.mercurial-scm.org/D4099

narrow: drop checkambig=True when restoring backup IIUC, checkambig is about updating timestamps of the file while renaming. That's important for the dirstate, but we never check the timestamp of the narrowspec file. We can therefore avoid checking passing checkambig=True. Differential Revision: https://phab.mercurial-scm.org/D4098

narrow: remove a repo file-cache invalidation It's unclear why this was needed. All tests pass without it. I asked Kyle Lippincott (who added the check) and he also doesn't remember what it was for. Differential Revision: https://phab.mercurial-scm.org/D4097

narrow: call narrowspec.{save,restore,clear}backup directly I want to move .hg/narrowspec to .hg/store/narrowspec and we need to decouple the narrowspec update from the dirstate update for that. This patch lets the callers call the narrowspec backup functions directly, in addition to the dirstate backup functions. The narrowspec methods are made to check if narrowing is enabled. For that, a repo instance was needed, which all the callers luckily already had available. Differential Revision: https://phab.mercurial-scm.org/D4096

index: don't add 1 to length variables A lot of "+ 1" and "-1" were mechanically added to ease the transition in 781b2720d2ac (index: don't include nullid in len(), 2018-07-20). Let's clean it up now. Differential Revision: https://phab.mercurial-scm.org/D4106

index: drop support for nullid at position len(index) in index_node I think no callers exist since at least a3dacabd476b (index: don't allow index[len(index)] to mean nullid, 2018-07-20). Differential Revision: https://phab.mercurial-scm.org/D4105

index: return False for "len(index) in index" Since we no longer accept index[len(index)], we should clearly make "len(index) in index" return False. This should have been part of a3dacabd476b (index: don't allow index[len(index)] to mean nullid, 2018-07-20) Differential Revision: https://phab.mercurial-scm.org/D4104

fileset: combine union of basic patterns into single matcher This appears to improve query performance in a big repository than I thought. Writing less Python in a hot loop, faster computation we gain. $ hg files --cwd mozilla-central --time 'set:a* + b* + c* + d* + e*' (orig) time: real 0.670 secs (user 0.640+0.000 sys 0.030+0.000) (new) time: real 0.210 secs (user 0.180+0.000 sys 0.020+0.000)

fileset: reorder 'or' expression by weight

fileset: introduce weight constants for readability These constants are defined in the filesetlang module since it's the bottommost module depending on WEIGHT_CHECK_FILENAME, and extensions will be likely to import it to process function arguments. Credit for the naming goes to Augie Fackler.

sparse: use named parameters in i18n strings This should give more hints about what the %s means, and allow reordering.

sparse: do not include operator in i18n strings

fix: compute changed lines lazily to make whole-file fixer tools faster Differential Revision: https://phab.mercurial-scm.org/D4100

resolve: support commands.resolve.confirm option with --mark flag Now, commands.resolve.confirm config option also respect --mark; and confirm to mark all 'unresolved' files as 'resolved'. It will confirm only when you don't pass any pats i.e 'hg resolve -m', because when no file pats are passed then --mark's default functionality is to mark all unresolved files. And if user has given file pats then I think there is no need to confirm. Differential Revision: https://phab.mercurial-scm.org/D4101

resolve: add commands.resolve.mark-check=abort to tweakdefaults Differential Revision: https://phab.mercurial-scm.org/D4072

resolve: graduate resolve.mark-check from experimental, add docs Since this hasn't been in a release yet, I'm not bothering to add an alias for the experimental name of the config variable. Differential Revision: https://phab.mercurial-scm.org/D4071

shortest: use 'x' prefix to disambiguate from revnum if configured Differential Revision: https://phab.mercurial-scm.org/D4042

revisions: allow "x123" to refer to nodeid prefix "123" When resolving "123" to a revision, we try to interpret it as revnum before we try to interpret it as a nodeid hex prefix. This can lead to the shortest valid prefix being longer than necessary. This patch lets us write such nodeids in a shorter form by prefixing them with "x" instead of adding more hex digits until they're longer than the longest decimal revnum. On my hg repo with almost 69k revisions, turning this feature on saves on average 0.4% on the average nodeid length. That clearly doesn't justify this patch. However, it becomes more usefule when combined with the earlier patches in this series that let you disambiguate nodeid prefixes within a configured revset. Note that we attempt to resolve symbols as nodeid prefixes after we've exhausted all other posibilities, so this is a backwards compatible change (only queries that would previously fail may now succeed). I've still hidden this feature behind an experiemntal config option so we can roll it back if needed. Differential Revision: https://phab.mercurial-scm.org/D4041

shortest: make isrev() a top-level function I'm going to add another caller in the next patch. Differential Revision: https://phab.mercurial-scm.org/D4040

shortest: cache disambiguation revset This makes it actually useful. In compared the time in my hg repo with 69.6k revisions and with a disambiguation revset of "not public()" that matches 563 visible revisions. I ran "time hg log -T '{shortest(node1,)}' -r 0:1000" (no revisions within the revset in that revision range). Before this patch, it took 57s and after it took 0.7s. Differential Revision: https://phab.mercurial-scm.org/D4039

index: don't allow index[len(index)] to mean nullid Now everything else has been cleaned up and we can drop support for getting the nullid from the end of the index (from *past* the end actually, since we reduced the length in the previous patch). Differential Revision: https://phab.mercurial-scm.org/D4023

index: don't include nullid in len() I suspect the reason the nullid is in the index in the last position is that it lets index[i] for regular revision number, even when index was just a regular Python list. An alternative solution would have been to reserve revision number 0 for the null revision. I don't know why that wasn't done. Now that we have classes backing the index, we can easily make index[-1] get the nullid without having to put it last in the list and including it in the len(). This patch just hides the nullid -- it will still be accessible at index[len(index)]. I realize that this will be annoying when checking out across this commit for debugging (including bisection). Differential Revision: https://phab.mercurial-scm.org/D4022

index: replace insert(-1, e) method by append(e) method I want to make index[len(index) - 1] be the tip revision, not null revision as it is today. insert(-1, e) will then make little sense. Since insert() currently requires the first argument to be -1, it seems simpler to replace it by a method that allows insertion only at the end. Note that revlogoldindex already has this method (by virtue of extending list). Differential Revision: https://phab.mercurial-scm.org/D4021

pure: create type for revlog v0 index The new type takes care of hiding the nullid entry. This type helps us transition away from including that entry at index[len(index)-1]. Differential Revision: https://phab.mercurial-scm.org/D4020

index: drop support for negative indexes into the index I want index[i] to work for any valid revnum including -1 (as it already does), and I also want len(index) to be the number of revisions in the index (not counting the null revision), so it cannot also support negative revision numbers other than -1 for nullid. I didn't bother removing support for it from revlog v0. Differential Revision: https://phab.mercurial-scm.org/D4019

index: handle index[-1] as nullid more explicitly I find it more intuitive to check if "pos == -1" than to first add the index length (which includes one extra item for the nullid) and compare that to "length - 1". However, because test-parseindex2.py compares the whole index (up to len(index)-1), we need to also preserve that other check for a little while more. I'll remove it soon. Differential Revision: https://phab.mercurial-scm.org/D4018

index: store nullrev as -1 in nodetree Nothing important, it just seems more natural to not map nullrev to INT_MAX. We just need to change the revision encoding a little to make space for the -1. Differential Revision: https://phab.mercurial-scm.org/D4005

index: create function for deleting node from nodetree A node is marked deleted by nt_insert(self, node, -1), but "-1" could mean so many things (notably nullrev), so let's hide it in a function. The function could possibly also remove nodes from the tree in the future (it currently only marks them deleted). Differential Revision: https://phab.mercurial-scm.org/D4004

revlog: remove some knowledge of sentinel nullid in index I think the "-2" to mean "last position in index, not counting the null revision at the end" is an implementation detail of the index that we should avoid spreading knowledge of. I hope we can even remove support for index[-2]. Differential Revision: https://phab.mercurial-scm.org/D4016

scmutil: make shortest() respect disambiguation revset The previous patch would let you use a shorter prefix if the prefix is unique within a configured revset. However, that's not very useful if there's no simple way of knowing what that shorter prefix is. This patch adapts the shortest() template function to use the shorter prefixes for nodes in the configured revset. This is currently extremely slow, because it calculates the revset for each call to shortest(). To make this faster, the next patch will start caching the revset instance. Ideally we'd cache a prefix tree instance instead. Differential Revision: https://phab.mercurial-scm.org/D4038

lookup: add option to disambiguate prefix within revset When resolving a nodeid prefix that is not unique within the repo and the user has configured a revset that they want to disambiguate within, we now try to look up within that revset before we fail. If there is a unique match within the revset, we use that. This is of course most effective at allowing a short prefix if the revset contains few nodes. For most of our internal users at Google, "not public()" is sufficiently small that a hex digit or two is enough. The implementation is currently pretty slow, but good enough for small revsets (which is the expected use case). The scan in the revset is linear. We may want to use a prefix tree if we want to allow users to use a larger revset. Credit for the idea goes to Kyle Lippincott. Differential Revision: https://phab.mercurial-scm.org/D4037

revlog: use specialized exception for ambiguous prefix lookup It's useful to be able to catch a specific exception for this case. We'll use it soon. Differential Revision: https://phab.mercurial-scm.org/D4036

narrowspec: remove the unused _parsestoredpatterns() function Previous patch started using sparse.parseconfig() to read narrowspec and now we don't need this function. Differential Revision: https://phab.mercurial-scm.org/D4058

narrowspec: use sparse.parseconfig() to parse narrowspec file (BC) This also make narrow files use 'include' and 'exclude' instead of plural forms which are 'includes' and 'excludes'. This is BC because existing narrowspecs have to replace excludes, includes words with exclude and include i.e. their singular versions. Differential Revision: https://phab.mercurial-scm.org/D4057

sparse: add an action argument to parseconfig() This will help us in reusing this function to parse narrow config files and unfiying the config file parsing logic. Differential Revision: https://phab.mercurial-scm.org/D4056

tests: avoid unnecessarily looking inside .hg/ in test-narrow* Many of the narrow tests were looking at .hg/narrowspec. I think most of them were written before we added `hg tracked`, but now that we have that command, we should fix the tests. There were also a few instances of tests looking at .hg/dirstate. I fixed those to use `hg files` instead. Differential Revision: https://phab.mercurial-scm.org/D4093

narrow: extract part of narrowspec backup to core Just a little preparation for the next patch. Differential Revision: https://phab.mercurial-scm.org/D4095

narrow: move requirement constant from changegroup to repository As suggested by Gregory Szorc. Differential Revision: https://phab.mercurial-scm.org/D4094

merge with stable

tests: update test expectations in pre-2.7.9 branch of this test As far as I can tell this is the only spot that got overlooked. Rough test methodology (run inside an hg working copy): docker run --rm -v `pwd`:/hg -it debian:wheezy bash apt-get update apt-get install python python-dev build-essential unzip mercurial \ wget libbz2-dev make testpy-2.7.8 You could try and use the system Python, but it's 2.7.3 and has lots of issues for hg these days that are not worth fixing. Differential Revision: https://phab.mercurial-scm.org/D4070

fileset: optimize 'x and not y' to 'x - y' 'x - y' is first rewritten to 'x and not y' so that x and y are reordered by weight.

fileset: reorder 'and' expression to evaluate basic patterns first Timing of a crafted example (when disk cache is warm): $ hg files set:'binary() and path:contrib' (orig) time: real 0.140 secs (user 0.120+0.000 sys 0.020+0.000) (new) time: real 0.040 secs (user 0.030+0.000 sys 0.010+0.000)

fileset: roughly adjust weights of functions ... in order to move status predicates far away from basic patterns. I don't know if each weight is appropriate, but it should be good enough as a start.

fileset: add stub for weight-based optimization The main purpose of this change is to group basic patterns, which can be mapped to a plain matcher. I'm not so interested in a weight of each function.

fileset: reject 'negate' node early while transforming parsed tree That's how a 'negate' node is processed in revset.

fileset: drop 'group' node from tree to be evaluated This helps rewriting a parsed tree.

fileset: add phase to transform parsed tree This isn't strictly necessary, but I decided to just follow the strategy of the revset parsing.

py3: whitelist another 3 tests caught by the ratchet Differential Revision: https://phab.mercurial-scm.org/D4069

testrunner: allow multiple #testcases This lets you have multiple #testcases declarations and they're taken to be different dimensions of the test, so their cross product becomes the total set of test cases. For example: #testcases obsstore-on obsstore-off #testcases manifest-flat manifest-tree $ hg rebase ... ... #if obsstore-on $ hg log ... #endif Note that this is an excellent way to slow down the test suite, so use it with care. The feature is implemented by replacing most of the "case" variables that were strings before by an array of strings with each item a different dimension of the test case. The file names are created by joining the dimensions by "#" (e.g. test-foo.t#obsstore-on#manifest-flat). Differential Revision: https://phab.mercurial-scm.org/D4049

testrunner: use "#" for "test cases" suffix in .err filename too This seems like a natural follow-up to b865bba56db1 (run-tests: update the test case name format, 2018-05-13). Differential Revision: https://phab.mercurial-scm.org/D4052

resolve: add confirm config option This config setting gives a functionality to confirm before it re-merge all unresolved files. If this config is enabled, when you run 'hg resolve --all' it will prompt with a msg "re-merge all unresolved files (yn)?" To enable this functionality: [commands] resolve.confirm = True Differential Revision: https://phab.mercurial-scm.org/D3988

revlog: remove side effect from failed nt_init() If nt_init() successfully allocates memory for the node tree but then runs out of memory while trying to insert nullid into it, it will leave the node tree pointer set on the index object. That means that future node tree operations will think that the tree is properly initialized. It seems very unlikely to make a difference in practice (if nt_init() runs out of memory, not much else will probably work), but since I spotted it, I figured I might as well fix it. Differential Revision: https://phab.mercurial-scm.org/D4028

revlog: remove micro-optimization for looking up only nullid index_find_node() would call nt_find() before initializing the node tree. nt_find() would then return -2 unless the requested revision was the null revision. I can't imagine what scenario that is optimizing for, and doing the initialization earlier makes the code simpler and easier to follow, so that's what this patch does. Differential Revision: https://phab.mercurial-scm.org/D4027

revlog: remove unnecessary output parameter from node_check() The "nodelen" output parameter is always set to 20 if the function returns successfully. Differential Revision: https://phab.mercurial-scm.org/D4026

narrow: move status-filtering to core and to ctx One of my recent changes from repo.status(ctx1, ctx2) to ctx1.status(ctx2) broke some of our Google-internal tests. The problem turned out to be that the narrow extension was overriding repo.status() to make it filter out paths outside the narrowspec. When I changed to ctx1.status(ctx2), then that filtering obviously got lost. ctx.status() seems like a better method to do the filtering in, so this patch moves the filtering into that method, thereby also moving it out of the extension and into core. Differential Revision: https://phab.mercurial-scm.org/D4068

amend: support "history-editing-backup" config option Now, amend is in the list of those history editing commands which support `history-editing-backup` config option. If you don't want to store any backup then just use this config. [ui] hisotry-editing-backup = False Current status of list of history editing commands which support this config: 1. histedit 2. rebase 3. amend Differential Revision: https://phab.mercurial-scm.org/D3968

rebase: move "backup" flag to rebaseruntime It was getting messy to populate "backup" flag to calls of `_finishrebase()` and `_prepareabortorcontinue`, so made some changes to move "backup" flag to rbsrt. Differential Revision: https://phab.mercurial-scm.org/D4055

i18n: format warning of hggettext in standard compiler error style Now, hggettext specific warning messages are formatted in: FILENAME:LINENO:MESSAGE This allows editors to jump into corresponded line easily.

i18n: omit redundant translatable synopsis text to avoid xgettext warning This empty translatable synopsis text causes xgettext warning below: Empty msgid. It is reserved by GNU gettext: gettext("") returns the header entry with meta information, not the empty string. This "synopsis" argument of @command annotation is omitable.

i18n: avoid substitution of PYFILES at runtime for readability of output This substitution decreases readability of "make update-pot" output, because PYFILES consists of many files. This patch makes "make update-pot" show "find mercurial hgext doc -name '*.py'" instead of many *.py files at runtime.

util: make new timedcmstats class Python 3 compatible

fix: add a monkey-patchable point after all new revisions have been committed Differential Revision: https://phab.mercurial-scm.org/D4048

help: add quotes to a few commands we point to I didn't know that 'hg help "revsets.x or y"' was valid syntax, so the quoting is extra useful here to make it clear that that is an actual command. Differential Revision: https://phab.mercurial-scm.org/D4059

linelog: port to Python 3 Differential Revision: https://phab.mercurial-scm.org/D4051

exchange: move simple narrow changegroup generation from extension The narrow extension completely replaced the function generating the changegroup part when a narrow changegroup was requested. Previous commits have taught the in-core changegroup code how to filter files based on a matcher. This commit teaches the in-core bundle2 part generation code to construct a matcher based on arguments. It will also emit a bundle2 part describing the narrow spec. I believe the changegroup part generation code in the narrow extension is now limited to ellipsis serving mode. i.e. core is now capable of narrow changegroup generation when ellipsis mode is disabled. Differential Revision: https://phab.mercurial-scm.org/D4014

exchange: make narrow ACL presence imply narrow=True And refactor the logic for determining when to invoke our custom changegroup part function so it is only conditional on narrow being set. This makes it more obvious under which conditions we should invoke the custom implementation. Also use raw strings so **kwargs works on Python 3. Differential Revision: https://phab.mercurial-scm.org/D4013

changegroup: inline prune() logic from narrow prune() needs to ellide manifests that aren't part of the narrow matcher. The code is violating storage abstractions, so a comment has been added. Keep in mind the impetus for moving this code to core is so changegroup code can be refactored to be storage agnostic. Differential Revision: https://phab.mercurial-scm.org/D4012

fileset: extract language processing part to new module (API) I'll add a couple more functions that work on parsed tree. % wc -l mercurial/fileset*.py 559 mercurial/fileset.py 135 mercurial/filesetlang.py 694 total

fileset: flatten 'or' nodes to unnest unionmatchers This also makes it easier to compile a union of basic patterns into a single regexp pattern.

fileset: flatten arguments list Just prepares for flattening 'or' nodes. This change would have no impact on performance.

debugfileset: add option to show matcher representation

debugfileset: backport --show-stage option from debugrevspec I'll add some static optimizations to fileset.

lfs: remove callstatus property from 'lfs()' fileset It was added at 91aac8e6604d, but is no longer needed since a fileset expression is now compiled into an "open" matcher. See ff5b6fca1082 for details.

rebase: support "history-editing-backup" config option If you don't want to store any backup while rebasing, you can use `history-editing-backup` config option. [ui] history-editing-backup = # True or False Current status of list of commands which supports this config: 1. histedit 2. rebase Differential Revision: https://phab.mercurial-scm.org/D3887

extensions: add detailed loading information This lets you track down what exactly is happening during extension loading, and how long various steps took.

util: create a context manager to handle timing The context manager is pulled out of the timed decorator function, and refactored to provide a stats instance, with added tests.

linelog: fix import statements to pass the import checker on py3 Differential Revision: https://phab.mercurial-scm.org/D4050

linelog: add a Python implementation of the linelog datastructure This datastructure was originally developed by Jun Wu at Facebook, inspired by SCCS weaves. It's useful as a cache for blame information, but also is the magic that makes `hg absorb` easy to implement. In service of importing the code to Mercurial, I wanted to actually /understand/ it, and once I did I decided to take a run at implementing it. The help/internals/linelog.txt document is the README from Jun Wu's implementaiton. It all applies to our linelog implementation. Differential Revision: https://phab.mercurial-scm.org/D3990

changegroup: move file matcher from narrow extension Sparse changegroup generation requires the use of a matcher to filter which files are relevant. This commit moves the file matcher from the narrow extension to core and updates the narrow extension to use it. I'm not sure why the narrow extension was storing the matcher as a callable that resolved to a matcher. So I changed it to be a simple matcher instance. In addition, code from narrow to intersect the matcher with the local narrow spec is now performed automatically when the changegroup packer is created. If a matcher is not passed into getbundler() an alwaysmatcher() is assumed. This ensures that a matcher is always defined for all operations. Differential Revision: https://phab.mercurial-scm.org/D4011

resolve: add option to warn/abort on -m with unresolved conflict markers When a user is dropped out of Mercurial to a terminal to resolve files, we emit messages like: conflicts while merging file1! (edit, then use 'hg resolve --mark') conflicts while merging file2! (edit, then use 'hg resolve --mark') We don't mention a file name in the hint, so some users might do something like `$EDITOR file1; hg resolve --mark`, see that it says "(no more unresolved files)" and forget to deal with file2 before running the next command. Even if we did mention a file name in the hint, it's too easy to forget it (maybe the merge spans a couple days or something). This option lets us inform the user that they might have missed something. In the scenario above, the output would be something like: warning: the following files still have conflict markers: file2 (no more unresolved files) Differential Revision: https://phab.mercurial-scm.org/D4035

exchange: refactor control flow of _getbundlechangegrouppart() The use of early return makes the control flow of this function much easier to reason about IMO. Differential Revision: https://phab.mercurial-scm.org/D4010

exchange: move _computeellipsis() from narrow This is also referenced as part of the narrow changegroup code and therefore needs to move to core before we can integrate the narrow changegroup code into core. Differential Revision: https://phab.mercurial-scm.org/D4009

exchange: move narrow acl functionality into core This function is called by the custom changegroup generation code in the narrow extension. I want to move that changegroup code into core. That means we need to move this function. The code is kinda hacky in that assumes existence of REMOTE_USER, which is only present on authenticated HTTP requests. I've added a comment indicating that. Differential Revision: https://phab.mercurial-scm.org/D4008

exchange: move disabling of rev-branch-cache bundle part out of narrow I'm attempting to refactor changegroup code in order to better support alternate storage backends. The narrow extension is performing a lot of monkeypatching to this code and it is making it difficult to reason about how everything works. I'm reasonably certain I would be unable to abstract storage without requiring extensive rework of narrow. I believe it is less effort to move narrow code into core so it can be accounted for when changegroup code is refactored. So I'll be doing that. The first part of this is integrating the disabling of the cache:rev-branch-cache bundle2 part into core. This doesn't seem like it is related to changegroup, but narrow's modifications to changegroup are invasive and also require taking its code for bundle generation and exchange into core in order for the changegroup code to work. Differential Revision: https://phab.mercurial-scm.org/D4007

dispatch: show a short error message when invalid global option given Similar reasoning as the previous patch. Differential Revision: https://phab.mercurial-scm.org/D4025

dispatch: don't show list of commands on bogus command If a command is ambiguous, you get this: $ hg ve hg: command 've' is ambiguous: verify version [255] If you typo a command, you get this: $ hg comit hg: unknown command 'comit' (did you mean one of commit, incoming, mycommit?) [255] But if you completely mistype a command so it no longer looks like any existing commands, you get a full list of commands. That might be useful the first time you use Mercurial, but after that it's probably more annoying than help, especially if you have the pager enabled and have a short terminal. Let's instead give a short hint telling the user to run `hg help` for more help. Differential Revision: https://phab.mercurial-scm.org/D4024

histedit: avoid repeating name of state file in a few places We can depend on the state object instead. Differential Revision: https://phab.mercurial-scm.org/D4006

py3: stop rewriting xrange() to pycompat.xrange() We now require the use of pycompat.xrange() in source. One less feature in the module importer gets us one step closer to Python 3 native source code. Differential Revision: https://phab.mercurial-scm.org/D4034

check-code: ban use of bare xrange() We want everyone to use pycompat.xrange(). Differential Revision: https://phab.mercurial-scm.org/D4033

global: use pycompat.xrange() On Python 3, our module importer automatically rewrites xrange() to pycompat.xrange(). We want to move away from the custom importer on Python 3. This commit converts all instances of xrange() to use pycompat.xrange(). Differential Revision: https://phab.mercurial-scm.org/D4032

pycompat: add xrange alias for Python 2 We assign xrange on Python 3 but not Python 2. Adding the missing symbol on Python 2 allows us to use `pycompat.xrange` to get a generator range function. Differential Revision: https://phab.mercurial-scm.org/D4031

manifest: persist the manifestfulltext cache Reconstructing the manifest from the revlog takes time, so much so that there already is a LRU cache to avoid having to load a manifest multiple times. This patch persists that LRU cache in the .hg/cache directory, so we can re-use this cache across hg commands. Commit benchmark (run on Macos 10.13 on a 2017-model Macbook Pro with Core i7 2.9GHz and flash drive), testing without and with patch run 5 times, baseline is r2a227782e754: * committing to an existing file, against the mozilla-central repository. Baseline real time average 1.9692, with patch 1.3786. A new debugcommand "hg debugmanifestfulltextcache" lets you inspect the cache, clear it, or add specific manifest nodeids to it. When calling repo.updatecaches(), the manifest(s) for the working copy parents are added to the cache. The hg perfmanifest command has an additional --clear-disk switch to clear this cache when testing manifest loading performance. Using this command to test performance on the firefox repository for revision f947d902ed91, whose manifest has a delta chain length of 60540, we see: $ hg perfmanifest f947d902ed91 --clear-disk ! wall 0.972253 comb 0.970000 user 0.850000 sys 0.120000 (best of 10) $ hg debugmanifestfulltextcache -a `hg log --debug -r f947d902ed91 | grep manifest | cut -d: -f3` Cache contains 1 manifest entries, in order of most to least recent: id: 0294517df4aad07c70701db43bc7ff24c3ce7dbc, size 25.6 MB Total cache data size 25.6 MB, on-disk 0 bytes $ hg perfmanifest f947d902ed91 ! wall 0.036748 comb 0.040000 user 0.020000 sys 0.020000 (best of 100) Worst-case scenario: a manifest text loaded from a single delta; in the firefox repository manifest node 9a1246ff762e is the chain base for the manifest attached to revision f947d902ed91. Loading this from a full cache file is just as fast as without the cache; the extra node ids ensure a big full cache: $ for node in 9a1246ff762e 1a1922c14a3e 54a31d11a36a 0294517df4aa; do > hgd debugmanifestfulltextcache -a $node > /dev/null > done $ hgd perfmanifest -m 9a1246ff762e ! wall 0.077513 comb 0.080000 user 0.030000 sys 0.050000 (best of 100) $ hgd perfmanifest -m 9a1246ff762e --clear-disk ! wall 0.078547 comb 0.080000 user 0.070000 sys 0.010000 (best of 100)

debug: allow specifying a manifest node rather than a revision

packaging: always execute builddeb from source root Without this, dockerdeb and the make targets that invoke it may fail depending on the pwd at the time of invocation.

perfmoonwalk: make work with filtered repo This also fixes the out-of-range "len(repo)" that weirdly works (and gets the nullid). I suspect it wasn't intentional to include the timing of reading the null revision. Differential Revision: https://phab.mercurial-scm.org/D4017

revlog: don't include trailing nullrev in revlog.revs(stop=len(revlog)) This was an odd side effect of the nullid entry that's in the index. The existing callers (mostly repair.py) seem to have handled it fine. It doesn't seem intentional, and it's pretty surprising, so let's remove that surprise. Differential Revision: https://phab.mercurial-scm.org/D4015

patchbomb: ensure all headers and values given to email mod are native strings This lets test-patch-bookmark.t only fail with some harmless header output changes on Python 3, so I think patchbomb might be basically useful on Python 3 now. Differential Revision: https://phab.mercurial-scm.org/D3952

context: delete an obsolete comment Obsolete since 91618801d5c3 (context: raise ProgrammingError on repo['my-tag'], 2018-07-06). Differential Revision: https://phab.mercurial-scm.org/D4002

templatekw: use ctx1.status(ctx2) instead of repo.status(ctx1, ctx2) Differential Revision: https://phab.mercurial-scm.org/D4001

patch: use ctx1.status(ctx2) instead of repo.status(ctx1, ctx2) Differential Revision: https://phab.mercurial-scm.org/D4000

fileset: use ctx1.status(ctx2) instead of repo.status(ctx1, ctx2) Differential Revision: https://phab.mercurial-scm.org/D3999

uncommit: use ctx1.status(ctx2) instead of repo.status(ctx1, ctx2) Differential Revision: https://phab.mercurial-scm.org/D3998

fix: use ctx1.status(ctx2) instead of repo.status(ctx1, ctx2) Differential Revision: https://phab.mercurial-scm.org/D3997

amend: use ctx1.status(ctx2) instead of repo.status(ctx1, ctx2) Differential Revision: https://phab.mercurial-scm.org/D3996

dispatch: making all hg abortions be output with a specific label This allows abortions to be highlighted specially and separately from warnings - for instance, red is a reasonable color for when hg aborts, but is overly dramatic for most warnings produced elsewhere. Differential Revision: https://phab.mercurial-scm.org/D3967

ui: fix implicit unicode-to-bytes conversion introduced in 9df29b7c62cf This is harmless, unless you try and run hg with HGUNICODEPEDANTRY enabled. It's technically wrong, so let's go ahead and fix it. Differential Revision: https://phab.mercurial-scm.org/D3989

merge with stable

Added signature for changeset 33ac6a72308a

Added tag 4.7 for changeset 33ac6a72308a

revlog: fix descendant deprecated method Fix the descendant deprecated method introduced earlier in this cycle. This was caught by Yuya, thank you.

ui: remove commands.status.terse=u from ui.tweakdefaults commands.status.terse=u can add significant overhead when operating on large repositories. Using the Firefox repository: HGRCPATH= hg --time status time: real 1.340 secs (user 0.960+0.000 sys 0.380+0.000) HGRCPATH= hg --time --config commands.status.terse=u status time: real 2.420 secs (user 2.070+0.000 sys 0.360+0.000) HGRCPATH= hg --time --config extensions.fsmonitor= status time: real 0.080 secs (user 0.050+0.010 sys 0.040+0.000) HGRCPATH= ~/src/hg/hg --time --config extensions.fsmonitor= --config commands.status.terse=u status time: real 2.470 secs (user 2.080+0.000 sys 0.390+0.000) The performance regression - especially when fsmonitor is being used - is too much to stomach for the 4.7 release. We've decided to remove commands.status.terse=u from ui.tweakdefaults until we can improve its performance, hopefully in the 4.8 cycle. This commit effectively backs out 6acf41bb8d40.

context: safegaurd against 'lx' being passed as file flag in manifest Subversion can have a file as executable link. When using hgsubversion, we will have both islink and isexec True. This will lead to _flags being set to `lx`. However, manifest expects flag to be one-byte so it will crash if 'lx' is passed. Also it's impossible to have an executable link. This patch will safegaurd us from having 'lx' being a possible value. This was authored by Ivan Lezhankin from Yandex. Differential Revision: https://phab.mercurial-scm.org/D3985

localrepo: unconditionally enable general delta with sparse revlogs This come as an extra security, better safe than sorry.

sparse-revlog: also use sparse-revlog config as a general delta trigger Sparse revlog rely on general delta, so we should make sure it is used.

fileset: suppress EACCES while reading arbitrary paths via filectx API On Windows, EACCES is raised in place of EISDIR. This patch simply adds EACCES to the list of errors to be ignored since I think it's okay for filesets to treat inaccessible working-copy files as not existing.

merge: do the trivial resolution after updating sparse checkout In merge, we do trivial resolution for files which were deleted on one side and changed on other. When sparse extension in involved that file might not be present in wdir and trivial resolution can lead to file not found error. This patch make sure we updates the sparse checkout before doing the trivial resolution. This fixes the test failure demonstrated in previous patch. Differential Revision: https://phab.mercurial-scm.org/D3984

sparse: add test showing `hg merge` is broken while using sparse extension This patch adds a test to demonstrate that `hg merge` is broken in some cases while using sparse extension. The case is when you have a file which is: * modified between current wdir parent and merge base * deleted between merge base and merge destination * excluded from sparse checkout Doing `hg merge` results in file not found error. Next patch will fix this. Differential Revision: https://phab.mercurial-scm.org/D3983

context: use hex nodeid in error about filtered node I went a little too far in my cleanup in 9231148ea599 (context: convert to hex for error message only for 20-byte changeid, 2018-04-06). I missed that the case where a binary nodeid refers to a filtered node. Differential Revision: https://phab.mercurial-scm.org/D3987

doc: fix underline length for config title (issue5949) This was an error in the initial commit. Thanks go to Marcel Svitalský for reporting the bug.

clone: process 'lookup' return as an arbitrary symbol In theory, checkout is expected to be a node here because it was returned by peer.lookup. In practice, multiple important extensions (like hg-git, hg-subversion) use peers not backed by a mercurial repository where lookup cannot return a node. Allowing arbitrary symbols is necessary to make these extensions working with 4.7. We should probably introduce a new API in Core to have these extensions to work without abusing the lookup API. In the meantime, a small change to restore compatibility in 4.7 seems in order.

tests: use inline Python instead of sed to add trailing whitespace The sed invocation was failing on OS X and FreeBSD. I'm far too lazy to diagnose that, so just use some inline Python to fix the build.

context: add missing b prefix This fixes mercurial/context.py:593: SyntaxError: cannot mix bytes and nonbytes literals (context.py, line 593) in Python 3.

doctest: convert matcher root to native path Otherwise it wouldn't be caught by a fast path of pathutil.canonpath(), and fall back to file identity checks.

test-fileset: make con.xml in output conditionally available

test-obsmarker-template: run mkcommit in subshell to isolate envvars I don't know if it is a feature or a bug, but it appears that environment variables passed to a shell function persist on /bin/sh.

test-obsmarker-template: add missing HGENCODING=latin-1 It just worked by accident on some Linux sh.

gitweb: add link to graph error.tmpl wasn't consistent with other templates in gitweb in that it was missing a "graph" link. This commit makes it consistent.

hgweb: strip trailing '/' in apppath before appending '/static/' (issue5943) Differential Revision: https://phab.mercurial-scm.org/D3978

config: rename `revlog` section into `storage` The idea was suggested by Gregory Szorc on IRC. It is more generic and seems better. It is probably best to rename the section before it ever makes into an official (non-rc) release. The only config option currently in this section have been prefixed with `revlog` to clarify it applies to `revlog` related storage.

Added signature for changeset e90130af47ce

Added tag 4.7rc0 for changeset e90130af47ce

configitems: restore alias for format.aggressivemergedeltas 913ca175c4ae broke BC by dropping support for reading format.aggressivemergedeltas. Let's restore it. Differential Revision: https://phab.mercurial-scm.org/D3966

worker: call selector.close() to release polling resources

release: merge default into stable for 4.7 release freeze

histedit: drop --no-backup option Dropping this option because now we have a better option than passing --no-backup flag every time, now user can set a config in hgrc: [ui] history-editing-backup = False This config aims to operate on every history editing command and it is still work in progress. As yuya suggessted it probably to late to add full support this config, so making this as an experimental config. Differential Revision: https://phab.mercurial-scm.org/D3965

aggressivemergedelta: document rename and move to `revlog` section The config does not follow our naming guideline and "Aggressive" is probably a word to keep away from users. The option does not truly fit in the `format` section. It can be turned on and off for existing repository without much consequence regarding compatibility. A new `revlog` option is created to control behavior related to revlog writing and reading. We can see multiple other config options that could be migrated there. * format.maxchainlen * experimental.mmapindexthreshold * experimental.sparse-read.density-threshold (in an updated form) * experimental.sparse-read.min-gap-size (in an updated form) In addition, we can foresee at least a couple of sparse-revlog related option coming too (to reduce delta chain length and increase snapshot reuse) These two extra options might fit there too. Unless we want to create a section dedicated to caches and performance. * format.chunkcachesize * format.manifestcachesize For now, we only migrate `optimize-delta-parent-choice` since it is getting out of experimental. It is too close to the release to move the other one. In addition, we still lack proper the prioritization of alias that would help renaming them without bad consequence for users. (Not fully happy about the `revlog` name but could not find better).

aggressivemergedeltas: rename variable internally The "aggressivemergedeltas" name is not great. First, it is quite long, second, we would rather have less "Aggressive" names within the project. We are about to rename the config option, so it seems the appropriate time to rename the internal variable.

config: document the purpose of the `format` config section The config section now have a top level documentation to clarify its intend and usage. In particular, user are now explain when the option are taken in account and how to convert repository. There are an handful of experimental options in this section that does not match its definition. They should be relocated to other section before getting out of experimental. (see next changeset for one example).

macosx: fixing macOS version generation after db9d1dd01bf0 With the Python3 change, the string is now something like version = b'4.6.2+848-88be288e8ac1' where it was previously just: version = '4.6.2+848-88be288e8ac1' Differential Revision: https://phab.mercurial-scm.org/D3964

histedit: add history-editing-backup config option Instead of passing --no-backup option every time you don't want to store backup, now you can set config option: [ui] history-editing-backup = False This option aims to operate on every history editing command. Differential Revision: https://phab.mercurial-scm.org/D3901

merge: mark file gets as not thread safe (issue5933) In default installs, this has the effect of disabling the thread-based worker on Windows when manifesting files in the working directory. My measurements have shown that with revlog-based repositories, Mercurial spends a lot of CPU time in revlog code resolving file data. This ends up incurring a lot of context switching across threads and slows down `hg update` operations when going from an empty working directory to the tip of the repo. On mozilla-unified (246,351 files) on an i7-6700K (4+4 CPUs): before: 487s wall after: 360s wall (equivalent to worker.enabled=false) cpus=2: 379s wall Even with only 2 threads, the thread pool is still slower. The introduction of the thread-based worker (02b36e860e0b) states that it resulted in a "~50%" speedup for `hg sparse --enable-profile` and `hg sparse --disable-profile`. This disagrees with my measurement above. I theorize a few reasons for this: 1) Removal of files from the working directory is I/O - not CPU - bound and should benefit from a thread pool (unless I/O is insanely fast and the GIL release is near instantaneous). So tests like `hg sparse --enable-profile` may exercise deletion throughput and aren't good benchmarks for worker tasks that are CPU heavy. 2) The patch was authored by someone at Facebook. The results were likely measured against a repository using remotefilelog. And I believe that revision retrieval during working directory updates with remotefilelog will often use a remote store, thus being I/O and not CPU bound. This probably resulted in an overstated performance gain. Since there appears to be a need to enable the thread-based worker with some stores, I've made the flagging of file gets as thread safe configurable. I've made it experimental because I don't want to formalize a boolean flag for this option and because this attribute is best captured against the store implementation. But we don't have a proper store API for this yet. I'd rather cross this bridge later. It is possible there are revlog-based repositories that do benefit from a thread-based worker. I didn't do very comprehensive testing. If there are, we may want to devise a more proper algorithm for whether to use the thread-based worker, including possibly config options to limit the number of threads to use. But until I see evidence that justifies complexity, simplicity wins. Differential Revision: https://phab.mercurial-scm.org/D3963

worker: ability to disable thread unsafe tasks The worker on Windows is implemented using a thread pool. If worker tasks are not thread safe, badness can occur. In addition, if tasks are executing CPU bound code and holding onto the GIL, there will be non-substantial overhead in Python context switching between active threads. This can result in significant slowdowns of tasks. This commit teaches the code for determining whether to use a worker to take thread safety into account. Effectively, thread unsafe tasks don't use the thread-based worker on Windows. Differential Revision: https://phab.mercurial-scm.org/D3962

worker: rename variable to reflect constant Differential Revision: https://phab.mercurial-scm.org/D3961

worker: use one pipe per posix worker and select() in parent process This allows us to pass results larger than PIPE_BUF through the pipes without interleaving them. This is necessary now because "hg fix" sends file contents as the result from workers. Differential Revision: https://phab.mercurial-scm.org/D3960

tests: refactor common bundle2 capabilities Differential Revision: https://phab.mercurial-scm.org/D1945

debug: move extensions debug behind a dedicated flag Since b86664c81833, we process the `--debug` flag earlier. This is overall good and useful, but has at least one negative side effect. Previously the debug message we report when trying to import extensions were issued before we processed the `--debug` flag. Now they happen after. Before: $ ./hg id --debug 21f507b8de2f9c1606e9aeb5ec7d2a6dedb7a4a7 tip After: $ ./hg id --debug ☿ (revset-bench) could not import hgext.evolve (No module named evolve): trying hgext3rd.evolve could not import hgext.mercurial_keyring (No module named mercurial_keyring): trying hgext3rd.mercurial_keyring could not import hgext3rd.mercurial_keyring (No module named mercurial_keyring): trying mercurial_keyring could not import hgext.hggit (No module named hggit): trying hgext3rd.hggit could not import hgext3rd.hggit (No module named hggit): trying hggit 21f507b8de2f9c1606e9aeb5ec7d2a6dedb7a4a7 tip (This get worse if --traceback is used). To work around this, we move this extensions related debug message behind a new flag 'devel.debug.extensions' and restore the previous output. I'm not fully happy about using the 'devel' section for a flag that can be used by legitimate users to debug extensions issues. However, it fits well next to other `devel.devel.*` options and is mostly used by extensions author anyway. We might move it to another, more appropriate section in the future (using alias).

curses: use "text" interface if TERM=dumb Differential Revision: https://phab.mercurial-scm.org/D3948

windows: expand '~/' and '~\' to %USERPROFILE% when translating to cmd.exe It's convenient to be able to reference hooks in a portable location on any platform.

windows: replace single quote with double quote when translating to cmd.exe Since cmd.exe doesn't understand single quotes, single quotes to prevent $var expansion is basically unusable without this. Single quote isn't allowed in a path name, so it seems unlikely to come up otherwise.

hook: only print the note about native cmd translation if it actually changes This makes it so that it will never occur on a non Windows platform.

hook: disable the shell to native command translation by default There are other things I want to add like ~ expansion and translating single to double quotes for cmd.exe. So off by default is safer. I'm having second thoughts about the name, but I don't have any better ideas.

setup: allow to run setup.py with python 3 without a mercurial checkout Some people may want to test mercurial in a python 3 environment through e.g. pip, in which case setup.py doesn't run in a mercurial checkout, so the hack in setup.py to allow python 3 cannot be overcome. This change allows a manual override with the HGPYTHON3 environment variable. Additionally, when for some reason the version is unknown (for crazy people like me, who have a git checkout of the mercurial repo), the version variable ends up being an unicode string, which fails the `isinstance(version, bytes)` assertion. So fix that at the same time. Differential Revision: https://phab.mercurial-scm.org/D3958

upgrade: enable adding or removing sparse-revlog requirement

upgrade: add information about sparse-revlog Show information about sparse-revlog in debugformat, just like other requirements.

sparse-revlog: implement algorithm to write sparse delta chains (issue5480) The classic behavior of revlog._isgooddeltainfo is to consider the span size of the whole delta chain, and limit it to 4 * textlen. Once sparse-revlog writing is allowed (and enforced with a requirement), revlog._isgooddeltainfo considers the span of the largest chunk as the distance used in the verification, instead of using the span of the whole delta chain. In order to compute the span of the largest chunk, we need to slice into chunks a chain with the new revision at the top of the revlog, and take the maximal span of these chunks. The sparse read density is a parameter to the slicing, as it will stop when the global read density reaches this threshold. For instance, a density of 50% means that 2 of 4 read bytes are actually used for the reconstruction of the revision (the others are part of other chains). This allows a new revision to be potentially stored with a diff against another revision anywhere in the history, instead of forcing it in the last 4 * textlen. The result is a much better compression on repositories that have many concurrent branches. Here are a comparison between using deltas from current upstream (aggressive-merge-deltas on by default) and deltas from a sparse-revlog Comparison of `.hg/store/` size: mercurial (6.74% merges): before: 46,831,873 bytes after: 46,795,992 bytes (no relevant change) pypy (8.30% merges): before: 333,524,651 bytes after: 308,417,511 bytes -8% netbeans (34.21% merges): before: 1,141,847,554 bytes after: 1,131,093,161 bytes -1% mozilla-central (4.84% merges): before: 2,344,248,850 bytes after: 2,328,459,258 bytes -1% large-private-repo-A (merge 19.73%) before: 41,510,550,163 bytes after: 8,121,763,428 bytes -80% large-private-repo-B (23.77%) before: 58,702,221,709 bytes after: 8,351,588,828 bytes -76% Comparison of `00manifest.d` size: mercurial (6.74% merges): before: 6,143,044 bytes after: 6,107,163 bytes pypy (8.30% merges): before: 52,941,780 bytes after: 27,834,082 bytes -48% netbeans (34.21% merges): before: 130,088,982 bytes after: 119,337,636 bytes -10% mozilla-central (4.84% merges): before: 215,096,339 bytes after: 199,496,863 bytes -8% large-private-repo-A (merge 19.73%) before: 33,725,285,081 bytes after: 390,302,545 bytes -99% large-private-repo-B (23.77%) before: 49,457,701,645 bytes after: 1,366,752,187 bytes -97% The better delta chains provide a performance boost in relevant repositories: pypy, bundling 1000 revisions: before: 1.670s after: 1.149s -31% Unbundling got a bit slower. probably because the sparse algorithm is still pure python. pypy, unbundling 1000 revisions: before: 4.062s after: 4.507s +10% Performance of bundle/unbundle in repository with few concurrent branches (eg: mercurial) are unaffected. No significant differences have been noticed then timing `hg push` and `hg pull` locally. More state timings are being gathered. Same as for aggressive-merge-delta, better delta comes with longer delta chains. Longer chains have a performance impact. For example. The length of the chain needed to get the manifest of pypy's tip moves from 82 item to 1929 items. This moves the restore time from 3.88ms to 11.3ms. Delta chain length is an independent issue that affects repository without this changes. It will be dealt with independently. No significant differences have been observed on repositories where `sparse-revlog` have not much effect (mercurial, unity, netbeans). On pypy, small differences have been observed on some operation affected by delta chain building and retrieval. pypy, perfmanifest before: 0.006162s after: 0.017899s +190% pypy, commit: before: 0.382 after: 0.376 -1% pypy, status: before: 0.157 after: 0.168 +7% More comprehensive and stable timing comparisons are in progress.

sparse-revlog: new requirement enabled with format.sparse-revlog The meaning of the new 'sparse-revlog' requirement is that the revlogs are allowed to contain wider delta chains with larger holes between the interesting chunks. These sparse delta chains should be read in several chunks to avoid a potential explosion of memory usage. Former version won't know how to read a delta chain in several chunks. They would keep reading them in a single read, and therefore would be subject to the potential memory explosion. Hence this new requirement: only versions having support of sparse-revlog reading should be allowed to read such a revlog. Implementation of this new algorithm and tools to enable or disable the requirement will follow in the next changesets.

revlog: extract `deltainfo.distance` for future conditional redefinition This commit exist to make the next one clearer.

shelve: pick the most recent shelve if none specified for --patch/--stat Differential Revision: https://phab.mercurial-scm.org/D3950

shelve: improve help text for --patch and --stat It's not currently obvious why "hg shelve -p" fails, since -p doesn't take an argument. Differential Revision: https://phab.mercurial-scm.org/D3949

ssh: avoid reading beyond the end of stream when using compression Compressed streams can be used as part of getbundle. The normal read() operation of bufferedinputpipe will try to fulfill the request exactly and can deadlock if the server sends less as it is done. At the same time, the bundle2 logic will stop reading when it believes it has gotten all parts of the bundle, which can leave behind end of stream markers as used by bzip2 and zstd. To solve this, introduce a new optional unbufferedread interface and provided it in bufferedinputpipe and doublepipe. If there is buffered data left, it will be returned, otherwise it will issue a single read request and return whatever it obtains. Reorganize the decompression handlers to try harder to read until the end of stream, especially if the requested read can already be fulfilled. Check for end of stream is messy with Python 2, none of the standard compression modules properly exposes it. At least with zstd and bzip2, decompressing will remember EOS and fail for empty input after the EOS has been seen. For zlib, the only way to detect it with Python 2 is to duplicate the decompressobj and force some additional data into it. The common handler can be further optimized, but works as PoC. Differential Revision: https://phab.mercurial-scm.org/D3937

revset: add larger test for heads(ancestors(…)) It is important to not regress on this benchmark so we move it into the "base" file. And we add another benchmark with more than two revisions.

revset-benchmark: use a generic revset to test `heads(commonancestors())` This allow to benchmark revset performance in other repositories than just the mercurial one.

revlog: reintroduce `revlog.descendant` as deprecated Reintroduce `revlog.descendant` to help extensions authors update their extensions in order to use the new API.

context: reintroduce `ctx.descendant` as deprecated Reintroduce `ctx.descendant` to help extensions authors update their extensions in order to use the new API.

obsolete: explode if metadata contains invalid UTF-8 sequence (API) The current metadata API can be a source of bugs since it forces callers to process encoding conversion by themselves. So let's make it reject bad data as a last ditch. I assume there's no metadata field which is supposed to store arbitrary BLOB like transplant_source.

obsolete: store user name and note in UTF-8 (issue5754) (BC) Before, user names were stored in local encoding and transferred across repositories, which made it impossible to restore non-ASCII user names on different platforms. This patch fixes new markers to be encoded in UTF-8 and decoded back to local encoding when displaying. Existing markers are unfixable so they may result in mojibake. I don't like the API that requires metadata dict to be UTF-8 encoded, which is a source of bugs, but there's no abstraction layer to process the encoding thingy efficiently. So we apply the same rule as extras dict to obsstore metadata.

obsolete: clarify users in markerusers() never contain None

revset: special case commonancestors(none()) to be empty set This matches the behavior of ancestor(none()). From an implementation perspective, ancestor() and commonancestors() are intersection, and ancestors() is union, so it would make some sense that commonancestors(none()) returned all revisions. However, ancestor(none()) isn't implemented as such, which breaks ancestor(x) == max(commonancestors(x)). From a user perspective, ancestors of nothing is nothing whichever type of operation the ancestor predicate does.

revset: clarify heads() order doesn't matter while computing common ancestors Follows up 5460926352ee and 52f19a840543.

hghave: require clang-format >= 6 due to output change

cext: reformat with clang-format 6.0 It appears some changes in clang-format affect our code. I didn't dig into that deeper since the new output looks better.

py3: use bytes() to byte-stringify Abort message in handleremotechangegroup()

py3: don't str() to byte-stringify object in test-bundle2-remote-changegroup.t

py3: byte-stringify literals in extension in test-bundle2-remote-changegroup.t # skip-blame just some b''

py3: open file in binary mode in test-bundle2-remote-changegroup.t

statprof: small if cleanup Explicitly testing for None to avoid comparison bugs.

store: assert the fncache have been loaded if dirty This should catch fncache corruption as the one that existed in `perffncachewrite`.

perffncachewrite: load fncache after lock is acquired Without this patch, running perffncachewrite on a repository destroy its fncache. Lock Acquisition drops various caches, including the fncache one. Then writing of an non-loaded fncache result into an empty one.

perf: add a 'perf.all-timing' option to display more than best time Minimal time is a useful information, but it is useful to have a wider view on the performance picture.

perf: document the perfmanifest command

fileset: remove fullmatchctx class It's exactly the same as matchctx.

fileset: remove subset and unused filtering functions from matchctx

fileset: remove callexisting flag and mctx.existing() (API) They are no longer needed since any files are included as long as they are passed in to the matcher.

fileset: rewrite predicates to return matcher not closed to subset (API) (BC) This makes fileset expression open to any input, so that we can just say "hg status 'set: not binary()'" to select text files including unknowns. With this and removal of subset computation, 'set:**' becomes as fast as 'glob:**'. Further optimization will probably be possible by narrowing the file tree to compute status for example. This also fixes 'subrepo()' to not ignore the current mctx.subset. .. bc:: The fileset expression may include untracked files by default. Use ``tracked()`` to explicitly filter out files not existing at the context revision.

highlight: use matcher API to test if file should be pygmentized Prepares for the upcoming API change. We'll no longer have to call the fileset function directly since the cost of the matcher-based fileset will become O(number of tests) from O(number of files in subset).

fileset: parse argument of size() by predicate function This change is necessary to pass in a size expression to predicatematcher. See the next patch.

fileset: add "tracked()" to explicitly select files in the revision I'm going to rewrite filesets to be match predicates, which means basic patterns such as '*' will no longer be "closed" to the subset constructed from the ctx. Good thing is that 'hg status "set:not binary()"' can include unknown files out of the box, and fileset computation will likely to be faster as we won't have to walk dirstate twice, for example. Bad thing is that we can't select files at a certain revision by 'set:revs(REV, **)' since '**' is "open" to any paths. So, this patch introduces "tracked()" as a replacement for the '**' in the example above.

fileset: rewrite andset() to not use mctx.narrow() New code is less efficient than the original, but it helps porting andset() to matcher composition. This will be cleaned up later. This effectively disables the fullmatchctx magic since mctx will never be demoted to the matchctx. The fullmatchctx class will be removed later.