changegroup: pass ellipsis roots into cgpacker constructor And rename the internal variable to conform with naming conventions. Differential Revision: https://phab.mercurial-scm.org/D4089

changegroup: move revision maps to cgpacker And remove the underscores so the variables conform to our naming convention. The logic in _close() should be the only thing warranting scrutiny during review. Differential Revision: https://phab.mercurial-scm.org/D4088

changegroup: move changelogdone into cgpacker Looking at what it is used for, it feels like there is a better way to implement all this. So recording a TODO to track that. Differential Revision: https://phab.mercurial-scm.org/D4087

changegroup: declare shallow flag in constructor Thus begins the process of better formalizing ellipses and shallow changegroup generation mode so it is tracked by cgpacker at construction time instead of bolted on after the fact by a wrapper function. Differential Revision: https://phab.mercurial-scm.org/D4086

changegroup: make some packer attributes private These methods and attributes are low level and should not be called or outside outside of instances. Indicate as such through naming. Differential Revision: https://phab.mercurial-scm.org/D4085

changegroup: rename cg1packer to cgpacker There is now only a single class. We don't need to encode the version in its name since the version is a lie. Differential Revision: https://phab.mercurial-scm.org/D4084

changegroup: control delta parent behavior via constructor The last remaining override on cg2packer related to parent delta computation. We pass a parameter to the constructor to control whether to delta against the previous revision and we inline all parent delta logic into a single function. With this change, cg2packer is empty, so it has been deleted. Differential Revision: https://phab.mercurial-scm.org/D4083

changegroup: control reordering via constructor argument cg2packer.__init__ exists just to override self._reorder. Let's parameterize this behavior via an argument to the parent's __init__. The logic for self._reorder is kinda wonky. None is used as a special value and the value should be None in some situations. It is probably worth rewriting this logic to make behavior more explicit. This will likely happen as part of future work to control the delta generation process that I have planned. Differential Revision: https://phab.mercurial-scm.org/D4082

changegroup: consolidate tree manifests sending into cg1packer Previously, we overrode a method to control how manifests were serialized. This method was redefined on cg3packer to send tree manifests. This commit moves the tree manifests sending variation to cg1packer and teaches the cgpacker constructor to control which version to use. After these changes, cg3packer was empty. So it has been removed. Differential Revision: https://phab.mercurial-scm.org/D4081

changegroup: pass end of manifests marker into constructor cg3 inserts a custom marker in the stream once all manifests have been transferred. This is currently abstracted out by overriding a method. Let's pass the end of manifests marker in as an argument to avoid the extra method. Differential Revision: https://phab.mercurial-scm.org/D4080

changegroup: pass function to build delta header into constructor Previously, the delta header struct format was defined on each class and each class had a separate function for building the delta header. We replace both of these with an argument to __init__ containing a callable that can format a delta header given a revisiondelta instance. Differential Revision: https://phab.mercurial-scm.org/D4079

changegroup: make delta header struct formatters actual structs Why we weren't using compiled Struct instances, I don't know. They make code simpler. In theory they are faster. Although I don't believe I was able to measure any meaningful change. That could be because this code is often dominated by compression, deltafication, and function call overhead. Differential Revision: https://phab.mercurial-scm.org/D4078

changegroup: pass version into constructor Currently, the version is an attribute on each class. Passing the argument into the constructor gets us one step closer to eliminating cg2packer and cg3packer. Differential Revision: https://phab.mercurial-scm.org/D4077

changegroup: define functions for creating changegroup packers Currently, we have 3 classes for changegroup generation. Each class handles a specific changegroup format. And each subsequent version's class inherits from the previous one. The interface for the classes is not very well defined and a lot of version-specific behavior is behind overloaded functions. This approach adds complexity and makes changegroup generation difficult to reason about. Upcoming commits will be consolidating these 3 classes so differences between changegroup versions and changegroup generation are controlled by parameters to a single constructor / type rather than by overriding class attributes via inheritance. We begin this process by building dedicated functions for creating each changegroup packer instance. Currently they just call the constructor on the appropriate class. This will soon change. Differential Revision: https://phab.mercurial-scm.org/D4076

changegroup: capture revision delta in a data structure The current changegroup generation code is tightly coupled to the revlog API. This tight coupling makes it difficult to implement alternate storage backends without requiring a large surface area of the revlog API to be exposed. This is not desirable. In order to support changegroup generation with non-revlog storage, we'll need to abstract the concept of delta generation. This commit is the first step down that road. We introduce a data structure for representing a delta in a changegroup. The API still leaves a lot to be desired. But at least we now have separation between data and actions performed on it. As part of this, we tweak behavior slightly: we no longer concatenate the delta prefix with the metadata header. Instead, we track and emit the prefix as a separate chunk. This shouldn't have any meaningful impact since all the chunks just get sent to the wire, the compressor, etc. Because we're introducing a new object, this does add some overhead to changegroup execution. `hg perfchangegroupchangelog` on my clone of the Mercurial repo (~40,000 visible revisions in the changelog) slows down a bit: ! wall 1.268600 comb 1.270000 user 1.270000 sys 0.000000 (best of 8) ! wall 1.419479 comb 1.410000 user 1.410000 sys 0.000000 (best of 8) With for `hg bundle -t none-v2 -a /dev/null`: before: real 6.610 secs (user 6.460+0.000 sys 0.140+0.000) after: real 7.210 secs (user 7.060+0.000 sys 0.140+0.000) I plan to claw back this regression in future commits. And I may even do away with this data structure once the refactor is complete. For now, it makes things easier to comprehend. Differential Revision: https://phab.mercurial-scm.org/D4075

changegroup: inline ellipsisdata() There's only one caller of it. I don't think it needs to exist as a standalone function. Differential Revision: https://phab.mercurial-scm.org/D4074

changegroup: rename "revlog" variables "revlog" shadows the module import. But more importantly, changegroup generation should be storage agnostic and not assume the existence of revlogs. Let's rename the thing providing revision storage to "store" to reflect this ideal property. Differential Revision: https://phab.mercurial-scm.org/D4073

changegroup: move generate() modifications from narrow Narrow had a custom version of generate() that was essentially a copy of generate() with inline additions to facilitate ellipses serving. This commit inlines those modifications into generate(). Differential Revision: https://phab.mercurial-scm.org/D4067

changegroup: move generatefiles() from narrow The code is a bit ugly in that it overrides the linknodes function that is passed in as a function. I'd like to think that the caller of generatefiles() would pass in the appropriate function. We can clean this up later. Differential Revision: https://phab.mercurial-scm.org/D4066

changegroup: move _sortgroup() from narrow Differential Revision: https://phab.mercurial-scm.org/D4065

changegroup: move close() from narrow More of the same. Differential Revision: https://phab.mercurial-scm.org/D4064

changegroup: move revchunk() from narrow The monkeypatched revchunk for ellipses serving is a completely independent implementation. We model it as such in the changegroup code. revchunk() is now a simple proxy function. Again, I wish we had better APIs here. Especially since this narrow code is part of cg1packer and cg1packer can't be used with narrow. Class inheritance is wonky. And I will definitely be making changes to changegroup code for delta generation. As part of the code move, `node.nullrev` was replaced by `nullrev`. And a reference to `orig` was replaced to call `self._revchunknormal` directly. Differential Revision: https://phab.mercurial-scm.org/D4063

changegroup: move deltaparent() from narrow I'm not keen on performing the attribute sniff to test for presence of ellipses mode: I'd rather we use a separate packer instance that was ellipses mode specific. But I've tried to formalize a better API without narrow in core and I can't make sense of all the monkeypatching. My goal is to inline as much of the monkeypatching as possible then refactor the changegroup generation API. We add this code to the cg2packer because narrow doesn't work with cg1. Differential Revision: https://phab.mercurial-scm.org/D4062

changegroup: move _packellipsischangegroup() from narrow The behavior here is not ideal, as the function constructs a packer then adds attributes to it. This will be cleaned up in subsequent commits. Moving this code is necessary to move the remainder of the bundle2-level changegroup part generation code into core. Differential Revision: https://phab.mercurial-scm.org/D4061

changegroup: move ellipsisdata() from narrow This is a pretty straightforward copy of the function. Differential Revision: https://phab.mercurial-scm.org/D4060

fileset: narrow status computation by left-hand-side of 'and' node Timing with warm disk cache: $ hg status --cwd mozilla-central 'set:path:build/ and unknown()' --time (orig) time: real 1.970 secs (user 1.560+0.000 sys 0.410+0.000) (new) time: real 0.330 secs (user 0.310+0.000 sys 0.020+0.000)

fileset: move copy constructor of matchctx near __init__

fileset: build status according to 'withstatus' hint _switchcallers is no longer needed since 'withstatus' node is reinserted for arguments of functions like revs(). New matchctx instance is created per 'withstatus' to make sure that status tuple is available only for children of the 'withstatus' node.

fileset: insert hints where status should be computed This will allow us to compute status against a narrowed set of files. For example, "path:build/ & (unknown() + missing())" is rewritten as "path:build/ & <withstatus>(unknown() + missing(), 'unknown missing')", and the status call can be narrowed by the left-hand-side matcher, "path:build/". mctx.buildstatus() calls will be solely processed by getmatchwithstatus().

fileset: move buildstatus() to matchctx method In future patches, file status will be computed while evaluating a parsed tree. This patch provides a matchctx interface to build status.

fileset: keep basectx by matchctx

fileset: pass in basectx to _buildstatus() I'll make matchctx remember both ctx and basectx so that file status between them can be computed later. This prepares for the change.

resolve: update commands.resolve.confirm help text Included --mark and --unmark in the help text of resolve.confirm.config. Differential Revision: https://phab.mercurial-scm.org/D4103

resolve: support confirm config option with --unmark flag Now, commands.resolve.confirm also respect --unmark option; and confirm to unresolve all resolved files. It will confirm only when no files pats are passed (same as --mark), because when no pats are passed the default is to mark resolved files as unresolved. And if user has passed file pats then I think there is no need to confirm for that. Differential Revision: https://phab.mercurial-scm.org/D4102

resolve: correct behavior of mark-check=none to match docs Differential Revision: https://phab.mercurial-scm.org/D4121

narrow: move .hg/narrowspec to .hg/store/narrowspec (BC) The narrowspec is more closely related to the store than to the working copy. For example, if the narrowspec changes, the set of revlogs also needs to change (the working copy may change, but that depends on which commit is checked out). Also, when using the share extension, the narrowspec needs to be shared along with the store. This patch therefore moves the narrowspec into the store/ directory. This is clearly a breaking change, but I haven't bothered trying to fall back to reading the narrowspec from the old location (.hg/), because there are very few users of narrow out there. (We'll add a temporary hack to our Google-internal extension to handle the migration.) Differential Revision: https://phab.mercurial-scm.org/D4099

narrow: drop checkambig=True when restoring backup IIUC, checkambig is about updating timestamps of the file while renaming. That's important for the dirstate, but we never check the timestamp of the narrowspec file. We can therefore avoid checking passing checkambig=True. Differential Revision: https://phab.mercurial-scm.org/D4098

narrow: remove a repo file-cache invalidation It's unclear why this was needed. All tests pass without it. I asked Kyle Lippincott (who added the check) and he also doesn't remember what it was for. Differential Revision: https://phab.mercurial-scm.org/D4097

narrow: call narrowspec.{save,restore,clear}backup directly I want to move .hg/narrowspec to .hg/store/narrowspec and we need to decouple the narrowspec update from the dirstate update for that. This patch lets the callers call the narrowspec backup functions directly, in addition to the dirstate backup functions. The narrowspec methods are made to check if narrowing is enabled. For that, a repo instance was needed, which all the callers luckily already had available. Differential Revision: https://phab.mercurial-scm.org/D4096

index: don't add 1 to length variables A lot of "+ 1" and "-1" were mechanically added to ease the transition in 781b2720d2ac (index: don't include nullid in len(), 2018-07-20). Let's clean it up now. Differential Revision: https://phab.mercurial-scm.org/D4106

index: drop support for nullid at position len(index) in index_node I think no callers exist since at least a3dacabd476b (index: don't allow index[len(index)] to mean nullid, 2018-07-20). Differential Revision: https://phab.mercurial-scm.org/D4105

index: return False for "len(index) in index" Since we no longer accept index[len(index)], we should clearly make "len(index) in index" return False. This should have been part of a3dacabd476b (index: don't allow index[len(index)] to mean nullid, 2018-07-20) Differential Revision: https://phab.mercurial-scm.org/D4104

fileset: combine union of basic patterns into single matcher This appears to improve query performance in a big repository than I thought. Writing less Python in a hot loop, faster computation we gain. $ hg files --cwd mozilla-central --time 'set:a* + b* + c* + d* + e*' (orig) time: real 0.670 secs (user 0.640+0.000 sys 0.030+0.000) (new) time: real 0.210 secs (user 0.180+0.000 sys 0.020+0.000)

fileset: reorder 'or' expression by weight

fileset: introduce weight constants for readability These constants are defined in the filesetlang module since it's the bottommost module depending on WEIGHT_CHECK_FILENAME, and extensions will be likely to import it to process function arguments. Credit for the naming goes to Augie Fackler.

sparse: use named parameters in i18n strings This should give more hints about what the %s means, and allow reordering.

sparse: do not include operator in i18n strings

fix: compute changed lines lazily to make whole-file fixer tools faster Differential Revision: https://phab.mercurial-scm.org/D4100