demandimport: instrument python 2 code with trace events This causes the evaluation of an import in Python 3 to emit some trace data. There's some interesting wrinkles in here, like the fact that before we even hit dispatch we've demand-imported `sys` several times, despite the fact that `sys` was already fully loaded as one of the first few statements in the `hg` script. I don't think that's actually costing us a ton of performance, but it's probably something we should investigate fixing some day. Differential Revision: https://phab.mercurial-scm.org/D4347

dispatch: have dispatch.dispatch and dispatch._runcatch emit trace events Differential Revision: https://phab.mercurial-scm.org/D4345

tracing: new module to make tracing events in hg easier This lives in hgdemandimport because I want to instrument a bunch of low-level stuff including the bare `hg` script and demandimport, so it can't live at a higher layer. Differential Revision: https://phab.mercurial-scm.org/D4344

tests: add support for emitting trace events to run-tests Right now this is pretty basic, but it's a start. Differential Revision: https://phab.mercurial-scm.org/D4343

contrib: new script to read events from a named pipe and emit catapult traces I'm starting to get more serious about getting some insight into where we're spending our time, both in hg itself but also in the test suite. As a first pass, I'm going to try and produce catapult traces[0] that can be viewed with Chrome's `about:tracing` tool. 0: https://docs.google.com/document/d/1CvAClvFfyA5R-PhYUmn5OOQtYMH4h6I0nSsKchNAySU/edit#heading=h.nso4gcezn7n1 Differential Revision: https://phab.mercurial-scm.org/D4342

fastannotate: pconvert paths from the server for Windows I'm guessing that the right thing to do here is to convert the paths on the server, but I know this is a WIP, and I don't know where that needs to happen. I'm just trying to eliminate the malicious path warnings in the tests.

test-fastannotate: close fd before unlinking to keep Windows happy

test-fastannotate: fix trivial output differences on Windows

fastannotate: make the default value for `fastannotate.useflock` dynamic fcntl.flock isn't available on Windows.

narrow: add a --narrowspec flag to clone command This patch adds a --narrowspec flag to `hg clone` command in narrow extension which can be used to read a file and parse narrowspecs from it and use it while cloning a repository. The --narrowspec flag assumes that the user wanted to narrow the clone. Tests are added both for ellipsis and non-ellipsis mode. Differential Revision: https://phab.mercurial-scm.org/D4156

manifest: use rev() instead of nodemap.__contains__ nodemap is an implementation detail of revlogs and isn't appropriate to expose on the manifest storage API. While revlogs don't have a __contains__, they do have lookup() for resolving a value to a node. And this calls rev(), whose API is documented to raise LookupError if a node doesn't exist. And the parameters to LookupError are identical to what was being raised here. So this change should be backwards compatible. Differential Revision: https://phab.mercurial-scm.org/D4279

manifest: rename manifestlog._treeinmem to ._treemanifests Not sure what "inmem" was supposed to indicate. This object is an interface to manifest data on disk as well as "in memory" (assuming that's what "inmem" means). Differential Revision: https://phab.mercurial-scm.org/D4278

manifest: add getstorage() to manifestlog and use it globally It is a common pattern to obtain a directory manifest storage instance (a manifestrevlog) by going through manifestlog._revlog.dirlog(). Why access to storage and caching of other manifests is done through manifestrevlog instead of manifestlog, I don't know. This commit establishes a getstorage(tree) API on manifestlog and imanifestlog that provides a public API for accessing manifest storage. All consumers previously using private attributes have been updated to use this new method. .. api:: manifestlog now has a getstorage(tree) method It should be used for obtaining an object representing the manifest's storage implementation. Accessing manifestlog._revlog should be avoided. Differential Revision: https://phab.mercurial-scm.org/D4277

manifest: rename dir argument and attribute to tree dir shadows a built-in. We use it throughout the manifest code, which is unfortunate. This commit updates just manifestrevlog to be more well behaved. .. api:: renamed manifest.manifestrevlog.__init__ dir argument to tree Differential Revision: https://phab.mercurial-scm.org/D4276

manifest: set appropriate cache entry when clearing _dirlogcache() manifestrevlog applies to any manifest, not just the root manifest. Resetting the cache and populating its root entry with self is not correct when the instance does not refer to the root manifest. This has no test fallout. So I'm guessing we only ever call clearcaches() on the root manifest. Or we have no test coverage that are impacted by a bad cache on a non-root manifestrevlog. Differential Revision: https://phab.mercurial-scm.org/D4275

manifest: remove addgroup() from manifestlog and imanifestlog addgroup() is part of the storage interface for manifests. Unfortunately, we don't yet have a formal storage interface for manifests. (One will be established in subsequent commits.) One thing is for sure, addgroup() doesn't belong on imanifestlog - at least not unless we extend that interface to encompass storage. For now, let's access addgroup() on the _revlog attribute, just like we do for tree manifests. Even though this violates visibility, it is consistent. Differential Revision: https://phab.mercurial-scm.org/D4274

repository: clarify role of imanifestlog Looking at the implementation of manifest.manifestlog, it is highly tailored towards servicing the root manifest. So clarify that in the interface docstring. Differential Revision: https://phab.mercurial-scm.org/D4273

changegroup: change topics during generation Changegroup generation and consumption currently uses different sets of topics. Generation uses "bundling" and consumption uses the name of the thing being consumed. This commit makes the topic and unit names consistent on both ends. The reason I chose to change the generation side is because when "bundling" is used for 3 different stages, it is unclear which stage of changegroup generation we're in. By making the topics "changesets," "manifests," and "files," one can get a better feel for how far along in the bundling process we are. Differential Revision: https://phab.mercurial-scm.org/D4272

changegroup: rename mfs to manifests In the name of readability. Differential Revision: https://phab.mercurial-scm.org/D4271

changegroup: clean up changelog callback changelog.read() calls changelog.changelogrevion() then turns it into a tuple. Let's call changelogrevision() directly and used named attributes to make code easier to read. While we're here, also change some variable names to make code easier to read. Differential Revision: https://phab.mercurial-scm.org/D4270

changegroup: call rev() on manifestlog instance rev() is part of the imanifestlog interface and should be used instead of using the private revlog instance, which is an implementation detail. Differential Revision: https://phab.mercurial-scm.org/D4269

manifest: rename dir to tree to avoid shadowing built-in And update the argument name in the imanifestlog interface. Differential Revision: https://phab.mercurial-scm.org/D4268

repository: remove candelta() from ifileindex candelta() was previously called by changegroup code. With delta generation moved to the storage API, there are no more external consumers of this method and it can be removed from the storage interface. Differential Revision: https://phab.mercurial-scm.org/D4236

changegroup: rename dir to tree to avoid shadowing a built-in Differential Revision: https://phab.mercurial-scm.org/D4235

repository: remove storedeltachains from ifilestorage The ifilestorage interface was bootstrapped from requirements of callers outside the storage implementation (revlogs). I believe we even made some members public so they could be part of the interface! Historically, the changegroup code was a gross offender when it came to accessing low-level storage primitives. There are a handful of members on the ifilestorage interface that are/were used only for changegroup code. With the recent refactor of changegroup code and the establishment of a formal API on the storage interface for producing revision deltas, the changegroup code is no longer accessing these low-level primitives related to delta generation directly. Instead, things are abstracted away in the storage implementation. This means we can remove elements from the storage interface that are no longer needed. We start with "storedeltachains." We remove it from the interface. Then we make it a private attribute and update all references. .. api:: storedeltachains has been dropped from ifilestorage interface .. api:: storedeltachains on revlog classes is now _storedeltachains Differential Revision: https://phab.mercurial-scm.org/D4227

repository: establish API for emitting revision deltas With our revision delta and revision delta request interfaces defined, it is now time to define a method on storage interfaces for using them. So far, the only storage interface that is well-defined and used is file storage. So that is the only interface we need to add a method on. We define an ``emitrevisiondeltas()`` method that takes an iterable of ``irevisiondeltarequest``s and turns them into ``irevisiondelta`` instances. changegroup._handlerevisiondeltarequest() and the looping logic from changegroup.deltagroup() has effectively been moved to revlog.emitrevisiondeltas(). Our filelog wrapper class proxies its emitrevisiondeltas() to the internal revlog instance. The simple store test extension used to verify sanity of storage abstractions has also implemented emitrevisiondeltas() for file storage and the test harness when run with this extension doesn't seem to exhibit any regressions. Rather than create a shared type to represent revision deltas, each storage backend has its own type and the class name identifies where the revision delta was derived from. Differential Revision: https://phab.mercurial-scm.org/D4226

repository: formalize interfaces for revision deltas and requests Now that we've sufficiently abstracted how revision deltas are produced in changegroup code, we can now start the process of formalizing that as part of the interfaces defined in the repository module. This commit essentially converts the revisiondelta and revisiondeltarequest classes into well-defined interfaces. This is not strictly necessary. But I want all types formalized by the storage interface to have interfaces. This makes it much easier to test for interface conformance and for implementing new storage backends. Because the interface is documented, comments and docstrings from changegroup.py have been dropped. Differential Revision: https://phab.mercurial-scm.org/D4225

changegroup: move node sorting into deltagroup() The 3 callers of deltagroup() all sort nodes and convert to integer revs immediately before calling deltagroup(). Stepping back a bit, aside from the constraints on node ordering that always apply (namely that a node must follow its ancestor), it makes sense for sorting to be an implementation detail of the store. Moving the sorting into deltagroup() will open the door for moving sorting into a storage API. The test for the changelog in deltagroup() to impact the sorting method is a bit hacky. I suspect this will be resolved once we establish a formal interface for delta group generation and port changelog to it. Differential Revision: https://phab.mercurial-scm.org/D4224

cext: fix truncation warnings in revlog on Windows

cext: fix revlog compiler error on Windows

shortest: use nodetree for finding shortest node within revset This speeds up `hg log -T '{shortest(node,1)}\n'` in my repo from 12s to 4.5s. That's very close to the 4.1s it takes without the disambiguation revset configured. My repo has 69.5k revisions, of which 550 were in the configured revset ("not public()"). Differential Revision: https://phab.mercurial-scm.org/D4120

index: fix a comment about overflow-checking There's no "argument-checking" done in this method. This was a bad copy paste. Differential Revision: https://phab.mercurial-scm.org/D4339

tests: demonstrate a problem with renames on the p2 side of a conversion I think this is related to the octopus merge being sloppy, and that's having a cascading affect on the fixup merge. If this change is made on p1 (specifically with the 'Added parent file' commit), the failure doesn't occur. The file modification with the rename doesn't seem to be necessary, but it's what's happening in a production repo where I first noticed, so I left it. This is an example of the manifest divergence I'd been seeing, which wasn't fixed by Yuya's recent changes. This is separate from the changelog divergence I was also seeing[1]. Probably nobody cares about bzr anymore, but this will also affect git, since the octopus fixup code is in the hg sink. [1] https://www.mercurial-scm.org/pipermail/mercurial-devel/2018-August/120473.html

tests: add a test to show the added/modified/removed files in a bzr convert

tests: show added/modified/removed files when logging repos converted from bzr There's a bug related to incorrect status values that can be triggered by an octopus fixup merge. Seeing what the current status is (instead of just the aggregate list) will hopefully help track this down.

tests: drop an unnecessary redirect to /dev/null This eliminates the need to conditionalize the color output, because NUL on Windows is a tty. Credit to Yuya for recognizing this.

py3: whitelist another test caught by the ratchet Differential Revision: https://phab.mercurial-scm.org/D4335

index: move raise_revlog_error() further up I will add another caller below it. Differential Revision: https://phab.mercurial-scm.org/D4119

index: make node tree a Python object Differential Revision: https://phab.mercurial-scm.org/D4118

index: rename _fix_index() since it no longer fixes the index Since c0d411ea6639 (index: drop support for negative indexes into the index, 2018-07-20), it always returns the input (unless it raises). Differential Revision: https://phab.mercurial-scm.org/D4334

index: don't include nullid in boundary check in pure code Yet another change that should probably have been part of a3dacabd476b (index: don't allow index[len(index)] to mean nullid, 2018-07-20). Differential Revision: https://phab.mercurial-scm.org/D4333

fastannotate: address pyflakes concerns Two unused imports, one unused local variable assignment. Differential Revision: https://phab.mercurial-scm.org/D4337

fastannotate: move some global state mutation to extsetup() Differential Revision: https://phab.mercurial-scm.org/D4202

fastannotate: rip out specialized support for remotefilelog remotefilelog can choose to collaborate with fastannotate for now if it needs to, and in the future when we make good on our longstanding desire to move remotefilelog-like features in-house we'll make sure things are well-supported via a reasonable interface. Differential Revision: https://phab.mercurial-scm.org/D4201

fastannotate: record some TODO items from indygreg's review Some of these I'll address more or less right away, but some will take a while for us to resolve. Differential Revision: https://phab.mercurial-scm.org/D4200

fastannotate: write out files from server in a predictable order This is mostly for test output stabilization, but it *might* be slightly better with a ton of annotate caches because we'll write to one directory at a time rather than bouncing around randomly. Differential Revision: https://phab.mercurial-scm.org/D3995

fastannotate: initial import from Facebook's hg-experimental I made as few changes as I could to get the tests to pass, but this was a bit involved due to some churn in the blame code since someone last gave fastannotate any TLC. There's still follow-up work here to rip out support for old versions of hg and to integrate the protocol with modern standards. Some performance numbers (all on my 2016 MacBook Pro with a 2.6Ghz i7): Mercurial mercurial/manifest.py traditional blame time: real 1.050 secs (user 0.990+0.000 sys 0.060+0.000) build cache time: real 5.900 secs (user 5.720+0.000 sys 0.110+0.000) fastannotate time: real 0.120 secs (user 0.100+0.000 sys 0.020+0.000) Mercurial mercurial/localrepo.py traditional blame time: real 3.330 secs (user 3.220+0.000 sys 0.070+0.000) build cache time: real 30.610 secs (user 30.190+0.000 sys 0.230+0.000) fastannotate time: real 0.180 secs (user 0.160+0.000 sys 0.020+0.000) mozilla-central dom/ipc/ContentParent.cpp traditional blame time: real 7.640 secs (user 7.210+0.000 sys 0.380+0.000) build cache time: real 98.650 secs (user 97.000+0.000 sys 0.950+0.000) fastannotate time: real 1.580 secs (user 1.340+0.000 sys 0.240+0.000) mozilla-central dom/base/nsDocument.cpp traditional blame time: real 17.110 secs (user 16.490+0.000 sys 0.500+0.000) build cache time: real 399.750 secs (user 394.520+0.000 sys 2.610+0.000) fastannotate time: real 1.780 secs (user 1.530+0.000 sys 0.240+0.000) So building the cache is expensive (but might be faster with xdiff enabled), but the blame results are *way* faster. Differential Revision: https://phab.mercurial-scm.org/D3994

util: improve handling of truncated compressed streams If the compressed stream is over as marked by the reader providing nothing new and the compression engine is not providing data, bail out. This can happen in practise when the server misbehaves and would result in an infinite loop otherwise. Differential Revision: https://phab.mercurial-scm.org/D4297