phases: enforce internal phase support We should not use the internal phase for repository without the requirement. Otherwise, older clients could have a look at the repository and see the internal changesets. For now, we introduce a low-level Programming error, more UI friendly error will be introduced later.

phases: add a repository requirement about internal phase For internal changeset to be properly hidden, the client version needs to support it. So we introduce a new repository requirement that will make sure clients touching a repository that uses internal phase supports the feature.

phases: add an internal phases The phase is meant to be used for internal commit that are a byproduct of command operation (eg:shelve). This changeset focus on getting the most important feature in, more advanced API is to be introduced in later changesets. The phase approach to handle internal has multiple advantages: * simple to implement, reuse optimized code, * fits well with existing phases. We don't want internal changeset to be exchanged or served. * easy to extend to for lifecycle handling. More thinking about internal changeset at https://www.mercurial-scm.org/wiki/InternalsPlan We add this new phases with a high number to leave some room to possible other phases. We also try out playing with the idea of "flag" for phases, each flag would convey a distinct meaning. We can drop the flag idea in the future (keeping the existing numbers). The flag property should still move in a monotonic direction (enabled -> disabled) or be immutable like the "internal" flag. To simplify the addition of this new phase, we introduce many placeholder phases. They are not meant to be used for now. Keeping `allphases` as a list ensure existing algorithm works fine. The full performance impact of adding that many hollow phases is unclear so far. The impact on phase computation is visible but not worrisome. Runnin `hg perfphase` in my mercurial development repository. Before: ! wall 0.001807 comb 0.000000 user 0.000000 sys 0.000000 (median of 1597) after: ! wall 0.001906 comb 0.000000 user 0.000000 sys 0.000000 (median of 1521)

stringutil: teach pprint() to recognize generators Otherwise they get serialized as e.g. <generator object X at 0x7f543d3d68c0> Differential Revision: https://phab.mercurial-scm.org/D4396

commitextras: no need to special case extras=[] Differential Revision: https://phab.mercurial-scm.org/D4405

commitextras: work nicely with other extensions Before this change, it doesn't add these extra fields when loaded alongside another extension that does a bunch of things, including wrapping commit. I did not investigate exactly why, but - the documentation of extensions.wrapfunction says to use subclassing to play nicely with other extensions - using subclassing does make commitextras work when loaded alongside my other extension Differential Revision: https://phab.mercurial-scm.org/D4404

phases: simplify revset in updatephases (Pointed by Yuya Nishihara)

context: use new names for unstable changesets in docstrings Plus slight corrections.

index: embed nodetree in index object to avoid reference cycle Since the index has a reference to a nodetree and the nodetree has a reference back to the index, there is a reference cycle, so the index (and its nodetree) can never be freed. This patch fixes that by making "nodetree" a plan C struct that the index can embed, and also introduces a new "nodetreeObject" that is a Python type wrapping the nodetree struct. Thanks to Yuya for noticing this and for suggesting the solution. All tests passed on the first attempt once it compiled (I guess C is like Haskell in this regard?). Differential Revision: https://phab.mercurial-scm.org/D4372

catapipe: make the file executable This silences test-check-execute.t. Differential Revision: https://phab.mercurial-scm.org/D4403

run-tests: fix the remaining failure of test-run-tests.py Looks like durin42 misindented this. This makes test-run-tests.py passes but I cannot reason why. Differential Revision: https://phab.mercurial-scm.org/D4402

run-tests: don't append rtendtracing logs if pipe is /dev/null If HGCATAPULTSERVERPIPE is not set, we set it to /dev/null, so we need to make sure the value is not /dev/null before appending logs. This fixes some of test-run-tests.t failures. Differential Revision: https://phab.mercurial-scm.org/D4401

cmdutil: fix typo in revert() Differential Revision: https://phab.mercurial-scm.org/D4381

log: respect graphshorten on terminal nodes (collapsing o-~ to just o~) Internally we have a custom template that's inspired by ones that we have seen in the community. Normally, this looks something like: o 0834ec17 spectral tip | crecord: support x to toggle single, X to toggle a range o ee932990 spectral @ | filemerge: allow specifying $hgeditor as merge-tools.X.executable @ 66f04611 matt_harbison | cext: fix truncation warnings in revlog on Windows o 42cc76d0 matt_harbison | cext: fix revlog compiler error on Windows ~ o bd63ada7 stable boris | phases: drop dead code in `newheads` ~ With graphshorten on, and the descriptions of the public nodes hidden, it looks like this, note that the commits right before the ~ are still "full height": o 0834ec17 spectral tip | crecord: support x to toggle single, X to toggle a range o ee932990 spectral @ | filemerge: allow specifying $hgeditor as merge-tools.X.executable @ 66f04611 matt_harbison o 42cc76d0 matt_harbison | ~ o bd63ada7 stable boris | ~ This patch makes them look like this, removing the | but keeping the ~: o 0834ec17 spectral tip | crecord: support x to toggle single, X to toggle a range o ee932990 spectral @ | filemerge: allow specifying $hgeditor as merge-tools.X.executable @ 66f04611 matt_harbison o 42cc76d0 matt_harbison ~ o bd63ada7 stable boris ~ This originally removed the ~s entirely, but this was determined to be too much information loss and potentially confusing. This would have looked like the following (note that the last commit is on a different branch than all of the ones above it, and they are *not* linearly related): o 0834ec17 spectral tip | crecord: support x to toggle single, X to toggle a range o ee932990 spectral @ | filemerge: allow specifying $hgeditor as merge-tools.X.executable @ 66f04611 matt_harbison o 42cc76d0 matt_harbison o bd63ada7 stable boris Differential Revision: https://phab.mercurial-scm.org/D4363

filemerge: fix the wrong placements of messages in prompt Mistakenly, the local-changed-other-deleted-message was put in other-change-local-deleted and vice versa. This followups on D4336. Thanks to Yuya who spotted this. Differential Revision: https://phab.mercurial-scm.org/D4375

filemerge: print the filename in quotes in prompt This is better as the filename is in quotes and user can understand better what the filename is and what is important part in the prompt. Differential Revision: https://phab.mercurial-scm.org/D4373

filelog: remove proxies to revlog These proxy attributes and methods were added to facilitate various debug* and perf* commands. Now that cmdutil.openrevlog() returns an actual revlog, we no longer need these on the filelog class. There's probably a few other attributes that could be removed. But this feels like a worthy start. Differential Revision: https://phab.mercurial-scm.org/D4359

debugcommands: introduce debugrevlogindex (BC) `hg debugindex` was originally invented for displaying revlog index data and is highly tailored towards that. e.g. it accepts a --format option to display index data for a particular revlog version and displays things like offset and length. As we support non-revlog storage, it makes sense for `hg debugindex` to display generic index data and for there to exist storage-specific or storage-aware debug* commands for dumping non-generic index data. This commit effectively renames `hg debugindex` to `hg debugrevlogindex` and replaces `hg debugindex` with a version that is storage agnostic. Tests using --format have been updated to use `hg debugrevlogindex`. Output is backwards compatible. The replacement command uses the formatter, which means output can be templatized. At some point, we may want to tweak output (e.g. to add the revision size). But I don't feel like taking a bigger BC break at the moment. The renamed command/function had to be moved because check-code enforces alphabetical ordering of commands in this file. Differential Revision: https://phab.mercurial-scm.org/D4358

debugcommands: use openstorage() in debugdata (BC) Nothing we're doing here requires a revlog. So use openstorage(). .. bc:: `hg debugdata` no longer accepts the path to a revlog file. Differential Revision: https://phab.mercurial-scm.org/D4357

tests: use inline Python for revlog test `hg debugdata` will soon stop accepting the raw path to a revlog file. Adjust a test accordingly. The changed test is for a security bug. So this should be reviewed with scrutiny. Differential Revision: https://phab.mercurial-scm.org/D4356

debugcommands: use openstorage() in debugindexdot And add test coverage for changelog and manifests. Differential Revision: https://phab.mercurial-scm.org/D4355

cmdutil: return a revlog from openrevlog() and split function The filelog class is a wrapper around a revlog instance. I have plans to give manifests and the changelog a similar treatment. When filelog was ported away from revlog and when I started writing patches to do the same for manifests, I noticed that a lot of debug* and perf* commands were relying on low-level revlog APIs like start(), end(), deltaparent(), etc. For filelog, I added these to the interface, even though I didn't want to because they don't belong on a generic storage interface. For manifest (and eventually changelog), the pain is too much to bear. We need to cut the tight coupling. These debug* and perf* commands use cmdutil.openrevlog() to obtain a revlog instance. This commit effectively renames openrevlog() to openstorage(), adds an argument to ensure a revlog instance is returned, and introduces a replacement openrevlog() that calls openstorage() such that a revlog instance is returned. By doing things this way, we allow the debug* and perf* commands to still work on revlog-based repositories without having to expose low-level revlog APIs in the storage interfaces. The practical side-effect of this on the current code base is we return a revlog instance instead of a filelog. The manifest and changelog are not affected at this time. Some of filelog's storage APIs are different from revlog. For example, read() strips the optional header containing copy/rename metadata. This may impact some perf* commands. But I don't think the impact is worth worrying about. Upcoming commits will port existing consumers to openstorage(), where appropriate. This commit does cause some test regressions when using the simple store. These will be fixed as commands are ported to use storage APIs. .. api:: cmdutil.openrevlog() now returns a revlog instance or aborts Previously, it would return a storage object, which may not be a revlog instance. Use the new cmdutil.openstorage() API to return an object conforming to the storage interface of the thing you are accessing if you don't need a revlog instance. Differential Revision: https://phab.mercurial-scm.org/D4354

merge: improve interactive one-changed one-deleted message (issue5550) I like the wording from the bug, so I figured I'd package it up in a change and see what people think. Differential Revision: https://phab.mercurial-scm.org/D4336

bookmark: add an --active flag to display the active bookmark There is currently no official simple way to retrieve the current bookmark. In particular for automation. We add a `--active` flag to the `hg bookmarks` command. When set, the command display the current bookmark name if any or return 1. For now, this flag is read-only. However sensible combinations exist with `--delete`, `--rename` and `--rev` and can be implemented later.

phase: use `_phase` revset instead of string interpolation The previous code was hackyer and assume all phases had an associated revset. The later might no longer be true once we introduce more internal phase.

phase: expose a `_phase(idx)` revset Internally phase related revset are calling the `_phase` function. We expose it as an internal revset. This is useful to refer to phase in revset doing debatable things around the phase name.

phase: handle phase with no command flag Before this changeset, all existing phases have a corresponding flag on the `hg phase` command. (eg: `hg phase --draft`). After this changeset, a phase can exists without having an associated flag. This is useful to introduce a new internal phase that we do not want to expose to user.

phase: explicitly exclude secret phase and above The comment explicitly mention the secret phase so we should as well use that in the code.

phase: use `trackedphases` in `_getphaserevsnative` Instead of manually listing secret and draft, simply use the `trackedphases` constant. The constant is already used by `_getphaserevsnative`.

lfs: add a progress bar when searching for blobs to upload The search itself can take an extreme amount of time if there are a lot of revisions involved. I've got a local repo that took 6 minutes to push 1850 commits, and 60% of that time was spent here (there are ~70K files): \ 58.1% wrapper.py: extractpointers line 297: pointers = extractpointers(... | 57.7% wrapper.py: pointersfromctx line 352: for p in pointersfromctx(ct... | 57.4% wrapper.py: pointerfromctx line 397: p = pointerfromctx(ctx, f, ... \ 38.7% context.py: __contains__ line 368: if f not in ctx: | 38.7% util.py: __get__ line 82: return key in self._manifest | 38.7% context.py: _manifest line 1416: result = self.func(obj) | 38.7% manifest.py: read line 472: return self._manifestctx.re... \ 25.6% revlog.py: revision line 1562: text = rl.revision(self._node) \ 12.8% revlog.py: _chunks line 2217: bins = self._chunks(chain, ... | 12.0% revlog.py: decompressline 2112: ladd(decomp(buffer(data, ch... \ 7.8% revlog.py: checkhash line 2232: self.checkhash(text, node, ... | 7.8% revlog.py: hash line 2315: if node != self.hash(text, ... | 7.8% revlog.py: hash line 2242: return hash(text, p1, p2) \ 12.0% manifest.py: __init__ line 1565: self._data = manifestdict(t... \ 16.8% context.py: filenode line 378: if not _islfs(fctx.filelog(... | 15.7% util.py: __get__ line 706: return self._filelog | 14.8% context.py: _filelog line 1416: result = self.func(obj) | 14.8% localrepo.py: file line 629: return self._repo.file(self... | 14.8% filelog.py: __init__ line 1134: return filelog.filelog(self... | 14.5% revlog.py: __init__ line 24: censorable=True)

i18n: merge i18n comments of translatable texts correctly Before this patch, i18n comments of translatable texts are lost at creation of hg.pot file, if: - same translatable text appears multiple times, - the 1st appearance does not have i18n comment, and - any of rest has it For example, previous patch for filemerge.py adds translatable texts with i18n comments, but these comments are lost, because: - automatically added texts in docstring of internal merge tools are picked up earlier than these translatable texts, because of location in filemerge.py - but docstring has no i18n comment This patch makes addentry() of posplit merge i18n comments of later translatable texts, in order to keep them at creation of hg.pot.

help: revise explanation about capability check while selecting merge tool This is follow up of 7c6044634957 and cded904f7acc. This patch adds explanations about: - notation in capability columns in the table - how capabilities of external merge tools are treated

filemerge: avoid putting translated text into docstring This is follow up of my mistake in e09fad982ef5. There is no merge tool, which has only one of binary or symlink capabilities, but this patch lists up all combinations of them for safety in the future. Maybe, it is too paranoid, though.

filemerge: make capability check for internal tools ignore merge-tools section This is follow up of 4d7b11877dd0. Before this patch, capability check of internal merge tools falls back to _toolbool(), which examines configurations in "merge-tools" section. But "hg help config" explicitly says that "merge-tools" section configures external merge tools. Therefore, this patch makes capability check for internal tools in hascapability() always ignore configurations in merge-tools section. In this patch, command line configurations below are added at tests in tests/test-merge-tools.t, in order to confirm that explicit configuration is intentionally ignored at tool selection. --config merge-tools.:INTERNAL_TOOL.CAPABILITY=true

merge with stable

scmutil: avoid quadratic membership testing (issue5969) tr.changes['revs'] is an xrange, which has an O(n) __contains__ implementation. The `rev not in newrevs` lookup a few lines below will therefore be O(n^2) if all incoming changesets are public. This issue isn't present on @ because 45e05d39d9ce introduced a custom type implementing an xrange primitive with O(1) contains and switched tr.changes['revs'] to be an instance of that type. We work around the problem on the stable branch by casting the xrange to a set. This is a bit hacky because it requires allocating memory to hold each integer in the range. But we are already holding the full set of pulled revision numbers in memory multiple times (such as in `tr.changes['phases']`). So this is a relatively minor problem. This issue has been present since the phases reporting code was introduced in the 4.7 cycle by eb9835014d20. This change should be reverted/ignored when stable is merged into default. On the mozilla-unified repository with 483492 changesets, `hg clone` time improves substantially: before: 1843.700s user; 29.810s sys after: 461.170s user; 29.360s sys

copies: correctly skip directories that have already been considered Previously, `if dsrc in invalid` would never be true, since we added `dsrc +"/"` to invalid, not `dsrc` itself. Since it's much more common for individual files (not whole directories) to be moved, it seemed cleaner to delay appending the "/" until we know we have some directory moves to actually consider. I haven't benchmarked this, but I imagine this is a mild performance win. Differential Revision: https://phab.mercurial-scm.org/D4284

merge with stable

match: make exactmatcher.visitchildrenset return file children as well Previously, if we had an exactmatcher like ['foo.txt', 'a/bar.txt', 'a/b/c/baz.txt'], we'd get back the following data: '.': {'a'} 'a': {'b'} 'a/b': {'c'} 'a/b/c': 'this' 'a/b/c/d': set() This was incorrect, since visitchildrenset explicitly says not to pay attention to 'foo.txt' and 'a/bar.txt' by not returning them or 'this'. Given the near impossibility of making visitchildrenset reliabbly produce only subdirectories, a previous commit has made it documented and expected that visitchildrenset can return a set containing both files and subdirectories to visit, instead of implying/requiring that visitchildrenset() return 'this' if there are files to visit. This makes the code for exactmatcher match this clarified documentation. Differential Revision: https://phab.mercurial-scm.org/D4365

match: document that visitchildrenset might return files At least when using includematcher, and probably most matchers, we do not know if a/b/f refers to a file 'f' in a/b, or a subdirectory 'f' in a/b, so most matchers will return {'f'} for visitchildrenset('a/b'). Arguably, all matchers could/should - for exactmatcher, we know that 'f' is a file, but there's no reason to return 'this' for visitchildrenset('a/b') causing code to investigate 'a/b/x', for example. Differential Revision: https://phab.mercurial-scm.org/D4364

util: make timedcm require the label (API) Differential Revision: https://phab.mercurial-scm.org/D4350

cleanup: make all uses of timedcm specify what they're timing It's not used in the timing itself, but it's valuable for the trace events we emit. Differential Revision: https://phab.mercurial-scm.org/D4349

util: make timedcm context manager also emit trace events Differential Revision: https://phab.mercurial-scm.org/D4348

demandimport: instrument python 2 code with trace events This causes the evaluation of an import in Python 3 to emit some trace data. There's some interesting wrinkles in here, like the fact that before we even hit dispatch we've demand-imported `sys` several times, despite the fact that `sys` was already fully loaded as one of the first few statements in the `hg` script. I don't think that's actually costing us a ton of performance, but it's probably something we should investigate fixing some day. Differential Revision: https://phab.mercurial-scm.org/D4347

dispatch: have dispatch.dispatch and dispatch._runcatch emit trace events Differential Revision: https://phab.mercurial-scm.org/D4345

tracing: new module to make tracing events in hg easier This lives in hgdemandimport because I want to instrument a bunch of low-level stuff including the bare `hg` script and demandimport, so it can't live at a higher layer. Differential Revision: https://phab.mercurial-scm.org/D4344

tests: add support for emitting trace events to run-tests Right now this is pretty basic, but it's a start. Differential Revision: https://phab.mercurial-scm.org/D4343

contrib: new script to read events from a named pipe and emit catapult traces I'm starting to get more serious about getting some insight into where we're spending our time, both in hg itself but also in the test suite. As a first pass, I'm going to try and produce catapult traces[0] that can be viewed with Chrome's `about:tracing` tool. 0: https://docs.google.com/document/d/1CvAClvFfyA5R-PhYUmn5OOQtYMH4h6I0nSsKchNAySU/edit#heading=h.nso4gcezn7n1 Differential Revision: https://phab.mercurial-scm.org/D4342

fastannotate: pconvert paths from the server for Windows I'm guessing that the right thing to do here is to convert the paths on the server, but I know this is a WIP, and I don't know where that needs to happen. I'm just trying to eliminate the malicious path warnings in the tests.

test-fastannotate: close fd before unlinking to keep Windows happy

test-fastannotate: fix trivial output differences on Windows

fastannotate: make the default value for `fastannotate.useflock` dynamic fcntl.flock isn't available on Windows.

narrow: add a --narrowspec flag to clone command This patch adds a --narrowspec flag to `hg clone` command in narrow extension which can be used to read a file and parse narrowspecs from it and use it while cloning a repository. The --narrowspec flag assumes that the user wanted to narrow the clone. Tests are added both for ellipsis and non-ellipsis mode. Differential Revision: https://phab.mercurial-scm.org/D4156

manifest: use rev() instead of nodemap.__contains__ nodemap is an implementation detail of revlogs and isn't appropriate to expose on the manifest storage API. While revlogs don't have a __contains__, they do have lookup() for resolving a value to a node. And this calls rev(), whose API is documented to raise LookupError if a node doesn't exist. And the parameters to LookupError are identical to what was being raised here. So this change should be backwards compatible. Differential Revision: https://phab.mercurial-scm.org/D4279

manifest: rename manifestlog._treeinmem to ._treemanifests Not sure what "inmem" was supposed to indicate. This object is an interface to manifest data on disk as well as "in memory" (assuming that's what "inmem" means). Differential Revision: https://phab.mercurial-scm.org/D4278

manifest: add getstorage() to manifestlog and use it globally It is a common pattern to obtain a directory manifest storage instance (a manifestrevlog) by going through manifestlog._revlog.dirlog(). Why access to storage and caching of other manifests is done through manifestrevlog instead of manifestlog, I don't know. This commit establishes a getstorage(tree) API on manifestlog and imanifestlog that provides a public API for accessing manifest storage. All consumers previously using private attributes have been updated to use this new method. .. api:: manifestlog now has a getstorage(tree) method It should be used for obtaining an object representing the manifest's storage implementation. Accessing manifestlog._revlog should be avoided. Differential Revision: https://phab.mercurial-scm.org/D4277

manifest: rename dir argument and attribute to tree dir shadows a built-in. We use it throughout the manifest code, which is unfortunate. This commit updates just manifestrevlog to be more well behaved. .. api:: renamed manifest.manifestrevlog.__init__ dir argument to tree Differential Revision: https://phab.mercurial-scm.org/D4276

manifest: set appropriate cache entry when clearing _dirlogcache() manifestrevlog applies to any manifest, not just the root manifest. Resetting the cache and populating its root entry with self is not correct when the instance does not refer to the root manifest. This has no test fallout. So I'm guessing we only ever call clearcaches() on the root manifest. Or we have no test coverage that are impacted by a bad cache on a non-root manifestrevlog. Differential Revision: https://phab.mercurial-scm.org/D4275

manifest: remove addgroup() from manifestlog and imanifestlog addgroup() is part of the storage interface for manifests. Unfortunately, we don't yet have a formal storage interface for manifests. (One will be established in subsequent commits.) One thing is for sure, addgroup() doesn't belong on imanifestlog - at least not unless we extend that interface to encompass storage. For now, let's access addgroup() on the _revlog attribute, just like we do for tree manifests. Even though this violates visibility, it is consistent. Differential Revision: https://phab.mercurial-scm.org/D4274

repository: clarify role of imanifestlog Looking at the implementation of manifest.manifestlog, it is highly tailored towards servicing the root manifest. So clarify that in the interface docstring. Differential Revision: https://phab.mercurial-scm.org/D4273

changegroup: change topics during generation Changegroup generation and consumption currently uses different sets of topics. Generation uses "bundling" and consumption uses the name of the thing being consumed. This commit makes the topic and unit names consistent on both ends. The reason I chose to change the generation side is because when "bundling" is used for 3 different stages, it is unclear which stage of changegroup generation we're in. By making the topics "changesets," "manifests," and "files," one can get a better feel for how far along in the bundling process we are. Differential Revision: https://phab.mercurial-scm.org/D4272

changegroup: rename mfs to manifests In the name of readability. Differential Revision: https://phab.mercurial-scm.org/D4271

changegroup: clean up changelog callback changelog.read() calls changelog.changelogrevion() then turns it into a tuple. Let's call changelogrevision() directly and used named attributes to make code easier to read. While we're here, also change some variable names to make code easier to read. Differential Revision: https://phab.mercurial-scm.org/D4270

changegroup: call rev() on manifestlog instance rev() is part of the imanifestlog interface and should be used instead of using the private revlog instance, which is an implementation detail. Differential Revision: https://phab.mercurial-scm.org/D4269

manifest: rename dir to tree to avoid shadowing built-in And update the argument name in the imanifestlog interface. Differential Revision: https://phab.mercurial-scm.org/D4268

repository: remove candelta() from ifileindex candelta() was previously called by changegroup code. With delta generation moved to the storage API, there are no more external consumers of this method and it can be removed from the storage interface. Differential Revision: https://phab.mercurial-scm.org/D4236

changegroup: rename dir to tree to avoid shadowing a built-in Differential Revision: https://phab.mercurial-scm.org/D4235

repository: remove storedeltachains from ifilestorage The ifilestorage interface was bootstrapped from requirements of callers outside the storage implementation (revlogs). I believe we even made some members public so they could be part of the interface! Historically, the changegroup code was a gross offender when it came to accessing low-level storage primitives. There are a handful of members on the ifilestorage interface that are/were used only for changegroup code. With the recent refactor of changegroup code and the establishment of a formal API on the storage interface for producing revision deltas, the changegroup code is no longer accessing these low-level primitives related to delta generation directly. Instead, things are abstracted away in the storage implementation. This means we can remove elements from the storage interface that are no longer needed. We start with "storedeltachains." We remove it from the interface. Then we make it a private attribute and update all references. .. api:: storedeltachains has been dropped from ifilestorage interface .. api:: storedeltachains on revlog classes is now _storedeltachains Differential Revision: https://phab.mercurial-scm.org/D4227

repository: establish API for emitting revision deltas With our revision delta and revision delta request interfaces defined, it is now time to define a method on storage interfaces for using them. So far, the only storage interface that is well-defined and used is file storage. So that is the only interface we need to add a method on. We define an ``emitrevisiondeltas()`` method that takes an iterable of ``irevisiondeltarequest``s and turns them into ``irevisiondelta`` instances. changegroup._handlerevisiondeltarequest() and the looping logic from changegroup.deltagroup() has effectively been moved to revlog.emitrevisiondeltas(). Our filelog wrapper class proxies its emitrevisiondeltas() to the internal revlog instance. The simple store test extension used to verify sanity of storage abstractions has also implemented emitrevisiondeltas() for file storage and the test harness when run with this extension doesn't seem to exhibit any regressions. Rather than create a shared type to represent revision deltas, each storage backend has its own type and the class name identifies where the revision delta was derived from. Differential Revision: https://phab.mercurial-scm.org/D4226

repository: formalize interfaces for revision deltas and requests Now that we've sufficiently abstracted how revision deltas are produced in changegroup code, we can now start the process of formalizing that as part of the interfaces defined in the repository module. This commit essentially converts the revisiondelta and revisiondeltarequest classes into well-defined interfaces. This is not strictly necessary. But I want all types formalized by the storage interface to have interfaces. This makes it much easier to test for interface conformance and for implementing new storage backends. Because the interface is documented, comments and docstrings from changegroup.py have been dropped. Differential Revision: https://phab.mercurial-scm.org/D4225

changegroup: move node sorting into deltagroup() The 3 callers of deltagroup() all sort nodes and convert to integer revs immediately before calling deltagroup(). Stepping back a bit, aside from the constraints on node ordering that always apply (namely that a node must follow its ancestor), it makes sense for sorting to be an implementation detail of the store. Moving the sorting into deltagroup() will open the door for moving sorting into a storage API. The test for the changelog in deltagroup() to impact the sorting method is a bit hacky. I suspect this will be resolved once we establish a formal interface for delta group generation and port changelog to it. Differential Revision: https://phab.mercurial-scm.org/D4224

cext: fix truncation warnings in revlog on Windows

cext: fix revlog compiler error on Windows

shortest: use nodetree for finding shortest node within revset This speeds up `hg log -T '{shortest(node,1)}\n'` in my repo from 12s to 4.5s. That's very close to the 4.1s it takes without the disambiguation revset configured. My repo has 69.5k revisions, of which 550 were in the configured revset ("not public()"). Differential Revision: https://phab.mercurial-scm.org/D4120

index: fix a comment about overflow-checking There's no "argument-checking" done in this method. This was a bad copy paste. Differential Revision: https://phab.mercurial-scm.org/D4339

tests: demonstrate a problem with renames on the p2 side of a conversion I think this is related to the octopus merge being sloppy, and that's having a cascading affect on the fixup merge. If this change is made on p1 (specifically with the 'Added parent file' commit), the failure doesn't occur. The file modification with the rename doesn't seem to be necessary, but it's what's happening in a production repo where I first noticed, so I left it. This is an example of the manifest divergence I'd been seeing, which wasn't fixed by Yuya's recent changes. This is separate from the changelog divergence I was also seeing[1]. Probably nobody cares about bzr anymore, but this will also affect git, since the octopus fixup code is in the hg sink. [1] https://www.mercurial-scm.org/pipermail/mercurial-devel/2018-August/120473.html

tests: add a test to show the added/modified/removed files in a bzr convert

tests: show added/modified/removed files when logging repos converted from bzr There's a bug related to incorrect status values that can be triggered by an octopus fixup merge. Seeing what the current status is (instead of just the aggregate list) will hopefully help track this down.

tests: drop an unnecessary redirect to /dev/null This eliminates the need to conditionalize the color output, because NUL on Windows is a tty. Credit to Yuya for recognizing this.

py3: whitelist another test caught by the ratchet Differential Revision: https://phab.mercurial-scm.org/D4335

index: move raise_revlog_error() further up I will add another caller below it. Differential Revision: https://phab.mercurial-scm.org/D4119

index: make node tree a Python object Differential Revision: https://phab.mercurial-scm.org/D4118

index: rename _fix_index() since it no longer fixes the index Since c0d411ea6639 (index: drop support for negative indexes into the index, 2018-07-20), it always returns the input (unless it raises). Differential Revision: https://phab.mercurial-scm.org/D4334

index: don't include nullid in boundary check in pure code Yet another change that should probably have been part of a3dacabd476b (index: don't allow index[len(index)] to mean nullid, 2018-07-20). Differential Revision: https://phab.mercurial-scm.org/D4333

fastannotate: address pyflakes concerns Two unused imports, one unused local variable assignment. Differential Revision: https://phab.mercurial-scm.org/D4337

fastannotate: move some global state mutation to extsetup() Differential Revision: https://phab.mercurial-scm.org/D4202

fastannotate: rip out specialized support for remotefilelog remotefilelog can choose to collaborate with fastannotate for now if it needs to, and in the future when we make good on our longstanding desire to move remotefilelog-like features in-house we'll make sure things are well-supported via a reasonable interface. Differential Revision: https://phab.mercurial-scm.org/D4201

fastannotate: record some TODO items from indygreg's review Some of these I'll address more or less right away, but some will take a while for us to resolve. Differential Revision: https://phab.mercurial-scm.org/D4200

fastannotate: write out files from server in a predictable order This is mostly for test output stabilization, but it *might* be slightly better with a ton of annotate caches because we'll write to one directory at a time rather than bouncing around randomly. Differential Revision: https://phab.mercurial-scm.org/D3995

fastannotate: initial import from Facebook's hg-experimental I made as few changes as I could to get the tests to pass, but this was a bit involved due to some churn in the blame code since someone last gave fastannotate any TLC. There's still follow-up work here to rip out support for old versions of hg and to integrate the protocol with modern standards. Some performance numbers (all on my 2016 MacBook Pro with a 2.6Ghz i7): Mercurial mercurial/manifest.py traditional blame time: real 1.050 secs (user 0.990+0.000 sys 0.060+0.000) build cache time: real 5.900 secs (user 5.720+0.000 sys 0.110+0.000) fastannotate time: real 0.120 secs (user 0.100+0.000 sys 0.020+0.000) Mercurial mercurial/localrepo.py traditional blame time: real 3.330 secs (user 3.220+0.000 sys 0.070+0.000) build cache time: real 30.610 secs (user 30.190+0.000 sys 0.230+0.000) fastannotate time: real 0.180 secs (user 0.160+0.000 sys 0.020+0.000) mozilla-central dom/ipc/ContentParent.cpp traditional blame time: real 7.640 secs (user 7.210+0.000 sys 0.380+0.000) build cache time: real 98.650 secs (user 97.000+0.000 sys 0.950+0.000) fastannotate time: real 1.580 secs (user 1.340+0.000 sys 0.240+0.000) mozilla-central dom/base/nsDocument.cpp traditional blame time: real 17.110 secs (user 16.490+0.000 sys 0.500+0.000) build cache time: real 399.750 secs (user 394.520+0.000 sys 2.610+0.000) fastannotate time: real 1.780 secs (user 1.530+0.000 sys 0.240+0.000) So building the cache is expensive (but might be faster with xdiff enabled), but the blame results are *way* faster. Differential Revision: https://phab.mercurial-scm.org/D3994

util: improve handling of truncated compressed streams If the compressed stream is over as marked by the reader providing nothing new and the compression engine is not providing data, bail out. This can happen in practise when the server misbehaves and would result in an infinite loop otherwise. Differential Revision: https://phab.mercurial-scm.org/D4297

zsh_completion: update all options It's just too hard to further split this patch. What it does: - adds missing flags - removes flags that are no longer there - updates flag descriptions and argument names - adds * where using the same option many times is okay - groups with () mutually exclusive flags that can't be used together - adds + and = to options that need arguments - removes + and = from options that don't take any arguments - fixes minor issues and a typo Differential Revision: https://phab.mercurial-scm.org/D4267

zsh_completion: run hg with HGPLAINEXCEPT=alias to list commands This is what contrib/bash_completion does, looks like a sensible thing to do. Differential Revision: https://phab.mercurial-scm.org/D4266

zsh_completion: remove completion for obsolete graphlog extension Differential Revision: https://phab.mercurial-scm.org/D4265

zsh_completion: declare appropriate local parameters for ->string form When ->string form is used for _arguments, the function that calls it must declare appropriate local parameters. Managing local return value is needed to tell the completion system if our function succeeded in suggesting something or not, plus without that `hg diff -<TAB>` doesn't look right. While at it, fix a copypaste error (s/diff_files/revert_files/). Docs: http://zsh.sourceforge.net/Doc/Release/Completion-System.html Differential Revision: https://phab.mercurial-scm.org/D4264

zsh_completion: complete files from `hg st -mar` for commit and diff Useful to have and is consistent with our bash completion file. This is also an improvement coming from upstream, downstream or sidestream (depends on how you look at it) hg completion file -- the one that lives in zsh's own repo. Differential Revision: https://phab.mercurial-scm.org/D4263

zsh_completion: drop -w, add -S to _arguments From the docs: http://zsh.sourceforge.net/Doc/Release/Completion-System.html _arguments -w: "allow option stacking even if one or more of the options take arguments", so something like "-xy xarg yarg". Mercurial's command line parser doesn't support that, so the flag got removed in zsh 92584634d3d3 and bf8b61182043. _arguments -S: "do not complete options after a ‘--’ appearing on the line". By options it means things like "-x" and "--foo", but it will still complete arguments like files. Differential Revision: https://phab.mercurial-scm.org/D4262

zsh_completion: rename _hg_style_opts to _hg_template_opts --style is a deprecated alias for --template, and doesn't show up in help without --verbose, so completing it doesn't make much sense. Let's s/style/template/ this variable to be more consistent with cmdutil.py, which is helpful to look at in parallel next time someone wants to update the completion rules. Differential Revision: https://phab.mercurial-scm.org/D4261

pycompat: wrap xrange for py2 to provide efficient __contains__ The C implementation of xrange in Python 2 provides a O(n) membership test, which is noticable on pull-based clones of large repositories. Avoid this by providing a wrapper class with O(1) membership test based on the edges of the range. Differential Revision: https://phab.mercurial-scm.org/D4313

phases: drop dead code in `newheads` There are new code earlier in the function achieving the same result. Spotted by Yuya Nishihara.

phases: fix `nullid` reference in newheads Spotted by Yuya Nishihara.

beautifygraph: don't warn about busted terminal if HGPLAIN is set This was breaking some automation for users that discovered the extension and turned it on. Differential Revision: https://phab.mercurial-scm.org/D4338

merge with stable

merge: do not delete untracked files silently (issue5962) 37450a122128, 2a774cae3a03, and 656ac240f392 weren't enough to prevent data loss. No unknown "files" weren't deleted before 7a8a16f8ea22, "context: also consider path conflicts when clearing unknown files."

node: correct hex representation of pseudo node ids

merge with stable

perf: fix typo in perfphasesremote()

revlog: fix typo in 'buildtext' name There are no such things as 'deltacomputer._buildtext'. I'm not sure why this code never ran, but we better fix the typo.

revlog: fix pure version of _partialmatch() to include nullid Before this patch, test-issue842.t and a few more tests fail when they try to refer to the null revision by using a "000.." prefix of it (or because they use the "shortest" template function which internally does that). This should have been part of my a3dacabd476b (index: don't allow index[len(index)] to mean nullid, 2018-07-20), but I had forgotten to update another part of the pure code there, so it didn't fail until a1f934573c0b (parsers: adjust pure-python version to mimic a3dacabd476b, 2018-08-09) and 65d5de1169dd (revlog: fix pure nodemap to not access missing index entry, 2018-08-17) fixed the other things I had missed. Differential Revision: https://phab.mercurial-scm.org/D4332

tests: conditionalize color output on abort This broke in afc4ad706f9c. I'm not sure why other aborts don't need to be conditionalized, but `hg help foo` does abort in red for both msys and cmd.exe.

tests: conditionalize out tests that msys is mangling This was exposed by ac0a87160012. The paths for convert and ssh respectively are being turned into: $TESTTMP\bzr+ssh:\foobar@selenic.com\baz\.hg 'C:\\C:\\Users\\Matt\\AppData\\Local\\Temp\\hgtests.8awa9x\\child1\\test-ssh-bundle1.t-sshv2\\nonexistent\\.hg The errno value raised is 123, and the message is The filename, directory name, or volume label syntax is incorrect

test-absorb: stabilize for no-execbit platforms

test-graft: add a missing output line for Windows

share: allow more than one level of directories to be created

clone: allow local cloning to create more than one level of directories I figure cloning a remote repository is more common, thus it's more likely that some people might be relying on the less restrictive behavior. Additionally, `hg init` will also create more than one level of missing directories. I also have a use case for reading the subrepos from .hgsub, and sharing them into the normal nested location on the server. SCM Manager doesn't host subrepos in the normal nested location, which is nice for deduplicating the repository data, but confuses `hg verify`. Some of the subrepos are in the root of the repositories, while others are one or two directories deep. So not having to build up the parent path first is desirable.

tests: demonstrate an inconsistency when cloning to a missing directory tree I noticed that `hg share` is unable to create more than one missing directory on the path, and thought it was inconsistent with clone. It turns out that the path for copying/linking the remote store has the same limitation, but cloning from a remote repo doesn't.

tests: stabilize test-merge-tools.t on Windows More fun with `hg import --bypass` to work around Windows limitations. The diffs were generated on Linux, and had a tab to terminate the `+++b/...` lines. But check-code complained about trailing whitespace, and it seems to run without them.

dagutil: remove module The previous commit removed the last consumer of this module. .. api:: dagutil module has been removed Some functionality has been moved to the dagop module. Other functionality can be accomplished via revsets. Differential Revision: https://phab.mercurial-scm.org/D4330

dagop: port revlogdag.linearize() to standalone function The code should functionally be identical. We also port the one consumer in changegroup to use the new standalone function. After this commit, dagutil is no longer used! Differential Revision: https://phab.mercurial-scm.org/D4329

dagutil: use revlog.parentrevs() for resolving parent revisions And remove parents() since it is no longer used. revlog.parentrevs() is almost the same as parents(). The main difference is that parentrevs() can return nullrev. dagop.headrevs() already handles nullrev. We add an inline check for nullrev in the other call site to account for the difference. .. api:: parents() removed from dagutil classes Use parentrevs() on the storage object instead. Differential Revision: https://phab.mercurial-scm.org/D4328