debugcommands: introduce debugrevlogindex (BC) `hg debugindex` was originally invented for displaying revlog index data and is highly tailored towards that. e.g. it accepts a --format option to display index data for a particular revlog version and displays things like offset and length. As we support non-revlog storage, it makes sense for `hg debugindex` to display generic index data and for there to exist storage-specific or storage-aware debug* commands for dumping non-generic index data. This commit effectively renames `hg debugindex` to `hg debugrevlogindex` and replaces `hg debugindex` with a version that is storage agnostic. Tests using --format have been updated to use `hg debugrevlogindex`. Output is backwards compatible. The replacement command uses the formatter, which means output can be templatized. At some point, we may want to tweak output (e.g. to add the revision size). But I don't feel like taking a bigger BC break at the moment. The renamed command/function had to be moved because check-code enforces alphabetical ordering of commands in this file. Differential Revision: https://phab.mercurial-scm.org/D4358

debugcommands: use openstorage() in debugdata (BC) Nothing we're doing here requires a revlog. So use openstorage(). .. bc:: `hg debugdata` no longer accepts the path to a revlog file. Differential Revision: https://phab.mercurial-scm.org/D4357

tests: use inline Python for revlog test `hg debugdata` will soon stop accepting the raw path to a revlog file. Adjust a test accordingly. The changed test is for a security bug. So this should be reviewed with scrutiny. Differential Revision: https://phab.mercurial-scm.org/D4356

debugcommands: use openstorage() in debugindexdot And add test coverage for changelog and manifests. Differential Revision: https://phab.mercurial-scm.org/D4355

cmdutil: return a revlog from openrevlog() and split function The filelog class is a wrapper around a revlog instance. I have plans to give manifests and the changelog a similar treatment. When filelog was ported away from revlog and when I started writing patches to do the same for manifests, I noticed that a lot of debug* and perf* commands were relying on low-level revlog APIs like start(), end(), deltaparent(), etc. For filelog, I added these to the interface, even though I didn't want to because they don't belong on a generic storage interface. For manifest (and eventually changelog), the pain is too much to bear. We need to cut the tight coupling. These debug* and perf* commands use cmdutil.openrevlog() to obtain a revlog instance. This commit effectively renames openrevlog() to openstorage(), adds an argument to ensure a revlog instance is returned, and introduces a replacement openrevlog() that calls openstorage() such that a revlog instance is returned. By doing things this way, we allow the debug* and perf* commands to still work on revlog-based repositories without having to expose low-level revlog APIs in the storage interfaces. The practical side-effect of this on the current code base is we return a revlog instance instead of a filelog. The manifest and changelog are not affected at this time. Some of filelog's storage APIs are different from revlog. For example, read() strips the optional header containing copy/rename metadata. This may impact some perf* commands. But I don't think the impact is worth worrying about. Upcoming commits will port existing consumers to openstorage(), where appropriate. This commit does cause some test regressions when using the simple store. These will be fixed as commands are ported to use storage APIs. .. api:: cmdutil.openrevlog() now returns a revlog instance or aborts Previously, it would return a storage object, which may not be a revlog instance. Use the new cmdutil.openstorage() API to return an object conforming to the storage interface of the thing you are accessing if you don't need a revlog instance. Differential Revision: https://phab.mercurial-scm.org/D4354

merge: improve interactive one-changed one-deleted message (issue5550) I like the wording from the bug, so I figured I'd package it up in a change and see what people think. Differential Revision: https://phab.mercurial-scm.org/D4336

bookmark: add an --active flag to display the active bookmark There is currently no official simple way to retrieve the current bookmark. In particular for automation. We add a `--active` flag to the `hg bookmarks` command. When set, the command display the current bookmark name if any or return 1. For now, this flag is read-only. However sensible combinations exist with `--delete`, `--rename` and `--rev` and can be implemented later.

phase: use `_phase` revset instead of string interpolation The previous code was hackyer and assume all phases had an associated revset. The later might no longer be true once we introduce more internal phase.

phase: expose a `_phase(idx)` revset Internally phase related revset are calling the `_phase` function. We expose it as an internal revset. This is useful to refer to phase in revset doing debatable things around the phase name.

phase: handle phase with no command flag Before this changeset, all existing phases have a corresponding flag on the `hg phase` command. (eg: `hg phase --draft`). After this changeset, a phase can exists without having an associated flag. This is useful to introduce a new internal phase that we do not want to expose to user.

phase: explicitly exclude secret phase and above The comment explicitly mention the secret phase so we should as well use that in the code.

phase: use `trackedphases` in `_getphaserevsnative` Instead of manually listing secret and draft, simply use the `trackedphases` constant. The constant is already used by `_getphaserevsnative`.

lfs: add a progress bar when searching for blobs to upload The search itself can take an extreme amount of time if there are a lot of revisions involved. I've got a local repo that took 6 minutes to push 1850 commits, and 60% of that time was spent here (there are ~70K files): \ 58.1% wrapper.py: extractpointers line 297: pointers = extractpointers(... | 57.7% wrapper.py: pointersfromctx line 352: for p in pointersfromctx(ct... | 57.4% wrapper.py: pointerfromctx line 397: p = pointerfromctx(ctx, f, ... \ 38.7% context.py: __contains__ line 368: if f not in ctx: | 38.7% util.py: __get__ line 82: return key in self._manifest | 38.7% context.py: _manifest line 1416: result = self.func(obj) | 38.7% manifest.py: read line 472: return self._manifestctx.re... \ 25.6% revlog.py: revision line 1562: text = rl.revision(self._node) \ 12.8% revlog.py: _chunks line 2217: bins = self._chunks(chain, ... | 12.0% revlog.py: decompressline 2112: ladd(decomp(buffer(data, ch... \ 7.8% revlog.py: checkhash line 2232: self.checkhash(text, node, ... | 7.8% revlog.py: hash line 2315: if node != self.hash(text, ... | 7.8% revlog.py: hash line 2242: return hash(text, p1, p2) \ 12.0% manifest.py: __init__ line 1565: self._data = manifestdict(t... \ 16.8% context.py: filenode line 378: if not _islfs(fctx.filelog(... | 15.7% util.py: __get__ line 706: return self._filelog | 14.8% context.py: _filelog line 1416: result = self.func(obj) | 14.8% localrepo.py: file line 629: return self._repo.file(self... | 14.8% filelog.py: __init__ line 1134: return filelog.filelog(self... | 14.5% revlog.py: __init__ line 24: censorable=True)

i18n: merge i18n comments of translatable texts correctly Before this patch, i18n comments of translatable texts are lost at creation of hg.pot file, if: - same translatable text appears multiple times, - the 1st appearance does not have i18n comment, and - any of rest has it For example, previous patch for filemerge.py adds translatable texts with i18n comments, but these comments are lost, because: - automatically added texts in docstring of internal merge tools are picked up earlier than these translatable texts, because of location in filemerge.py - but docstring has no i18n comment This patch makes addentry() of posplit merge i18n comments of later translatable texts, in order to keep them at creation of hg.pot.

help: revise explanation about capability check while selecting merge tool This is follow up of 7c6044634957 and cded904f7acc. This patch adds explanations about: - notation in capability columns in the table - how capabilities of external merge tools are treated

filemerge: avoid putting translated text into docstring This is follow up of my mistake in e09fad982ef5. There is no merge tool, which has only one of binary or symlink capabilities, but this patch lists up all combinations of them for safety in the future. Maybe, it is too paranoid, though.

filemerge: make capability check for internal tools ignore merge-tools section This is follow up of 4d7b11877dd0. Before this patch, capability check of internal merge tools falls back to _toolbool(), which examines configurations in "merge-tools" section. But "hg help config" explicitly says that "merge-tools" section configures external merge tools. Therefore, this patch makes capability check for internal tools in hascapability() always ignore configurations in merge-tools section. In this patch, command line configurations below are added at tests in tests/test-merge-tools.t, in order to confirm that explicit configuration is intentionally ignored at tool selection. --config merge-tools.:INTERNAL_TOOL.CAPABILITY=true

merge with stable

scmutil: avoid quadratic membership testing (issue5969) tr.changes['revs'] is an xrange, which has an O(n) __contains__ implementation. The `rev not in newrevs` lookup a few lines below will therefore be O(n^2) if all incoming changesets are public. This issue isn't present on @ because 45e05d39d9ce introduced a custom type implementing an xrange primitive with O(1) contains and switched tr.changes['revs'] to be an instance of that type. We work around the problem on the stable branch by casting the xrange to a set. This is a bit hacky because it requires allocating memory to hold each integer in the range. But we are already holding the full set of pulled revision numbers in memory multiple times (such as in `tr.changes['phases']`). So this is a relatively minor problem. This issue has been present since the phases reporting code was introduced in the 4.7 cycle by eb9835014d20. This change should be reverted/ignored when stable is merged into default. On the mozilla-unified repository with 483492 changesets, `hg clone` time improves substantially: before: 1843.700s user; 29.810s sys after: 461.170s user; 29.360s sys

copies: correctly skip directories that have already been considered Previously, `if dsrc in invalid` would never be true, since we added `dsrc +"/"` to invalid, not `dsrc` itself. Since it's much more common for individual files (not whole directories) to be moved, it seemed cleaner to delay appending the "/" until we know we have some directory moves to actually consider. I haven't benchmarked this, but I imagine this is a mild performance win. Differential Revision: https://phab.mercurial-scm.org/D4284

merge with stable

match: make exactmatcher.visitchildrenset return file children as well Previously, if we had an exactmatcher like ['foo.txt', 'a/bar.txt', 'a/b/c/baz.txt'], we'd get back the following data: '.': {'a'} 'a': {'b'} 'a/b': {'c'} 'a/b/c': 'this' 'a/b/c/d': set() This was incorrect, since visitchildrenset explicitly says not to pay attention to 'foo.txt' and 'a/bar.txt' by not returning them or 'this'. Given the near impossibility of making visitchildrenset reliabbly produce only subdirectories, a previous commit has made it documented and expected that visitchildrenset can return a set containing both files and subdirectories to visit, instead of implying/requiring that visitchildrenset() return 'this' if there are files to visit. This makes the code for exactmatcher match this clarified documentation. Differential Revision: https://phab.mercurial-scm.org/D4365

match: document that visitchildrenset might return files At least when using includematcher, and probably most matchers, we do not know if a/b/f refers to a file 'f' in a/b, or a subdirectory 'f' in a/b, so most matchers will return {'f'} for visitchildrenset('a/b'). Arguably, all matchers could/should - for exactmatcher, we know that 'f' is a file, but there's no reason to return 'this' for visitchildrenset('a/b') causing code to investigate 'a/b/x', for example. Differential Revision: https://phab.mercurial-scm.org/D4364

util: make timedcm require the label (API) Differential Revision: https://phab.mercurial-scm.org/D4350

cleanup: make all uses of timedcm specify what they're timing It's not used in the timing itself, but it's valuable for the trace events we emit. Differential Revision: https://phab.mercurial-scm.org/D4349

util: make timedcm context manager also emit trace events Differential Revision: https://phab.mercurial-scm.org/D4348

demandimport: instrument python 2 code with trace events This causes the evaluation of an import in Python 3 to emit some trace data. There's some interesting wrinkles in here, like the fact that before we even hit dispatch we've demand-imported `sys` several times, despite the fact that `sys` was already fully loaded as one of the first few statements in the `hg` script. I don't think that's actually costing us a ton of performance, but it's probably something we should investigate fixing some day. Differential Revision: https://phab.mercurial-scm.org/D4347

dispatch: have dispatch.dispatch and dispatch._runcatch emit trace events Differential Revision: https://phab.mercurial-scm.org/D4345

tracing: new module to make tracing events in hg easier This lives in hgdemandimport because I want to instrument a bunch of low-level stuff including the bare `hg` script and demandimport, so it can't live at a higher layer. Differential Revision: https://phab.mercurial-scm.org/D4344

tests: add support for emitting trace events to run-tests Right now this is pretty basic, but it's a start. Differential Revision: https://phab.mercurial-scm.org/D4343

contrib: new script to read events from a named pipe and emit catapult traces I'm starting to get more serious about getting some insight into where we're spending our time, both in hg itself but also in the test suite. As a first pass, I'm going to try and produce catapult traces[0] that can be viewed with Chrome's `about:tracing` tool. 0: https://docs.google.com/document/d/1CvAClvFfyA5R-PhYUmn5OOQtYMH4h6I0nSsKchNAySU/edit#heading=h.nso4gcezn7n1 Differential Revision: https://phab.mercurial-scm.org/D4342

fastannotate: pconvert paths from the server for Windows I'm guessing that the right thing to do here is to convert the paths on the server, but I know this is a WIP, and I don't know where that needs to happen. I'm just trying to eliminate the malicious path warnings in the tests.