revlog: add option to mmap revlog index Following on from Jun Wu's patch last October[1], we have found that using mmap for the revlog index in repos with large revlogs gives a noticable performance improvment (~110ms on each hg invocation), particularly for commands that don't touch the index very much. This changeset adds this as an option, activated by a new experimental config option so that it can be enabled on a per-repo basis. The configuration option specifies an index size threshold at which Mercurial will switch to using mmap to access the index. If the configuration option is not specified, the default remains to load the full file, which seems to be the best option for smaller repos. Some initial performance numbers for average of 5 invocations of `hg log -l 5` for different cache states: | Repo: | HG | FB | |---|---|---| | Index size: | 2.3MB | much bigger | | read (warm): | 237ms | 432ms | | mmap (warm): | 227ms | 321ms | | | (-3%) | (-26%) | | read (cold): | 397ms | 696ms | | mmap (cold): | 410ms | 888ms | | | (+3%) | (+28%) | [1] https://www.mercurial-scm.org/pipermail/mercurial-devel/2016-October/088737.html Test Plan: `hg log --config experimental.mmapindex=true` Differential Revision: https://phab.mercurial-scm.org/D477

util: add an mmapread method This is useful for large files that are only partly touched. Test Plan: Will be used and tested in a later patch. Differential Revision: https://phab.mercurial-scm.org/D476

changegroup: remove dictionary creation from deltachunk Previously delta chunk returned a dictionary. Now that we consume deltachunk within changegroup (instead of outside in revlog) we can just return a tuple and have it be returned directly by deltaiter. Differential Revision: https://phab.mercurial-scm.org/D746

bundlerepo: update to use new deltaiter api Differential Revision: https://phab.mercurial-scm.org/D745

debug: update debugbundle to use new deltaiter api Changegroup now has a deltaiter api for easy iteration over a series of deltas. Let's use that in the debugbundle command. Differential Revision: https://phab.mercurial-scm.org/D744

revlog: add revmap back to revlog.addgroup The recent c8b6ed51386b patch removed the linkmapper argument from addgroup, as part of trying to make addgroup more agnostic from the changegroup format. It turns out that the changegroup can't resolve linkrevs while iterating over the deltas, because applying the deltas might affect the linkrev resolution. For example, when applying a series of changelog entries, the linkmapper just returns len(cl). If we're iterating over the deltas without applying them to the changelog, this results in incorrect linkrevs. This was caught by the hgsql extension, which reads the revisions before applying them. The fix is to return linknodes as part of the delta iterator, and let the consumer choose what to do. Differential Revision: https://phab.mercurial-scm.org/D730

rebase: also include other namespaces in changeset description This makes use of the generic method of listing bookmarks and tags, so other extensions that add other namespaces will get their names added too. This does mean that bookmarks will come before tags, just like we apparently decided to order them in the "hg log" output. It doesn't seem like people would be parsing the rebase output anyway. We also did 79ab5369d55a (rebase: use _ctxdesc in one more place, 2017-08-29) recently, so now seems like a good time. Differential Revision: https://phab.mercurial-scm.org/D741

tersestatus: sort the dictionary before iterating on it There has report of flakiness in test-status-terse.t. In the terse code, we are iterating on a dictionary without sorting and since python dicts are unordered, that can be a reason behind the flakiness. Before we have a better implementation for the terse thing, let's make sure current implementation possess minimum bugs. Differential Revision: https://phab.mercurial-scm.org/D740

copytrace: use the full copytracing method if only drafts are involved This patch adds the functionality to use the full copytracing even if `experimental.copytrace = heuristics` in cases when drafts are involved. This is also a part of copytrace extension in fbext. This also adds tests which are also taken from fbext. .. feature:: The `heuristics` option for `experimental.copytrace` performs full copytracing if both source and destination branches contains non-public changsets only. Differential Revision: https://phab.mercurial-scm.org/D625

templates: introduce a obsfateoperation() function Differential Revision: https://phab.mercurial-scm.org/D723

obsmarker: track operation by default We added support for including the operation responsible for creating the obsmarker in 3546a771e376 (obsolete: add operation metadata to rebase/amend/histedit obsmarkers, 2017-05-09). However, soon thereafter, in f432897a9f49 (obsmarker: add an experimental flag controlling "operation" recording, 2017-05-20), it was hidden behind a config that was off by default. It seems unlikely that people will manually turn it on, and obsmarkers/evolution as a whole is still experimental anyway, so let's turn on the tracking by default. Differential Revision: https://phab.mercurial-scm.org/D722

uncommit: add an experimental.uncommitondirtywdir config There are people who wants to do `hg uncommit` on dirty working directory, so this patch adds a config which can be used to the achieve that. Adds tests for the same. Differential Revision: https://phab.mercurial-scm.org/D735

uncommit: don't allow bare uncommit on dirty working directory As per IRC discussion and suggestion by Augie, we should not allow bare `hg uncommit` on dirty working directory. Differential Revision: https://phab.mercurial-scm.org/D734

uncommit: rename the flag 'empty' to 'keep' which retains empty changeset This patch renames the flag 'empty' which retains the empty changeset if all the files are uncommitted to 'keep'. Differential Revision: https://phab.mercurial-scm.org/D726

tests: avoid repr in test-bundle2-format.t We don't *really* need it here, and it was making Python 3 harder.

tests: port test-bundle2-format inline helper script to Python 3 We're now very close to this test passing.

bundle2: portably grab first byte of part name for letter check

bundle2: make ValueError messages native strings

bundle2: update check for a generator to work on Python 3

bundle2: stop using %r to quote part names Valid part names are restricted to [a-zA-Z0-9_:-]+, so I'm not worried about having quoting present in places where we should have predominantly valid part names. This will significantly ease the Python 3 transition, and simultaneously isn't a BC because this is only in error messages that should never be shown.

blackbox: set lastui even if ui.log is not called (issue5518) `lastui` decides where (where is the `.hg`) to use if the current `ui` object does not have a `_bbrepo` associated. Previously it only gets set in `ui.log`, which means unless a `ui` with repo associated calls `log` with tracked event, blackbox does not know where to write its log. This patch makes `reposetup` set `lastui` so it so we could log some more events (see test changes). Differential Revision: https://phab.mercurial-scm.org/D655

blackbox: unindent a try block The try block is no longer necessary. Differential Revision: https://phab.mercurial-scm.org/D654

blackbox: simplify ui states It seems cleaner to just remove `_partialinit`, `copy`, `__init__`. This patch makes it so by using `getattr` in `log` so those fields do not need to be existed. Differential Revision: https://phab.mercurial-scm.org/D652

revset: move weight information to predicate Previously revset weight is hardcoded and cannot be modified. This patch moves it to predicate so newly registered revsets could define their weight to properly give static optimization some hint. Differential Revision: https://phab.mercurial-scm.org/D657

revset: remove "small" argument from "_optimize" `_optimize` calculates weights of subtrees. "small" affects some weight calculation (either 1 or 0.5). The weights are now only useful in `and` optimization where we might swap two arguments and use `andsmally`. In the real world, it seems unlikely that revsets with weight of 0.5 or 1 matters the `and` order optimization. I think the important thing is to get weights of expensive revsets right (ex. `contains`). This patch removes the `small` argument to simplify the interface. As for choosing between 0.5 vs 1, things returning a single revision (`ancestor`, `string`) has a weight of 0.5. Things returning multiple revisions returns 1. This could be sometimes useful in the `andsmally` optimization, ex. (((:)-2) & expensive()) & ((1-2) & expensive()) ^^^ ^ ^^^^^^^ ^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^ weight=1 weight=0.5 would have an `andsmally` optimization so `1-2` gets executed first, which seems to be desirable. Differential Revision: https://phab.mercurial-scm.org/D656

ui: fix progress debug log format strings to work on Python 3

tests: update `f` helper script to work on Python 3

tests: fix run-tests XML reporting on Python 3 cdatasafe wants to work in terms of bytes, but of course we have a unicode. Easy to work around, especially since we know we'll get utf-8 at the end.

tests: convert bisect support regex to binary The bisection feature of run-tests still fails tests with this because bisect itself doesn't work yet. We'll get there.

tests: decode bytes path to hg command before hitting with shellquote

tests: fix test-is-whitelisted check in run-tests Again, look for bytes in a set of bytes objects.

tests: fix run-tests "slow test" check Look for bytes in a set of bytes objects.