narrow: fix copies._fullcopytracing() narrowspec filtering in graft case I broke this too in 707c3804e607 (narrow: move copies overrides to core, 2018-09-28). Hopefully I'm done fixing things broken by that commit now. Differential Revision: https://phab.mercurial-scm.org/D5213

tests: demonstrate broken copies._fullcopytracing() Turns out copies._fullcopytracing() was also broken. Differential Revision: https://phab.mercurial-scm.org/D5212

narrow: make copies.pathcopies() filter with narrowspec again I broke this in 707c3804e607 (narrow: move copies overrides to core, 2018-09-28). Differential Revision: https://phab.mercurial-scm.org/D5203

tests: demonstrate broken copies.pathcopies() Differential Revision: https://phab.mercurial-scm.org/D5202

setup: explain to distutils how we write rc versions When we use a rc version number (e.g. 4.8rc0), bdist_msi is using distutils.StrictVersion to parse it into a tuple of numbers. By default, StrictVersion.version_re only recognizes [ab] for alpha/beta, where mercurial may use '-rc' or 'rc'. This change makes StrictVersion parse correctly our version numbers, so that bdist_msi doesn't fail on rc versions.

changegroup: restore default node ordering (issue6001) Changeset db5501d9 changed the default node ordering from "storage" to "linearize". While the new API is more explicit and cleaner, the "linearize" order is problematic on certain repositories like netbeans where it makes bundling slower the more nodes we bundle. Pushing and pulling 100 changesets was ~20% slower and pushing and pulling 1000 changesets was ~600% slower. A very quick analysis of profile traces showed that the pull operation was taking more time creating the delta. Putting back the old default order seems to be the safe option. With more time during the next cycle, we can understand better the impact of sorting with the DAG order by default, the source of the regression and how to mitigate it. /!\ We are still waiting for the full performance impact but with this patch, bundling and pulling locally (not on the performance workstation) 1000 changesets on the netbeans repository is as fast as before the regression. Differential Revision: https://phab.mercurial-scm.org/D5196

changegroup: introduce an explicit linear sorting We still need to linearize the revisions in some cases, introduce an explicit `linear` sorting before changing back the default order. Differential Revision: https://phab.mercurial-scm.org/D5195

py3: roll up threading.Thread constructor args into **kwargs The constructor doesn't have a `verbose` keyword argument in py3.

tests: glob over a single quote vs double quote difference on Windows

py3: port test-log-exthook.t to Python 3 For once an easy one. Differential Revision: https://phab.mercurial-scm.org/D5184

extensions: fix up many many debug logs that use %r This gets us a bunch closer on Python 3, but I'll still have to use a ton of sad globs. A previous version of this patch tried to preserve the %r formatting, but upon review Yuya noted that special characters in extension names is very unlikely, so we can just use %s and not sweat the quoting. Differential Revision: https://phab.mercurial-scm.org/D5002

blackbox: add configitem for format of log timestamps Sometimes blackbox logs are used to report performance problems, but the timestamps are only at second granularity, so often the timings have to stated separately by the reporter. This is inconvenient and error-prone, so I would like to include %f in the date format. This patch makes that possible.

narrow: replace filtering in list comprehension by set operations I didn't think of this while reviewing the patch. Differential Revision: https://phab.mercurial-scm.org/D5188

storage: update sqlitestore to use the new `deltamode` parameter While updating the filelog class, I forget to update the sqlitestore, tests are now passing with this patch. Differential Revision: https://phab.mercurial-scm.org/D5185

narrow: rework logic to check whether we need to widen and narrow This patch reworks logic which calculates whether we need to extend or narrow our working copy or not. We filter the addincludes, removeincludes, addexcludes and removeexcludes passed from user to the actual added and removed includes and excludes. What that means is a user can pass an already included path as addincludes, a path which is not included as removeincludes etc. In such situations the old logic use to think we need to do some work, whereas we don't need to do that work. In old logic, even if we don't have anything new to include but it believes we need to call widen, this adds some good amount of work on large repository. A widen calls involves computing incomming csets, calling the narrow_widen() which in non-ellipses cases goes through all the set of csets which are available which can take ~2-3 mins on large repos. Those 2-3 minutes are spend on doing nothing which a client can prevent by checking is there really anything which needs to be included. The tests changes shows that we don't go to the server anymore in such cases which is nice. Differential Revision: https://phab.mercurial-scm.org/D5183

tests: show that adding an already included path still calls narrow_widen() This patch adds tests demonstrating that we still go to the server in non-ellipses widening when we have that path already on the client and there is nothing new to download. The next patch will try to make client side logic smart and not go to the server if we don't need to download anything. Differential Revision: https://phab.mercurial-scm.org/D5182

graft: introduce --base option for using custom base revision while merging The graft command usually performs an internal merge of the current parent revision with the graft revision, using p1 of the grafted revision as base for the merge. As a trivial extension of this, we introduce the --base option to allow for using another base revision. This can be used as a building block for grafting and collapsing multiple changesets at once, or for grafting the resulting change from a merge as a single simple change. (This is kind of similar to backout --parent ... only different: this graft base must be an ancestor, but is usually *not* a parent.) This is probably an advanced use case, and we do thus not show it in the non-verbose help.

changegroup: add a option to create bundle with full snapshot only This is easy to implement now and can be useful for benchmarking.

changegroup: allow to force delta to be against p1 This new developer option is useful to general more "generic" bundle. Without this option, a bundle generated from the repository use deltas similar to the one stored in the specific repository it was generated from. This makes performance testing a bit tricky. Using deltas similar to the final result means all delta stored in the bundle can be applied to the target repository without any further processing (except for the rare case of a full snapshot). The application of such bundles (almost) never exercises the (slower) path of searching for a new valid delta. This result in unrealistic and too favorable timing and profile. Instead, we introduce an option to make sure all revisions are stored as a delta against p1. It might not be the best generation option, but it guarantees that the content will be "generic", not favoring a specific target.

fix: disable use of thread-based worker getfixes() accesses to repo, changectx, filectx, etc., so I believe there are code paths triggering data race. Mercurial API isn't thread safe in general.

storage: also use `deltamode argument` for ifiledata Now that lower level uses such argument, we can propagate the change to higher layers.

configitems: rename the config to prevent adding an alias in future Right now the config option looks like: [experimental.server] stream-narrow-clones= which does not match how config options are generally defined in core. So let's rename this to: [experimental] server.stream-narrow-clones= before the new release so that we don't have to add an alias in future for this. Differential Revision: https://phab.mercurial-scm.org/D5198

sparse-revlog: only refine delta candidates in the sparse case (issue6006) Starting with 5aef5afa8654, a valid delta parent might be "refined". This allows repository using sparse-revlog to produce better delta chain by using better intermediate snapshot base. However, this refining step was performed in all cases, including for repository not using sparse-revlog. This could produce a strange chain in the general delta case and corrupted repository in the non-general delta case. We now skip this step unless sparse-revlog is in use. In issue 6006, Yuya Nishihara provided a test case using an external repository, so we did not include it. Finding "laboratory" condition to reproduce this case and implementing an efficient test reproducing it is a bit tricky. We do not foresee to have the time to provide one by the release date. Differential Revision: https://phab.mercurial-scm.org/D5197

changegroup: refactor emitrevision to use a `deltamode` argument This new argument gathers the semantic of `sendfulltext` and `deltaprevious` in a single value. We are about to introduce a new type of constraints. Avoiding yet another argument sounds like a plus.

http: work around custom http client classes that refuse extra attrs I have no idea what is going on with our custom http client code at Google, but it chokes on these extra attributes we're tucking on http clients. Since it feels more than a little wrong to just stuff extra data on a client, let's degrade gracefully when the client class refuses the attributes.

branchmap: do not specify changelog as an argument Since (unfiltered)repo.changelog lookup gets as fast as __dict__ lookup, there's no point to pass in changelog instance. $ hg perfbranchmap --clear-revbranch -R mozilla-central ! base (orig) wall 20.593091 comb 20.600000 user 20.520000 sys 0.080000 (best of 3) (this) wall 20.129126 comb 20.130000 user 20.020000 sys 0.110000 (best of 3) This backs out most of the changes in 76d4272bd57b and 47c03042cd1d.

filecache: unimplement __set__() and __delete__() (API) Implementing __set__() implies that the descriptor can't be overridden by obj.__dict__, which means any property access involves slow function call. "Data descriptors with __set__() and __get__() defined always override a redefinition in an instance dictionary. In contrast, non-data descriptors can be overridden by instances." https://docs.python.org/2.7/reference/datamodel.html#invoking-descriptors This patch basically backs out 236bb604dc39, "scmutil: update cached copy when filecached attribute is assigned (issue3263)." The problem described in issue3263 (which is #3264 in Bugzilla) should no longer happen since repo._bookmarkcurrent has been moved to repo._bookmarks.active. We still have a risk of introducing similar bugs, but I think that's the cost we have to pay. $ hg perfrevset 'branch(tip)' -R mercurial (orig) wall 0.139511 comb 0.140000 user 0.140000 sys 0.000000 (best of 66) (prev) wall 0.114195 comb 0.110000 user 0.110000 sys 0.000000 (best of 81) (this) wall 0.099038 comb 0.110000 user 0.100000 sys 0.010000 (best of 93)

filecache: use try-except for faster __dict__ lookup Python function call is slow, and the cost could be significant here. $ hg perfrevset 'branch(tip)' -R mercurial (orig) wall 0.139511 comb 0.140000 user 0.140000 sys 0.000000 (best of 66) (this) wall 0.114195 comb 0.110000 user 0.110000 sys 0.000000 (best of 81)

crecord: make nextsametype() check that parent item exists (issue6009) Items that represent files in curses interface don't have parents.

help: describe what ui.tweakdefaults changes, concretely Currently, one has to look at the code. A couple things are suboptimal: - probably not translatable - lines don't get wrapped (a couple are a bit too long) but it seems to better this way than without help at all. Differential Revision: https://phab.mercurial-scm.org/D5187