setdiscovery: don't use dagutil for node -> rev conversion The node -> rev conversion is possible using standard storage APIs and doesn't need to involve the dagutil module. Differential Revision: https://phab.mercurial-scm.org/D4306

dagutil: remove externalize() and externalizeall() They are unused after the previous commit. .. api:: externalize() and externalizeall() removed from dagutil Use .node() on a storage primitive to perform revision to node conversions. Differential Revision: https://phab.mercurial-scm.org/D4305

setdiscovery: don't use dagutil for rev -> node conversions We don't need to use dagutil to perform a simple rev -> node conversion. I haven't measured, but the new code is likely faster, as we avoid extra function calls and avoid some attribute lookups. Differential Revision: https://phab.mercurial-scm.org/D4304

exchange: don't use dagutil We were only using it for simple node -> rev and parent revision lookups. These are exposed via the storage interface and we don't need to go through dagutil. Differential Revision: https://phab.mercurial-scm.org/D4303

revlog: only consider the span of the delta section Since the number of snapshots is limited we can exclude them from the logic checking size and number of reads. Limiting the span computation to the delta section will allow for further optimization.

revlog: ensure intermediate snapshot have decreasing size If the intermediate snapshot is bigger than the previous one, there is likely a better snapshot to be made at a different level.

revlog: bound number of snapshots in a chain To limit the number of snapshot chained, we enforce them to be smaller and smaller. This guarantee the number of snapshot in a chain will be bounded to a small number.

revlog: compute snapshot depth on delta info We need the information to be available when choosing delta.

debugrevlog: display snapshot details per depth This help in understanding the final structure of build manifest. All data about snapshot (full and intermediate) are gathered into a sub-list for clarity. Since we do not produce such snapshots yet, the only thing changing in test output is the way the information is presented.

revlog: add a method to retrieve snapshot depth Some snapshot property (eg: maximum size) will depend on their depth.

debugrevlog: include information about intermediate snapshots As we are about to create intermediate snapshots, we need to have a way to debug them. We start by adding very simple debug output and more detailed output will comes in next changesets.

revlog: also detect intermediate snapshots Also detect intermediate-snapshot done against another previous snapshot. Doing an intermediate snapshot instead of a full one can reduce the number of full snapshots we need. They are especially useful for content with a lot of churn on the same line (eg: the manifest) where having a delta over multiple revisions can end up being significantly smaller than the sum of these revision deltas. A revlog built using intermediate snapshots can be a bit smaller and reuse snapshot much more efficiently. This last property is useful combined with constraints on chain length. Using intermediate snapshot can produce repository with delta chain ten times shorter without impact on the storage size. Shorter chain lengths are faster to restore, greatly improving read performance. This changesets (and the following ones) focus on getting the core principle of intermediate snapshots into Mercurial core. Later changeset will introduce the strategy to create them.

revlog: add a method to tells whether rev is stored as a snapshot For now we only have one type of snapshot: full snapshot versus nullrev. However we are looking into adding intermediate snapshot where a large diff against another snapshot is performed instead of storing a full new text. The conditional is a bit strange and is done in order to help readability of a some later changesets.

debugrevlog: fix for non-manifest object The `filelog` object is no longer an actual revlog. Instead, the actual revlog is stored in the `_revlog` attribute.

merge with stable

remotephase: avoid full changelog iteration (issue5964) Changeset 88efb7d6bcb6 introduced a performance regression by triggering a full ancestors walk. This changeset reworks this logic so that we no longer walk down the full changelog. The motivation for 88efb7d6bcb6, issue5939, is still fixed. mercurial compared to a draft repository ---------------------------------------- 8eeed92475d5: 0.012637 seconds 88efb7d6bcb6: 0.202699 seconds (x16) 46da52f4b820: 0.215551 seconds (+6%) this code: 0.008397 seconds (-33% from base) The payload size reduction we see in `test-bookmarks-pushpull.t` comes from a more aggressive filter of nullid and is harmless.

nodes: expand/comment the magic nodes so they are more easily searchable We just encountered `000000000000modified`, and it was quite annoying to search for these, even though I knew they existed. For those that don't know that they exist, this is essentially impossible to search for :) (Technically we encountered it in its hex form, 3030303030303030303030306d6f646966696564, so I'm adding comments with those forms in case that's helpful to people in the future). Differential Revision: https://phab.mercurial-scm.org/D4331

revlog: obtain the first node at the lowest layer while building pure nodemap Just for clarity. This doesn't matter in practice since changelog.nodemap is accessed *before* filtered revisions get ready.

revlog: fix pure nodemap to not access missing index entry This bug was revealed by a3dacabd476b and a1f934573c0b.

changelog: remove copy of revlog.nodemap() It's been there since 2012, "clfilter: introduce `filteredrevs` attribute on changelog." I don't think we can apply changelog filtering to nodemap at this level, so this patch removes the nodemap stub completely.

remotephase: fast path newheads computation in simple case (issue5964) Changeset 88efb7d6bcb6 fixed the logic of `phases.newheads` but greatly regressed its performance (up to many order of magnitude). The first step to fix the regression is to exit early when there is no work to do. If there are no heads to filter or not roots to filter them, we don't have to do any work. This fixes the regression when talking to an all public changeset. The performance is even better than before. pypy, compared to an all public repo ------------------------------------ 8eeed92475d5: 0.005758 seconds 88efb7d6bcb6: 0.602517 seconds (x104) this code: 0.001508 seconds (-74% from base) mercurial compared to an all public repo ---------------------------------------- 8eeed92475d5: 0.000577 seconds 88efb7d6bcb6: 0.185316 seconds (x321) this code: 0.000150 seconds (-74% from base) The performance of newheads, when actual computations are required, is fixed in the next changeset.

perf: add a perfphasesremote command This command measure the time we spend analysing remote phase during push and pull and display some information relevant to this computation. The `test-contrib-perf.t` expected output has to be updated but I do need these module for this perf command.

sparse-revlog: fix delta validity computation When considering the validity of a delta with sparse-revlog, we check the size of the largest read. To do so, we use some regular logic with the extra delta information. Some of this logic was not handling this extra delta properly, confusing it with "nullrev". This confusion with nullrev lead to wrong results for this computation but preventing a crash. Changeset 781b2720d2ac on default revealed this error, crashing. This changeset fixes the logic on stable so that the computation is correct (and the crash is averted). The fix is made on stable as this will impact 4.7 clients interacting with sparse-revlog repositories (eg: created by later version).

branchmap: explicitly convert file into iterator Follows up 2a4bfbb52111. This is required for httprangereader, which is not an iterable itself.

branchmap: close cache file properly Follows up 2a4bfbb52111.

branchmap: strip '\n' read from cache file as before Follows up 2a4bfbb52111.

rebase: do not pass in user option to rollback in-memory merge conflict Nothing passed before e9e742bd0501.

rebase: add test for in-memory merge conflicts