Mercurial > hg
view tests/test-narrow-exchange.t @ 44363:f7459da77f23
nodemap: introduce an option to use mmap to read the nodemap mapping
The performance and memory benefit is much greater if we don't have to copy all
the data in memory for each information. So we introduce an option (on by
default) to read the data using mmap.
This changeset is the last one definition the API for index support nodemap
data. (they have to be able to use the mmaping).
Below are some benchmark comparing the best we currently have in 5.3 with the
final step of this series (using the persistent nodemap implementation in
Rust). The benchmark run `hg perfindex` with various revset and the following
variants:
Before:
* do not use the persistent nodemap
* use the CPython implementation of the index for nodemap
* use mmapping of the changelog index
After:
* use the MixedIndex Rust code, with the NodeTree object for nodemap access
(still in review)
* use the persistent nodemap data from disk
* access the persistent nodemap data through mmap
* use mmapping of the changelog index
The persistent nodemap greatly speed up most operation on very large
repositories. Some of the previously very fast lookup end up a bit slower because
the persistent nodemap has to be setup. However the absolute slowdown is very
small and won't matters in the big picture.
Here are some numbers (in seconds) for the reference copy of mozilla-try:
Revset Before After abs-change speedup
-10000: 0.004622 0.005532 0.000910 × 0.83
-10: 0.000050 0.000132 0.000082 × 0.37
tip 0.000052 0.000085 0.000033 × 0.61
0 + (-10000:) 0.028222 0.005337 -0.022885 × 5.29
0 0.023521 0.000084 -0.023437 × 280.01
(-10000:) + 0 0.235539 0.005308 -0.230231 × 44.37
(-10:) + :9 0.232883 0.000180 -0.232703 ×1293.79
(-10000:) + (:99) 0.238735 0.005358 -0.233377 × 44.55
:99 + (-10000:) 0.317942 0.005593 -0.312349 × 56.84
:9 + (-10:) 0.313372 0.000179 -0.313193 ×1750.68
:9 0.316450 0.000143 -0.316307 ×2212.93
On smaller repositories, the cost of nodemap related operation is not as big, so
the win is much more modest. Yet it helps shaving a handful of millisecond here
and there.
Here are some numbers (in seconds) for the reference copy of mercurial:
Revset Before After abs-change speedup
-10: 0.000065 0.000097 0.000032 × 0.67
tip 0.000063 0.000078 0.000015 × 0.80
0 0.000561 0.000079 -0.000482 × 7.10
-10000: 0.004609 0.003648 -0.000961 × 1.26
0 + (-10000:) 0.005023 0.003715 -0.001307 × 1.35
(-10:) + :9 0.002187 0.000108 -0.002079 ×20.25
(-10000:) + 0 0.006252 0.003716 -0.002536 × 1.68
(-10000:) + (:99) 0.006367 0.003707 -0.002660 × 1.71
:9 + (-10:) 0.003846 0.000110 -0.003736 ×34.96
:9 0.003854 0.000099 -0.003755 ×38.92
:99 + (-10000:) 0.007644 0.003778 -0.003866 × 2.02
Differential Revision: https://phab.mercurial-scm.org/D7894
author | Pierre-Yves David <pierre-yves.david@octobus.net> |
---|---|
date | Tue, 11 Feb 2020 11:18:52 +0100 |
parents | d7304434390f |
children | 95c4cca641f6 |
line wrap: on
line source
#testcases lfs-on lfs-off #if lfs-on $ cat >> $HGRCPATH <<EOF > [extensions] > lfs = > EOF #endif $ . "$TESTDIR/narrow-library.sh" create full repo $ hg init master $ cd master $ cat >> .hg/hgrc <<EOF > [narrow] > serveellipses=True > EOF $ mkdir inside $ echo 1 > inside/f $ mkdir inside2 $ echo 1 > inside2/f $ mkdir outside $ echo 1 > outside/f $ hg ci -Aqm 'initial' $ echo 2 > inside/f $ hg ci -qm 'inside 2' $ echo 2 > inside2/f $ hg ci -qm 'inside2 2' $ echo 2 > outside/f $ hg ci -qm 'outside 2' $ cd .. $ hg clone --narrow ssh://user@dummy/master narrow --include inside requesting all changes adding changesets adding manifests adding file changes added 3 changesets with 2 changes to 1 files new changesets *:* (glob) updating to branch default 1 files updated, 0 files merged, 0 files removed, 0 files unresolved $ hg clone --narrow ssh://user@dummy/master narrow2 --include inside --include inside2 requesting all changes adding changesets adding manifests adding file changes added 4 changesets with 4 changes to 2 files new changesets *:* (glob) updating to branch default 2 files updated, 0 files merged, 0 files removed, 0 files unresolved Can push to wider repo if change does not affect paths in wider repo that are not also in narrower repo $ cd narrow $ echo 3 > inside/f $ hg ci -m 'inside 3' $ hg push ssh://user@dummy/narrow2 pushing to ssh://user@dummy/narrow2 searching for changes remote: adding changesets remote: adding manifests remote: adding file changes remote: added 1 changesets with 1 changes to 1 files Can push to narrower repo if change affects only paths within remote's narrow spec $ cd ../narrow2 $ cat >> .hg/hgrc <<EOF > [narrow] > serveellipses=True > EOF $ hg co -r 'desc("inside 3")' 1 files updated, 0 files merged, 0 files removed, 0 files unresolved $ echo 4 > inside/f $ hg ci -m 'inside 4' $ hg push ssh://user@dummy/narrow pushing to ssh://user@dummy/narrow searching for changes remote: adding changesets remote: adding manifests remote: adding file changes remote: added 1 changesets with 1 changes to 1 files Can push to narrow repo if change affects only paths outside remote's narrow spec $ echo 3 > inside2/f $ hg ci -m 'inside2 3' TODO: this should be successful $ hg push ssh://user@dummy/narrow pushing to ssh://user@dummy/narrow searching for changes remote: adding changesets remote: adding manifests remote: adding file changes remote: transaction abort! remote: rollback completed remote: abort: data/inside2/f.i@4a1aa07735e6: unknown parent! (reporevlogstore !) remote: abort: data/inside2/f/index@4a1aa07735e6: no node! (reposimplestore !) abort: stream ended unexpectedly (got 0 bytes, expected 4) [255] Can pull from wider repo if change affects only paths outside remote's narrow spec $ echo 4 > inside2/f $ hg ci -m 'inside2 4' $ hg log -G -T '{rev} {node|short} {files}\n' @ 7 d78a96df731d inside2/f | o 6 8c26f5218962 inside2/f | o 5 ba3480e2f9de inside/f | o 4 4e5edd526618 inside/f | o 3 81e7e07b7ab0 outside/f | o 2 f3993b8c0c2b inside2/f | o 1 8cd66ca966b4 inside/f | o 0 c8057d6f53ab inside/f inside2/f outside/f $ cd ../narrow $ hg log -G -T '{rev} {node|short} {files}\n' o 4 ba3480e2f9de inside/f | @ 3 4e5edd526618 inside/f | o 2 81e7e07b7ab0 outside/f | o 1 8cd66ca966b4 inside/f | o 0 c8057d6f53ab inside/f inside2/f outside/f $ hg pull ssh://user@dummy/narrow2 pulling from ssh://user@dummy/narrow2 searching for changes adding changesets adding manifests adding file changes added 1 changesets with 0 changes to 0 files new changesets d78a96df731d (run 'hg update' to get a working copy) Check that the resulting history is valid in the full repo $ cd ../narrow2 $ hg push ssh://user@dummy/master pushing to ssh://user@dummy/master searching for changes remote: adding changesets remote: adding manifests remote: adding file changes remote: added 4 changesets with 4 changes to 2 files $ cd ../master $ hg verify checking changesets checking manifests crosschecking files in changesets and manifests checking files checked 8 changesets with 10 changes to 3 files Can not push to wider repo if change affects paths in wider repo that are not also in narrower repo $ cd ../master $ hg co -r 'desc("inside2 4")' 2 files updated, 0 files merged, 0 files removed, 0 files unresolved $ echo 5 > inside2/f $ hg ci -m 'inside2 5' $ hg log -G -T '{rev} {node|short} {files}\n' @ 8 5970befb64ba inside2/f | o 7 d78a96df731d inside2/f | o 6 8c26f5218962 inside2/f | o 5 ba3480e2f9de inside/f | o 4 4e5edd526618 inside/f | o 3 81e7e07b7ab0 outside/f | o 2 f3993b8c0c2b inside2/f | o 1 8cd66ca966b4 inside/f | o 0 c8057d6f53ab inside/f inside2/f outside/f $ cd ../narrow $ hg pull pulling from ssh://user@dummy/master searching for changes adding changesets adding manifests adding file changes added 1 changesets with 0 changes to 0 files new changesets * (glob) (run 'hg update' to get a working copy) TODO: this should tell the user that their narrow clone does not have the necessary content to be able to push to the target TODO: lfs shouldn't abort like this $ hg push ssh://user@dummy/narrow2 || true pushing to ssh://user@dummy/narrow2 searching for changes remote: adding changesets remote: adding manifests remote: adding file changes remote: added 1 changesets with 0 changes to 0 files (no-lfs-on !) remote: error: pretxnchangegroup.lfs hook raised an exception: data/inside2/f.i@f59b4e021835: no match found (lfs-on !) remote: transaction abort! (lfs-on !) remote: rollback completed (lfs-on !) remote: abort: data/inside2/f.i@f59b4e021835: no match found! (lfs-on !) abort: stream ended unexpectedly (got 0 bytes, expected 4) (lfs-on !)