Mercurial > hg
view tests/test-sparse-revlog.t @ 39492:a33f394b2bfd
snapshot: try intermediate snapshot against parents' base
# Regarding The Series Started By This Changeset
This is the first changesets of a group adjusting delta chain strategy to
build a useful chain of intermediate snapshots. The series will introduce a
full strategy to produce chains of multiple snapshots on top of which a
"usual" delta chain will be built.
That strategy will have multiple steps to maximize snapshot reuse, avoiding
pathological cases and improving overall compression in very branchy
repositories. An important property of sparse-revlog using such snapshot-chain
is that they can use very short delta chain without problematic impact on the
resulting compression. Shorter delta chains are important to achieve good
performance.
To make each step clear, we'll introduce them one by one.
See the end of this series for full details.
# Regarding This Changeset
Before this change, if we cannot store the current revision as a delta against
a "simple" candidate (p1, p2, prev), we created a new level-0 snapshot (also
called full snapshot).
As the first step, we introduce a simple strategy: try an intermediate level-1
snapshot against the chain base of the "current revision" parents.
The "current revision" is the one we are currently trying to store in the
revlog, triggering this search for a good delta base.
The first item in the chain is always a level-0 snapshot.
# Effect On The Test Repository
We can already see the effect on the test-repository. Most of the snapshots
have shifted from level 0 to level 1. The overall size has slightly decreased.
(However, keep in mind that this repository only emulates real data)
# Regarding Statistic
The current series focuses on improving the chain built. Improving the
performance of this logic will be done as a second step. Sparse-revlog is
still experimental and disabled by default.
We'll provide more statistic about resulting size and delta chain at the end
of this series.
author | Boris Feld <boris.feld@octobus.net> |
---|---|
date | Fri, 07 Sep 2018 11:17:29 -0400 |
parents | 4ca7a67c94c8 |
children | 3ca144f1c8dd |
line wrap: on
line source
==================================== Test delta choice with sparse revlog ==================================== Sparse-revlog usually shows the most gain on Manifest. However, it is simpler to general an appropriate file, so we test with a single file instead. The goal is to observe intermediate snapshot being created. We need a large enough file. Part of the content needs to be replaced repeatedly while some of it changes rarely. $ bundlepath="$TESTDIR/artifacts/cache/big-file-churn.hg" $ expectedhash=`cat "$bundlepath".md5` $ if [ ! -f "$bundlepath" ]; then > echo 'skipped: missing artifact, run "'"$TESTDIR"'/artifacts/scripts/generate-churning-bundle.py"' > exit 80 > fi $ currenthash=`f -M "$bundlepath" | cut -d = -f 2` $ if [ "$currenthash" != "$expectedhash" ]; then > echo 'skipped: outdated artifact, md5 "'"$currenthash"'" expected "'"$expectedhash"'" run "'"$TESTDIR"'/artifacts/scripts/generate-churning-bundle.py"' > exit 80 > fi $ cat >> $HGRCPATH << EOF > [format] > sparse-revlog = yes > [storage] > revlog.optimize-delta-parent-choice = yes > EOF $ hg init sparse-repo $ cd sparse-repo $ hg unbundle $bundlepath adding changesets adding manifests adding file changes added 5001 changesets with 5001 changes to 1 files (+89 heads) new changesets 9706f5af64f4:d9032adc8114 (5001 drafts) (run 'hg heads' to see heads, 'hg merge' to merge) $ hg up 1 files updated, 0 files merged, 0 files removed, 0 files unresolved updated to "d9032adc8114: commit #5000" 89 other heads for branch "default" $ hg log --stat -r 0:3 changeset: 0:9706f5af64f4 user: test date: Thu Jan 01 00:00:00 1970 +0000 summary: initial commit SPARSE-REVLOG-TEST-FILE | 10500 ++++++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 10500 insertions(+), 0 deletions(-) changeset: 1:724907deaa5e user: test date: Thu Jan 01 00:00:00 1970 +0000 summary: commit #1 SPARSE-REVLOG-TEST-FILE | 1068 +++++++++++++++++++++++----------------------- 1 files changed, 534 insertions(+), 534 deletions(-) changeset: 2:62c41bce3e5d user: test date: Thu Jan 01 00:00:00 1970 +0000 summary: commit #2 SPARSE-REVLOG-TEST-FILE | 1068 +++++++++++++++++++++++----------------------- 1 files changed, 534 insertions(+), 534 deletions(-) changeset: 3:348a9cbd6959 user: test date: Thu Jan 01 00:00:00 1970 +0000 summary: commit #3 SPARSE-REVLOG-TEST-FILE | 1068 +++++++++++++++++++++++----------------------- 1 files changed, 534 insertions(+), 534 deletions(-) $ f -s .hg/store/data/*.d .hg/store/data/_s_p_a_r_s_e-_r_e_v_l_o_g-_t_e_s_t-_f_i_l_e.d: size=72315280 $ hg debugrevlog * format : 1 flags : generaldelta revisions : 5001 merges : 625 (12.50%) normal : 4376 (87.50%) revisions : 5001 empty : 0 ( 0.00%) text : 0 (100.00%) delta : 0 (100.00%) snapshot : 145 ( 2.90%) lvl-0 : 15 ( 0.30%) lvl-1 : 130 ( 2.60%) deltas : 4856 (97.10%) revision size : 72315280 snapshot : 18481085 (25.56%) lvl-0 : 3016019 ( 4.17%) lvl-1 : 15465066 (21.39%) deltas : 53834195 (74.44%) chunks : 5001 0x78 (x) : 5001 (100.00%) chunks size : 72315280 0x78 (x) : 72315280 (100.00%) avg chain length : 18 max chain length : 45 max chain reach : 32095083 compression ratio : 23 uncompressed data size (min/max/avg) : 346468 / 346472 / 346471 full revision size (min/max/avg) : 200990 / 201151 / 201067 inter-snapshot size (min/max/avg) : 37202 / 173034 / 118962 level-1 (min/max/avg) : 37202 / 173034 / 118962 delta size (min/max/avg) : 10649 / 104791 / 11086 deltas against prev : 4185 (86.18%) where prev = p1 : 4139 (98.90%) where prev = p2 : 0 ( 0.00%) other : 46 ( 1.10%) deltas against p1 : 647 (13.32%) deltas against p2 : 24 ( 0.49%) deltas against other : 0 ( 0.00%)