bundle: include advisory rev branch cache part in bundle2 bundle
`hg bundle` command producing bundle2 will now include an optional part
containing the revision-branch cache data.
The data sent are mostly nodes so it is quite compact. The goal of the
rev-branch-cache is to speed up branch map computation, especially when the
branchmap gets invalidated so we send data for all exchanged changesets. In
addition, computing the relevant heads to send in case of partial pulling would
be challenging.
As a reminder, the rev branch cache data significantly speed up branch
computation. Having it around provides a small speedup to pull/clone and much
higher tolerance to branch map cache invalidation that might happens from later
commands.
On the Mercurial repository, computing the visible branchmap from scratch move
from 2.00 seconds to 0.34s (a -83% speedup).
Using this new part, Unbundling the full Mercurial repository moves from 25.736
seconds to 24.030 seconds (around -7% speedup). The bundle size increase is
around 3% (from 22.43 MB to 23.13MB)
On an half a million revision repository with twenty thousand
branches, computing the branchmap moves from 75 seconds to 45 second (-40%) if
the caches is used.
A bundle containing 50 000 changesets in such repository get a 0.5% size
increase from such part for a -3% unbundling time speedup.
--- a/mercurial/bundle2.py Wed Feb 21 17:26:22 2018 +0100
+++ b/mercurial/bundle2.py Wed Feb 21 17:33:00 2018 +0100
@@ -1591,6 +1591,7 @@
part.addparam('targetphase', '%d' % phases.secret, mandatory=False)
addparttagsfnodescache(repo, bundler, outgoing)
+ addpartrevbranchcache(repo, bundler, outgoing)
if opts.get('obsolescence', False):
obsmarkers = repo.obsstore.relevantmarkers(outgoing.missing)
--- a/tests/test-bundle-phases.t Wed Feb 21 17:26:22 2018 +0100
+++ b/tests/test-bundle-phases.t Wed Feb 21 17:33:00 2018 +0100
@@ -42,6 +42,7 @@
26805aba1e600a82e93661149f2313866a221a7b
f585351a92f85104bff7c284233c338b10eb1df7
9bc730a19041f9ec7cb33c626e811aa233efb18c
+ cache:rev-branch-cache -- {}
phase-heads -- {}
26805aba1e600a82e93661149f2313866a221a7b draft
$ hg strip --no-backup C
@@ -233,6 +234,7 @@
dc0947a82db884575bb76ea10ac97b08536bfa03
4e4f9194f9f181c57f62e823e8bdfa46ab9e4ff4
03ca77807e919db8807c3749086dc36fb478cac0
+ cache:rev-branch-cache -- {}
phase-heads -- {}
dc0947a82db884575bb76ea10ac97b08536bfa03 public
03ca77807e919db8807c3749086dc36fb478cac0 draft
@@ -258,6 +260,7 @@
changegroup -- {nbchanges: 2, targetphase: 2, version: 02}
112478962961147124edd43549aedd1a335e44bf
4e4f9194f9f181c57f62e823e8bdfa46ab9e4ff4
+ cache:rev-branch-cache -- {}
phase-heads -- {}
$ rm bundle
@@ -269,6 +272,7 @@
112478962961147124edd43549aedd1a335e44bf
dc0947a82db884575bb76ea10ac97b08536bfa03
4e4f9194f9f181c57f62e823e8bdfa46ab9e4ff4
+ cache:rev-branch-cache -- {}
phase-heads -- {}
dc0947a82db884575bb76ea10ac97b08536bfa03 public
$ rm bundle
@@ -280,6 +284,7 @@
changegroup -- {nbchanges: 2, targetphase: 2, version: 02}
4e4f9194f9f181c57f62e823e8bdfa46ab9e4ff4
03ca77807e919db8807c3749086dc36fb478cac0
+ cache:rev-branch-cache -- {}
phase-heads -- {}
03ca77807e919db8807c3749086dc36fb478cac0 draft
$ rm bundle
--- a/tests/test-bundle-type.t Wed Feb 21 17:26:22 2018 +0100
+++ b/tests/test-bundle-type.t Wed Feb 21 17:33:00 2018 +0100
@@ -76,6 +76,7 @@
Stream params: {}
changegroup -- {nbchanges: 1, version: 02}
c35a0f9217e65d1fdb90c936ffa7dbe679f83ddf
+ cache:rev-branch-cache -- {}
none-v2
% test bundle type bzip2
@@ -85,6 +86,7 @@
Stream params: {Compression: BZ}
changegroup -- {nbchanges: 1, version: 02}
c35a0f9217e65d1fdb90c936ffa7dbe679f83ddf
+ cache:rev-branch-cache -- {}
bzip2-v2
% test bundle type gzip
@@ -94,6 +96,7 @@
Stream params: {Compression: GZ}
changegroup -- {nbchanges: 1, version: 02}
c35a0f9217e65d1fdb90c936ffa7dbe679f83ddf
+ cache:rev-branch-cache -- {}
gzip-v2
% test bundle type none-v2
@@ -103,6 +106,7 @@
Stream params: {}
changegroup -- {nbchanges: 1, version: 02}
c35a0f9217e65d1fdb90c936ffa7dbe679f83ddf
+ cache:rev-branch-cache -- {}
none-v2
% test bundle type v2
@@ -112,6 +116,7 @@
Stream params: {Compression: BZ}
changegroup -- {nbchanges: 1, version: 02}
c35a0f9217e65d1fdb90c936ffa7dbe679f83ddf
+ cache:rev-branch-cache -- {}
bzip2-v2
% test bundle type v1
@@ -150,12 +155,12 @@
$ hg bundle -a -t gzip-v2 gzip-v2.hg
1 changesets found
$ f --size gzip-v2.hg
- gzip-v2.hg: size=427
+ gzip-v2.hg: size=468
$ hg --config experimental.bundlecomplevel=1 bundle -a -t gzip-v2 gzip-v2-level1.hg
1 changesets found
$ f --size gzip-v2-level1.hg
- gzip-v2-level1.hg: size=435
+ gzip-v2-level1.hg: size=475
$ cd ..
@@ -171,6 +176,7 @@
Stream params: {Compression: ZS}
changegroup -- {nbchanges: 1, version: 02}
c35a0f9217e65d1fdb90c936ffa7dbe679f83ddf
+ cache:rev-branch-cache -- {}
zstd-v2
% test bundle type zstd-v2
@@ -180,6 +186,7 @@
Stream params: {Compression: ZS}
changegroup -- {nbchanges: 1, version: 02}
c35a0f9217e65d1fdb90c936ffa7dbe679f83ddf
+ cache:rev-branch-cache -- {}
zstd-v2
--- a/tests/test-bundle.t Wed Feb 21 17:26:22 2018 +0100
+++ b/tests/test-bundle.t Wed Feb 21 17:33:00 2018 +0100
@@ -774,7 +774,7 @@
list of changesets:
1a38c1b849e8b70c756d2d80b0b9a3ac0b7ea11a
057f4db07f61970e1c11e83be79e9d08adc4dc31
- bundle2-output-bundle: "HG20", (1 params) 1 parts total
+ bundle2-output-bundle: "HG20", (1 params) 2 parts total
bundle2-output-part: "changegroup" (params: 1 mandatory 1 advisory) streamed payload
bundling: 1/2 changesets (50.00%)
bundling: 2/2 changesets (100.00%)
@@ -783,6 +783,7 @@
bundling: b 1/3 files (33.33%)
bundling: b1 2/3 files (66.67%)
bundling: x 3/3 files (100.00%)
+ bundle2-output-part: "cache:rev-branch-cache" streamed payload
== Test for issue3441
--- a/tests/test-clonebundles.t Wed Feb 21 17:26:22 2018 +0100
+++ b/tests/test-clonebundles.t Wed Feb 21 17:33:00 2018 +0100
@@ -163,7 +163,7 @@
by old clients.
$ f --size --hexdump full.hg
- full.hg: size=396
+ full.hg: size=442
0000: 48 47 32 30 00 00 00 0e 43 6f 6d 70 72 65 73 73 |HG20....Compress|
0010: 69 6f 6e 3d 47 5a 78 9c 63 60 60 d0 e4 76 f6 70 |ion=GZx.c``..v.p|
0020: f4 73 77 75 0f f2 0f 0d 60 00 02 46 46 76 26 4e |.swu....`..FFv&N|
@@ -188,7 +188,10 @@
0150: 88 75 34 36 75 04 82 55 17 14 36 a4 38 10 04 d8 |.u46u..U..6.8...|
0160: 21 01 9a b1 83 f7 e9 45 8b d2 56 c7 a3 1f 82 52 |!......E..V....R|
0170: d7 8a 78 ed fc d5 76 f1 36 25 81 89 c7 ad ec 90 |..x...v.6%......|
- 0180: 54 47 75 2b 89 49 b1 00 d2 8a eb 92 |TGu+.I......|
+ 0180: 54 47 75 2b 89 48 b1 b2 62 ce 8e ce 1e ae 56 41 |TGu+.H..b.....VA|
+ 0190: ae 61 ba 4e 41 8e 7e ce 1e ba 60 01 a0 14 23 58 |.a.NA.~...`...#X|
+ 01a0: 81 35 c8 7d 40 cc 04 e2 a4 a4 a6 25 96 e6 94 60 |.5.}@......%...`|
+ 01b0: 33 17 5f 54 00 00 01 1b 0a ec |3._T......|
$ echo "http://localhost:$HGPORT1/full.hg" > server/.hg/clonebundles.manifest
$ hg clone -U http://localhost:$HGPORT full-bundle
@@ -529,14 +532,14 @@
$ cat hg.pid >> $DAEMON_PIDS
$ hg -R server debuglfput gz-a.hg
- f6eca29e25359f6a92f1ea64527cdcf1b5abe62a
+ 14ee2f0b3f1d14aeeb2fe037e09fc295c3cf59f5
$ cat > server/.hg/clonebundles.manifest << EOF
- > largefile://f6eca29e25359f6a92f1ea64527cdcf1b5abe62a BUNDLESPEC=gzip-v2
+ > largefile://14ee2f0b3f1d14aeeb2fe037e09fc295c3cf59f5 BUNDLESPEC=gzip-v2
> EOF
$ hg clone -U http://localhost:$HGPORT largefile-provided --traceback
- applying clone bundle from largefile://f6eca29e25359f6a92f1ea64527cdcf1b5abe62a
+ applying clone bundle from largefile://14ee2f0b3f1d14aeeb2fe037e09fc295c3cf59f5
adding changesets
adding manifests
adding file changes
--- a/tests/test-debugbundle.t Wed Feb 21 17:26:22 2018 +0100
+++ b/tests/test-debugbundle.t Wed Feb 21 17:33:00 2018 +0100
@@ -34,12 +34,14 @@
changegroup -- {nbchanges: 2, version: 02}
0e067c57feba1a5694ca4844f05588bb1bf82342
991a3460af53952d10ec8a295d3d2cc2e5fa9690
+ cache:rev-branch-cache -- {}
Quiet output
$ hg debugbundle --quiet bundle2.hg
Stream params: {}
changegroup -- {nbchanges: 2, version: 02}
+ cache:rev-branch-cache -- {}
Verbose output:
@@ -78,5 +80,6 @@
c
b80de5d138758541c5f05265ad144ab9fa86d1db 0000000000000000000000000000000000000000 0000000000000000000000000000000000000000 991a3460af53952d10ec8a295d3d2cc2e5fa9690 0000000000000000000000000000000000000000 0
+ cache:rev-branch-cache -- {}
$ cd ..
--- a/tests/test-generaldelta.t Wed Feb 21 17:26:22 2018 +0100
+++ b/tests/test-generaldelta.t Wed Feb 21 17:33:00 2018 +0100
@@ -159,6 +159,7 @@
Stream params: {Compression: BZ}
changegroup -- {nbchanges: 1, version: 02}
1c5d4dc9a8b8d6e1750966d343e94db665e7a1e9
+ cache:rev-branch-cache -- {}
phase-heads -- {}
1c5d4dc9a8b8d6e1750966d343e94db665e7a1e9 draft
--- a/tests/test-obsolete-changeset-exchange.t Wed Feb 21 17:26:22 2018 +0100
+++ b/tests/test-obsolete-changeset-exchange.t Wed Feb 21 17:33:00 2018 +0100
@@ -95,10 +95,12 @@
Stream params: {Compression: BZ}
changegroup -- {nbchanges: 1, version: 02}
f89bcc95eba5174b1ccc3e33a82e84c96e8338ee
+ cache:rev-branch-cache -- {}
$ hg debugbundle ../f89bcc95eba5-obs.hg
Stream params: {Compression: BZ}
changegroup -- {nbchanges: 1, version: 02}
f89bcc95eba5174b1ccc3e33a82e84c96e8338ee
+ cache:rev-branch-cache -- {}
obsmarkers -- {}
version: 1 (70 bytes)
9d73aac1b2ed7d53835eaeec212ed41ea47da53a f89bcc95eba5174b1ccc3e33a82e84c96e8338ee 0 (Thu Jan 01 00:00:00 1970 +0000) {'user': 'test'}
--- a/tests/test-obsolete.t Wed Feb 21 17:26:22 2018 +0100
+++ b/tests/test-obsolete.t Wed Feb 21 17:33:00 2018 +0100
@@ -1427,6 +1427,7 @@
Stream params: {Compression: BZ}
changegroup -- {nbchanges: 1, version: 02}
e008cf2834908e5d6b0f792a9d4b0e2272260fb8
+ cache:rev-branch-cache -- {}
phase-heads -- {}
e008cf2834908e5d6b0f792a9d4b0e2272260fb8 draft
@@ -1468,6 +1469,7 @@
changegroup -- {nbchanges: 2, version: 02}
e016b03fd86fcccc54817d120b90b751aaf367d6
b0551702f918510f01ae838ab03a463054c67b46
+ cache:rev-branch-cache -- {}
obsmarkers -- {}
version: 1 (92 bytes)
e008cf2834908e5d6b0f792a9d4b0e2272260fb8 b0551702f918510f01ae838ab03a463054c67b46 0 (Thu Jan 01 00:00:00 1970 +0000) {'ef1': '8', 'operation': 'amend', 'user': 'test'}
--- a/tests/test-patchbomb.t Wed Feb 21 17:26:22 2018 +0100
+++ b/tests/test-patchbomb.t Wed Feb 21 17:33:00 2018 +0100
@@ -390,14 +390,15 @@
Content-Disposition: attachment; filename="bundle.hg"
Content-Transfer-Encoding: base64
- SEcyMAAAAA5Db21wcmVzc2lvbj1CWkJaaDkxQVkmU1nYvy2xAAAJf//7vFYQXD1/4H7R09C/470I
- Ak0E4pe4SIIIgQSgGEQOcLAA5VBKqeppMxTI0YjQNBgQMQDI0GgMhtR6I0GI2p6I0yeSEVT9MiYn
- qjCYQwCBtARptARgBNDEwAGiDCMA40NGjQaNA0AAAAADIAAAA0BkABktCk6qObVxZ2A/33KHibLr
- UQ4BwkgcPcmuCUAQZCztIWgR1SpBS6IqbIij4UFwhnnMkElcFTqoucIWbsBPK3l+6c+xYaVBWsJo
- aT0OV/YAOvLrziifDQMJOMIaaYce9agtI2EwQBAq089UiRU+evFHSLRBT7Wa/D/YBaUtU5ezvtr3
- 6yrIS4Iyp9VWESdWPEi6VjRjdcEY4HvbmDIVEAEVJIUrHNIBx/MmnBBRkw8tSlCQ8ABZxf5ejgBI
- pP5TSQPLVMYbq1qbBPmWN0LYVlAvRbP4X512kDQZ9y4TQbvoZmhe+54sRsEJ8GW3hMJjERh0NNlg
- aB+3Cw/4u5IpwoSGxfltiA==
+ SEcyMAAAAA5Db21wcmVzc2lvbj1CWkJaaDkxQVkmU1kHdO0GAAAN////vFcSXL9/8H7R09C/578I
+ Ak0E4pe4SIIIgQSgGEQOcLABGYYNKiaaZGEyYjJhGTTRpiHogxGmTRiGRkNMIwhhPSbQJtpQiJkn
+ poyk9I0PUeoNNNBkeUAHqGgD0Ro0NNBoBoaMagNBoNCpNPUemp6QGmgyaPSGmQxGhkZDQbUaBkHp
+ MhoaANNMhkIyIauvSJPL4aUXjIQemQXkoaqOKqAQDIABsZALnf0yCLAyvmktzDWBCVHO6bb6kCqE
+ ZobVEhmMBjs0oQzekgs6PgZSyII8zy9mmG9To49ZlN6TaX5BxlS7cJiuICUdyjNQPIIdQs1Qqqqk
+ JZ2/BksYcU4HQyssZcpkoMco6gRc888KF9BO7BvuSuIPz7A4crBoaQB+euFU1ilz8yIBBmNBDgRX
+ pVh4zkmPiSKcqRJxcshMqh0vkKlgQDTcOujtdmnMVBZfQiPPemcHm2098VJyHBAOqOwluyIKyG92
+ JAR0CCu9SB5q9DyPHUdc5yB5CurIZHt3GM0dCiQRIN0EAcQNmTYTiHdi6B6Dc/ma0hrmSCQXBzfU
+ BEwthEg0YGaJf4u5IpwoSAO6doMA
--===============*==-- (glob)
with a specific bundle type
--- a/tests/test-rebase-conflicts.t Wed Feb 21 17:26:22 2018 +0100
+++ b/tests/test-rebase-conflicts.t Wed Feb 21 17:33:00 2018 +0100
@@ -297,8 +297,9 @@
list of changesets:
e31216eec445e44352c5f01588856059466a24c9
2f2496ddf49d69b5ef23ad8cf9fb2e0e4faf0ac2
- bundle2-output-bundle: "HG20", (1 params) 2 parts total
+ bundle2-output-bundle: "HG20", (1 params) 3 parts total
bundle2-output-part: "changegroup" (params: 1 mandatory 1 advisory) streamed payload
+ bundle2-output-part: "cache:rev-branch-cache" streamed payload
bundle2-output-part: "phase-heads" 24 bytes payload
saved backup bundle to $TESTTMP/issue4041/.hg/strip-backup/e31216eec445-15f7a814-rebase.hg
3 changesets found
@@ -306,8 +307,9 @@
4c9fbe56a16f30c0d5dcc40ec1a97bbe3325209c
19c888675e133ab5dff84516926a65672eaf04d9
2a7f09cac94c7f4b73ebd5cd1a62d3b2e8e336bf
- bundle2-output-bundle: "HG20", 2 parts total
+ bundle2-output-bundle: "HG20", 3 parts total
bundle2-output-part: "changegroup" (params: 1 mandatory 1 advisory) streamed payload
+ bundle2-output-part: "cache:rev-branch-cache" streamed payload
bundle2-output-part: "phase-heads" 24 bytes payload
adding branch
bundle2-input-bundle: with-transaction
@@ -321,12 +323,15 @@
adding f1.txt revisions
added 2 changesets with 2 changes to 1 files
bundle2-input-part: total payload size 1686
+ bundle2-input-part: "cache:rev-branch-cache" supported
+ bundle2-input-part: total payload size 74
bundle2-input-part: "phase-heads" supported
bundle2-input-part: total payload size 24
- bundle2-input-bundle: 1 parts total
+ bundle2-input-bundle: 2 parts total
updating the branch cache
invalid branchheads cache (served): tip differs
rebase completed
+ truncating cache/rbc-revs-v1 to 0
Test minimization of merge conflicts
$ hg up -q null
--- a/tests/test-strip.t Wed Feb 21 17:26:22 2018 +0100
+++ b/tests/test-strip.t Wed Feb 21 17:33:00 2018 +0100
@@ -214,6 +214,7 @@
Stream params: {Compression: BZ}
changegroup -- {nbchanges: 1, version: 02}
264128213d290d868c54642d13aeaa3675551a78
+ cache:rev-branch-cache -- {}
phase-heads -- {}
264128213d290d868c54642d13aeaa3675551a78 draft
$ hg pull .hg/strip-backup/*
@@ -843,13 +844,13 @@
list of changesets:
6625a516847449b6f0fa3737b9ba56e9f0f3032c
d8db9d1372214336d2b5570f20ee468d2c72fa8b
- bundle2-output-bundle: "HG20", (1 params) 2 parts total
+ bundle2-output-bundle: "HG20", (1 params) 3 parts total
bundle2-output-part: "changegroup" (params: 1 mandatory 1 advisory) streamed payload
+ bundle2-output-part: "cache:rev-branch-cache" streamed payload
bundle2-output-part: "phase-heads" 24 bytes payload
saved backup bundle to $TESTTMP/issue4736/.hg/strip-backup/6625a5168474-345bb43d-backup.hg
updating the branch cache
invalid branchheads cache (served): tip differs
- truncating cache/rbc-revs-v1 to 24
$ hg log -G
o changeset: 2:5c51d8d6557d
| tag: tip
--- a/tests/test-tags.t Wed Feb 21 17:26:22 2018 +0100
+++ b/tests/test-tags.t Wed Feb 21 17:33:00 2018 +0100
@@ -729,6 +729,7 @@
f63cc8fe54e4d326f8d692805d70e092f851ddb1
40f0358cb314c824a5929ee527308d90e023bc10
hgtagsfnodes -- {}
+ cache:rev-branch-cache -- {}
Check that local clone includes cache data