changeset 36965:b89a7ef29013

bundle: include advisory rev branch cache part in bundle2 bundle

`hg bundle` command producing bundle2 will now include an optional part
containing the revision-branch cache data.

The data sent are mostly nodes so it is quite compact.  The goal of the
rev-branch-cache is to speed up branch map computation, especially when the
branchmap gets invalidated so we send data for all exchanged changesets. In
addition, computing the relevant heads to send in case of partial pulling would
be challenging.

As a reminder, the rev branch cache data significantly speed up branch
computation. Having it around provides a small speedup to pull/clone and much
higher tolerance to branch map cache invalidation that might happens from later
commands.

On the Mercurial repository, computing the visible branchmap from scratch move
from 2.00 seconds to 0.34s (a -83% speedup).

Using this new part, Unbundling the full Mercurial repository moves from 25.736
seconds to 24.030 seconds (around -7% speedup). The bundle size increase is
around 3% (from 22.43 MB to 23.13MB)


On an half a million revision repository with twenty thousand
branches, computing the branchmap moves from 75 seconds to 45 second (-40%) if
the caches is used.

A bundle containing 50 000 changesets in such repository get a 0.5% size
increase from such part for a -3% unbundling time speedup.

--- a/mercurial/bundle2.py	Wed Feb 21 17:26:22 2018 +0100
+++ b/mercurial/bundle2.py	Wed Feb 21 17:33:00 2018 +0100
@@ -1591,6 +1591,7 @@
         part.addparam('targetphase', '%d' % phases.secret, mandatory=False)
 
     addparttagsfnodescache(repo, bundler, outgoing)
+    addpartrevbranchcache(repo, bundler, outgoing)
 
     if opts.get('obsolescence', False):
         obsmarkers = repo.obsstore.relevantmarkers(outgoing.missing)

--- a/tests/test-bundle-phases.t	Wed Feb 21 17:26:22 2018 +0100
+++ b/tests/test-bundle-phases.t	Wed Feb 21 17:33:00 2018 +0100
@@ -42,6 +42,7 @@
       26805aba1e600a82e93661149f2313866a221a7b
       f585351a92f85104bff7c284233c338b10eb1df7
       9bc730a19041f9ec7cb33c626e811aa233efb18c
+  cache:rev-branch-cache -- {}
   phase-heads -- {}
       26805aba1e600a82e93661149f2313866a221a7b draft
   $ hg strip --no-backup C
@@ -233,6 +234,7 @@
       dc0947a82db884575bb76ea10ac97b08536bfa03
       4e4f9194f9f181c57f62e823e8bdfa46ab9e4ff4
       03ca77807e919db8807c3749086dc36fb478cac0
+  cache:rev-branch-cache -- {}
   phase-heads -- {}
       dc0947a82db884575bb76ea10ac97b08536bfa03 public
       03ca77807e919db8807c3749086dc36fb478cac0 draft
@@ -258,6 +260,7 @@
   changegroup -- {nbchanges: 2, targetphase: 2, version: 02}
       112478962961147124edd43549aedd1a335e44bf
       4e4f9194f9f181c57f62e823e8bdfa46ab9e4ff4
+  cache:rev-branch-cache -- {}
   phase-heads -- {}
   $ rm bundle
 
@@ -269,6 +272,7 @@
       112478962961147124edd43549aedd1a335e44bf
       dc0947a82db884575bb76ea10ac97b08536bfa03
       4e4f9194f9f181c57f62e823e8bdfa46ab9e4ff4
+  cache:rev-branch-cache -- {}
   phase-heads -- {}
       dc0947a82db884575bb76ea10ac97b08536bfa03 public
   $ rm bundle
@@ -280,6 +284,7 @@
   changegroup -- {nbchanges: 2, targetphase: 2, version: 02}
       4e4f9194f9f181c57f62e823e8bdfa46ab9e4ff4
       03ca77807e919db8807c3749086dc36fb478cac0
+  cache:rev-branch-cache -- {}
   phase-heads -- {}
       03ca77807e919db8807c3749086dc36fb478cac0 draft
   $ rm bundle

--- a/tests/test-bundle-type.t	Wed Feb 21 17:26:22 2018 +0100
+++ b/tests/test-bundle-type.t	Wed Feb 21 17:33:00 2018 +0100
@@ -76,6 +76,7 @@
   Stream params: {}
   changegroup -- {nbchanges: 1, version: 02}
       c35a0f9217e65d1fdb90c936ffa7dbe679f83ddf
+  cache:rev-branch-cache -- {}
   none-v2
   
   % test bundle type bzip2
@@ -85,6 +86,7 @@
   Stream params: {Compression: BZ}
   changegroup -- {nbchanges: 1, version: 02}
       c35a0f9217e65d1fdb90c936ffa7dbe679f83ddf
+  cache:rev-branch-cache -- {}
   bzip2-v2
   
   % test bundle type gzip
@@ -94,6 +96,7 @@
   Stream params: {Compression: GZ}
   changegroup -- {nbchanges: 1, version: 02}
       c35a0f9217e65d1fdb90c936ffa7dbe679f83ddf
+  cache:rev-branch-cache -- {}
   gzip-v2
   
   % test bundle type none-v2
@@ -103,6 +106,7 @@
   Stream params: {}
   changegroup -- {nbchanges: 1, version: 02}
       c35a0f9217e65d1fdb90c936ffa7dbe679f83ddf
+  cache:rev-branch-cache -- {}
   none-v2
   
   % test bundle type v2
@@ -112,6 +116,7 @@
   Stream params: {Compression: BZ}
   changegroup -- {nbchanges: 1, version: 02}
       c35a0f9217e65d1fdb90c936ffa7dbe679f83ddf
+  cache:rev-branch-cache -- {}
   bzip2-v2
   
   % test bundle type v1
@@ -150,12 +155,12 @@
   $ hg bundle -a -t gzip-v2 gzip-v2.hg
   1 changesets found
   $ f --size gzip-v2.hg
-  gzip-v2.hg: size=427
+  gzip-v2.hg: size=468
 
   $ hg --config experimental.bundlecomplevel=1 bundle -a -t gzip-v2 gzip-v2-level1.hg
   1 changesets found
   $ f --size gzip-v2-level1.hg
-  gzip-v2-level1.hg: size=435
+  gzip-v2-level1.hg: size=475
 
   $ cd ..
 
@@ -171,6 +176,7 @@
   Stream params: {Compression: ZS}
   changegroup -- {nbchanges: 1, version: 02}
       c35a0f9217e65d1fdb90c936ffa7dbe679f83ddf
+  cache:rev-branch-cache -- {}
   zstd-v2
   
   % test bundle type zstd-v2
@@ -180,6 +186,7 @@
   Stream params: {Compression: ZS}
   changegroup -- {nbchanges: 1, version: 02}
       c35a0f9217e65d1fdb90c936ffa7dbe679f83ddf
+  cache:rev-branch-cache -- {}
   zstd-v2
   
 

--- a/tests/test-bundle.t	Wed Feb 21 17:26:22 2018 +0100
+++ b/tests/test-bundle.t	Wed Feb 21 17:33:00 2018 +0100
@@ -774,7 +774,7 @@
   list of changesets:
   1a38c1b849e8b70c756d2d80b0b9a3ac0b7ea11a
   057f4db07f61970e1c11e83be79e9d08adc4dc31
-  bundle2-output-bundle: "HG20", (1 params) 1 parts total
+  bundle2-output-bundle: "HG20", (1 params) 2 parts total
   bundle2-output-part: "changegroup" (params: 1 mandatory 1 advisory) streamed payload
   bundling: 1/2 changesets (50.00%)
   bundling: 2/2 changesets (100.00%)
@@ -783,6 +783,7 @@
   bundling: b 1/3 files (33.33%)
   bundling: b1 2/3 files (66.67%)
   bundling: x 3/3 files (100.00%)
+  bundle2-output-part: "cache:rev-branch-cache" streamed payload
 
 == Test for issue3441
 

--- a/tests/test-clonebundles.t	Wed Feb 21 17:26:22 2018 +0100
+++ b/tests/test-clonebundles.t	Wed Feb 21 17:33:00 2018 +0100
@@ -163,7 +163,7 @@
 by old clients.
 
   $ f --size --hexdump full.hg
-  full.hg: size=396
+  full.hg: size=442
   0000: 48 47 32 30 00 00 00 0e 43 6f 6d 70 72 65 73 73 |HG20....Compress|
   0010: 69 6f 6e 3d 47 5a 78 9c 63 60 60 d0 e4 76 f6 70 |ion=GZx.c``..v.p|
   0020: f4 73 77 75 0f f2 0f 0d 60 00 02 46 46 76 26 4e |.swu....`..FFv&N|
@@ -188,7 +188,10 @@
   0150: 88 75 34 36 75 04 82 55 17 14 36 a4 38 10 04 d8 |.u46u..U..6.8...|
   0160: 21 01 9a b1 83 f7 e9 45 8b d2 56 c7 a3 1f 82 52 |!......E..V....R|
   0170: d7 8a 78 ed fc d5 76 f1 36 25 81 89 c7 ad ec 90 |..x...v.6%......|
-  0180: 54 47 75 2b 89 49 b1 00 d2 8a eb 92             |TGu+.I......|
+  0180: 54 47 75 2b 89 48 b1 b2 62 ce 8e ce 1e ae 56 41 |TGu+.H..b.....VA|
+  0190: ae 61 ba 4e 41 8e 7e ce 1e ba 60 01 a0 14 23 58 |.a.NA.~...`...#X|
+  01a0: 81 35 c8 7d 40 cc 04 e2 a4 a4 a6 25 96 e6 94 60 |.5.}@......%...`|
+  01b0: 33 17 5f 54 00 00 01 1b 0a ec                   |3._T......|
 
   $ echo "http://localhost:$HGPORT1/full.hg" > server/.hg/clonebundles.manifest
   $ hg clone -U http://localhost:$HGPORT full-bundle
@@ -529,14 +532,14 @@
   $ cat hg.pid >> $DAEMON_PIDS
 
   $ hg -R server debuglfput gz-a.hg
-  f6eca29e25359f6a92f1ea64527cdcf1b5abe62a
+  14ee2f0b3f1d14aeeb2fe037e09fc295c3cf59f5
 
   $ cat > server/.hg/clonebundles.manifest << EOF
-  > largefile://f6eca29e25359f6a92f1ea64527cdcf1b5abe62a BUNDLESPEC=gzip-v2
+  > largefile://14ee2f0b3f1d14aeeb2fe037e09fc295c3cf59f5 BUNDLESPEC=gzip-v2
   > EOF
 
   $ hg clone -U http://localhost:$HGPORT largefile-provided --traceback
-  applying clone bundle from largefile://f6eca29e25359f6a92f1ea64527cdcf1b5abe62a
+  applying clone bundle from largefile://14ee2f0b3f1d14aeeb2fe037e09fc295c3cf59f5
   adding changesets
   adding manifests
   adding file changes

--- a/tests/test-debugbundle.t	Wed Feb 21 17:26:22 2018 +0100
+++ b/tests/test-debugbundle.t	Wed Feb 21 17:33:00 2018 +0100
@@ -34,12 +34,14 @@
   changegroup -- {nbchanges: 2, version: 02}
       0e067c57feba1a5694ca4844f05588bb1bf82342
       991a3460af53952d10ec8a295d3d2cc2e5fa9690
+  cache:rev-branch-cache -- {}
 
 Quiet output
 
   $ hg debugbundle --quiet bundle2.hg
   Stream params: {}
   changegroup -- {nbchanges: 2, version: 02}
+  cache:rev-branch-cache -- {}
 
 Verbose output:
 
@@ -78,5 +80,6 @@
   
       c
       b80de5d138758541c5f05265ad144ab9fa86d1db 0000000000000000000000000000000000000000 0000000000000000000000000000000000000000 991a3460af53952d10ec8a295d3d2cc2e5fa9690 0000000000000000000000000000000000000000 0
+  cache:rev-branch-cache -- {}
 
   $ cd ..

--- a/tests/test-generaldelta.t	Wed Feb 21 17:26:22 2018 +0100
+++ b/tests/test-generaldelta.t	Wed Feb 21 17:33:00 2018 +0100
@@ -159,6 +159,7 @@
   Stream params: {Compression: BZ}
   changegroup -- {nbchanges: 1, version: 02}
       1c5d4dc9a8b8d6e1750966d343e94db665e7a1e9
+  cache:rev-branch-cache -- {}
   phase-heads -- {}
       1c5d4dc9a8b8d6e1750966d343e94db665e7a1e9 draft
 

--- a/tests/test-obsolete-changeset-exchange.t	Wed Feb 21 17:26:22 2018 +0100
+++ b/tests/test-obsolete-changeset-exchange.t	Wed Feb 21 17:33:00 2018 +0100
@@ -95,10 +95,12 @@
   Stream params: {Compression: BZ}
   changegroup -- {nbchanges: 1, version: 02}
       f89bcc95eba5174b1ccc3e33a82e84c96e8338ee
+  cache:rev-branch-cache -- {}
   $ hg debugbundle ../f89bcc95eba5-obs.hg
   Stream params: {Compression: BZ}
   changegroup -- {nbchanges: 1, version: 02}
       f89bcc95eba5174b1ccc3e33a82e84c96e8338ee
+  cache:rev-branch-cache -- {}
   obsmarkers -- {}
       version: 1 (70 bytes)
       9d73aac1b2ed7d53835eaeec212ed41ea47da53a f89bcc95eba5174b1ccc3e33a82e84c96e8338ee 0 (Thu Jan 01 00:00:00 1970 +0000) {'user': 'test'}

--- a/tests/test-obsolete.t	Wed Feb 21 17:26:22 2018 +0100
+++ b/tests/test-obsolete.t	Wed Feb 21 17:33:00 2018 +0100
@@ -1427,6 +1427,7 @@
   Stream params: {Compression: BZ}
   changegroup -- {nbchanges: 1, version: 02}
       e008cf2834908e5d6b0f792a9d4b0e2272260fb8
+  cache:rev-branch-cache -- {}
   phase-heads -- {}
       e008cf2834908e5d6b0f792a9d4b0e2272260fb8 draft
 
@@ -1468,6 +1469,7 @@
   changegroup -- {nbchanges: 2, version: 02}
       e016b03fd86fcccc54817d120b90b751aaf367d6
       b0551702f918510f01ae838ab03a463054c67b46
+  cache:rev-branch-cache -- {}
   obsmarkers -- {}
       version: 1 (92 bytes)
       e008cf2834908e5d6b0f792a9d4b0e2272260fb8 b0551702f918510f01ae838ab03a463054c67b46 0 (Thu Jan 01 00:00:00 1970 +0000) {'ef1': '8', 'operation': 'amend', 'user': 'test'}

--- a/tests/test-patchbomb.t	Wed Feb 21 17:26:22 2018 +0100
+++ b/tests/test-patchbomb.t	Wed Feb 21 17:33:00 2018 +0100
@@ -390,14 +390,15 @@
   Content-Disposition: attachment; filename="bundle.hg"
   Content-Transfer-Encoding: base64
   
-  SEcyMAAAAA5Db21wcmVzc2lvbj1CWkJaaDkxQVkmU1nYvy2xAAAJf//7vFYQXD1/4H7R09C/470I
-  Ak0E4pe4SIIIgQSgGEQOcLAA5VBKqeppMxTI0YjQNBgQMQDI0GgMhtR6I0GI2p6I0yeSEVT9MiYn
-  qjCYQwCBtARptARgBNDEwAGiDCMA40NGjQaNA0AAAAADIAAAA0BkABktCk6qObVxZ2A/33KHibLr
-  UQ4BwkgcPcmuCUAQZCztIWgR1SpBS6IqbIij4UFwhnnMkElcFTqoucIWbsBPK3l+6c+xYaVBWsJo
-  aT0OV/YAOvLrziifDQMJOMIaaYce9agtI2EwQBAq089UiRU+evFHSLRBT7Wa/D/YBaUtU5ezvtr3
-  6yrIS4Iyp9VWESdWPEi6VjRjdcEY4HvbmDIVEAEVJIUrHNIBx/MmnBBRkw8tSlCQ8ABZxf5ejgBI
-  pP5TSQPLVMYbq1qbBPmWN0LYVlAvRbP4X512kDQZ9y4TQbvoZmhe+54sRsEJ8GW3hMJjERh0NNlg
-  aB+3Cw/4u5IpwoSGxfltiA==
+  SEcyMAAAAA5Db21wcmVzc2lvbj1CWkJaaDkxQVkmU1kHdO0GAAAN////vFcSXL9/8H7R09C/578I
+  Ak0E4pe4SIIIgQSgGEQOcLABGYYNKiaaZGEyYjJhGTTRpiHogxGmTRiGRkNMIwhhPSbQJtpQiJkn
+  poyk9I0PUeoNNNBkeUAHqGgD0Ro0NNBoBoaMagNBoNCpNPUemp6QGmgyaPSGmQxGhkZDQbUaBkHp
+  MhoaANNMhkIyIauvSJPL4aUXjIQemQXkoaqOKqAQDIABsZALnf0yCLAyvmktzDWBCVHO6bb6kCqE
+  ZobVEhmMBjs0oQzekgs6PgZSyII8zy9mmG9To49ZlN6TaX5BxlS7cJiuICUdyjNQPIIdQs1Qqqqk
+  JZ2/BksYcU4HQyssZcpkoMco6gRc888KF9BO7BvuSuIPz7A4crBoaQB+euFU1ilz8yIBBmNBDgRX
+  pVh4zkmPiSKcqRJxcshMqh0vkKlgQDTcOujtdmnMVBZfQiPPemcHm2098VJyHBAOqOwluyIKyG92
+  JAR0CCu9SB5q9DyPHUdc5yB5CurIZHt3GM0dCiQRIN0EAcQNmTYTiHdi6B6Dc/ma0hrmSCQXBzfU
+  BEwthEg0YGaJf4u5IpwoSAO6doMA
   --===============*==-- (glob)
 
 with a specific bundle type

--- a/tests/test-rebase-conflicts.t	Wed Feb 21 17:26:22 2018 +0100
+++ b/tests/test-rebase-conflicts.t	Wed Feb 21 17:33:00 2018 +0100
@@ -297,8 +297,9 @@
   list of changesets:
   e31216eec445e44352c5f01588856059466a24c9
   2f2496ddf49d69b5ef23ad8cf9fb2e0e4faf0ac2
-  bundle2-output-bundle: "HG20", (1 params) 2 parts total
+  bundle2-output-bundle: "HG20", (1 params) 3 parts total
   bundle2-output-part: "changegroup" (params: 1 mandatory 1 advisory) streamed payload
+  bundle2-output-part: "cache:rev-branch-cache" streamed payload
   bundle2-output-part: "phase-heads" 24 bytes payload
   saved backup bundle to $TESTTMP/issue4041/.hg/strip-backup/e31216eec445-15f7a814-rebase.hg
   3 changesets found
@@ -306,8 +307,9 @@
   4c9fbe56a16f30c0d5dcc40ec1a97bbe3325209c
   19c888675e133ab5dff84516926a65672eaf04d9
   2a7f09cac94c7f4b73ebd5cd1a62d3b2e8e336bf
-  bundle2-output-bundle: "HG20", 2 parts total
+  bundle2-output-bundle: "HG20", 3 parts total
   bundle2-output-part: "changegroup" (params: 1 mandatory 1 advisory) streamed payload
+  bundle2-output-part: "cache:rev-branch-cache" streamed payload
   bundle2-output-part: "phase-heads" 24 bytes payload
   adding branch
   bundle2-input-bundle: with-transaction
@@ -321,12 +323,15 @@
   adding f1.txt revisions
   added 2 changesets with 2 changes to 1 files
   bundle2-input-part: total payload size 1686
+  bundle2-input-part: "cache:rev-branch-cache" supported
+  bundle2-input-part: total payload size 74
   bundle2-input-part: "phase-heads" supported
   bundle2-input-part: total payload size 24
-  bundle2-input-bundle: 1 parts total
+  bundle2-input-bundle: 2 parts total
   updating the branch cache
   invalid branchheads cache (served): tip differs
   rebase completed
+  truncating cache/rbc-revs-v1 to 0
 
 Test minimization of merge conflicts
   $ hg up -q null

--- a/tests/test-strip.t	Wed Feb 21 17:26:22 2018 +0100
+++ b/tests/test-strip.t	Wed Feb 21 17:33:00 2018 +0100
@@ -214,6 +214,7 @@
   Stream params: {Compression: BZ}
   changegroup -- {nbchanges: 1, version: 02}
       264128213d290d868c54642d13aeaa3675551a78
+  cache:rev-branch-cache -- {}
   phase-heads -- {}
       264128213d290d868c54642d13aeaa3675551a78 draft
   $ hg pull .hg/strip-backup/*
@@ -843,13 +844,13 @@
   list of changesets:
   6625a516847449b6f0fa3737b9ba56e9f0f3032c
   d8db9d1372214336d2b5570f20ee468d2c72fa8b
-  bundle2-output-bundle: "HG20", (1 params) 2 parts total
+  bundle2-output-bundle: "HG20", (1 params) 3 parts total
   bundle2-output-part: "changegroup" (params: 1 mandatory 1 advisory) streamed payload
+  bundle2-output-part: "cache:rev-branch-cache" streamed payload
   bundle2-output-part: "phase-heads" 24 bytes payload
   saved backup bundle to $TESTTMP/issue4736/.hg/strip-backup/6625a5168474-345bb43d-backup.hg
   updating the branch cache
   invalid branchheads cache (served): tip differs
-  truncating cache/rbc-revs-v1 to 24
   $ hg log -G
   o  changeset:   2:5c51d8d6557d
   |  tag:         tip

--- a/tests/test-tags.t	Wed Feb 21 17:26:22 2018 +0100
+++ b/tests/test-tags.t	Wed Feb 21 17:33:00 2018 +0100
@@ -729,6 +729,7 @@
       f63cc8fe54e4d326f8d692805d70e092f851ddb1
       40f0358cb314c824a5929ee527308d90e023bc10
   hgtagsfnodes -- {}
+  cache:rev-branch-cache -- {}
 
 Check that local clone includes cache data