view tests/test-getbundle.t @ 26623:5a95fe44121d

clonebundles: support for seeding clones from pre-generated bundles

Cloning can be an expensive operation for servers because the server
generates a bundle from existing repository data at request time. For
a large repository like mozilla-central, this consumes 4+ minutes
of CPU time on the server. It also results in significant network
utilization. Multiplied by hundreds or even thousands of clients and
the ensuing load can result in difficulties scaling the Mercurial server.

Despite generation of bundles being deterministic until the next
changeset is added, the generation of bundles to service a clone request
is not cached. Each clone thus performs redundant work. This is
wasteful.

This patch introduces the "clonebundles" extension and related
client-side functionality to help alleviate this deficiency. The
client-side feature is behind an experimental flag and is not enabled by
default.

It works as follows:

1) Server operator generates a bundle and makes it available on a
   server (likely HTTP).
2) Server operator defines the URL of a bundle file in a
   .hg/clonebundles.manifest file.
3) Client `hg clone`ing sees the server is advertising bundle URLs.
4) Client fetches and applies the advertised bundle.
5) Client performs equivalent of `hg pull` to fetch changes made since
   the bundle was created.

Essentially, the server performs the expensive work of generating a
bundle once and all subsequent clones fetch a static file from
somewhere. Scaling static file serving is a much more manageable
problem than scaling a Python application like Mercurial. Assuming your
repository grows less than 1% per day, the end result is 99+% of CPU
and network load from clones is eliminated, allowing Mercurial servers
to scale more easily. Serving static files also means data can be
transferred to clients as fast as they can consume it, rather than as
fast as servers can generate it. This makes clones faster.

Mozilla has implemented similar functionality of this patch on
hg.mozilla.org using a custom extension. We are hosting bundle files in
Amazon S3 and CloudFront (a CDN) and have successfully offloaded
>1 TB/day in data transfer from hg.mozilla.org, freeing up significant
bandwidth and CPU resources. The positive impact has been stellar and
I believe it has proved its value to be included in Mercurial core. I
feel it is important for the client-side support to be enabled in core
by default because it means that clients will get faster, more reliable
clones and will enable server operators to reduce load without
requiring any client-side configuration changes (assuming clients are
up to date, of course).

The scope of this feature is narrowly and specifically tailored to
cloning, despite "serve pulls from pre-generated bundles" being a valid
and useful feature. I would eventually like for Mercurial servers to
support transferring *all* repository data via statically hosted files.
You could imagine a server that siphons all pushed data to bundle files
and instructs clients to apply a stream of bundles to reconstruct all
repository data. This feature, while useful and powerful, is
significantly more work to implement because it requires the server
component have awareness of discovery and a mapping of which changesets
are in which files. Full, clone bundles, by contrast, are much simpler.

The wire protocol command is named "clonebundles" instead of something
more generic like "staticbundles" to leave the door open for a new, more
powerful and more generic server-side component with minimal backwards
compatibility implications. The name "bundleclone" is used by Mozilla's
extension and would cause problems since there are subtle differences
in Mozilla's extension.

Mozilla's experience with this idea has taught us that some form of
"content negotiation" is required. Not all clients will support all
bundle formats or even URLs (advanced TLS requirements, etc). To ensure
the highest uptake possible, a server needs to advertise multiple
versions of bundles and clients need to be able to choose the most
appropriate from that list one. The "attributes" in each
server-advertised entry facilitate this filtering and sorting. Their
use will become apparent in subsequent patches.

Initial inspiration and credit for the idea of cloning from static files
belongs to Augie Fackler and his "lookaside clone" extension proof of
concept.

#require serve

= Test the getbundle() protocol function =

Create a test repository:

  $ hg init repo
  $ cd repo
  $ hg debugbuilddag -n -m '+2 :fork +5 :p1 *fork +6 :p2 /p1 :m1 +3' > /dev/null
  $ hg log -G --template '{node}\n'
  o  10c14a2cc935e1d8c31f9e98587dcf27fb08a6da
  |
  o  4801a72e5d88cb515b0c7e40fae34180f3f837f2
  |
  o  0b2f73f04880d9cb6a5cd8a757f0db0ad01e32c3
  |
  o    8365676dbab05860ce0d9110f2af51368b961bbd
  |\
  | o  5686dbbd9fc46cb806599c878d02fe1cb56b83d3
  | |
  | o  13c0170174366b441dc68e8e33757232fa744458
  | |
  | o  63476832d8ec6558cf9bbe3cbe0c757e5cf18043
  | |
  | o  700b7e19db54103633c4bf4a6a6b6d55f4d50c03
  | |
  | o  928b5f94cdb278bb536eba552de348a4e92ef24d
  | |
  | o  f34414c64173e0ecb61b25dc55e116dbbcc89bee
  | |
  | o  8931463777131cd73923e560b760061f2aa8a4bc
  | |
  o |  6621d79f61b23ec74cf4b69464343d9e0980ec8b
  | |
  o |  bac16991d12ff45f9dc43c52da1946dfadb83e80
  | |
  o |  ff42371d57168345fdf1a3aac66a51f6a45d41d2
  | |
  o |  d5f6e1ea452285324836a49d7d3c2a63cfed1d31
  | |
  o |  713346a995c363120712aed1aee7e04afd867638
  |/
  o  29a4d1f17bd3f0779ca0525bebb1cfb51067c738
  |
  o  7704483d56b2a7b5db54dcee7c62378ac629b348
  
  $ cd ..


= Test locally =

Get everything:

  $ hg debuggetbundle repo bundle
  $ hg debugbundle bundle
  7704483d56b2a7b5db54dcee7c62378ac629b348
  29a4d1f17bd3f0779ca0525bebb1cfb51067c738
  713346a995c363120712aed1aee7e04afd867638
  d5f6e1ea452285324836a49d7d3c2a63cfed1d31
  ff42371d57168345fdf1a3aac66a51f6a45d41d2
  bac16991d12ff45f9dc43c52da1946dfadb83e80
  6621d79f61b23ec74cf4b69464343d9e0980ec8b
  8931463777131cd73923e560b760061f2aa8a4bc
  f34414c64173e0ecb61b25dc55e116dbbcc89bee
  928b5f94cdb278bb536eba552de348a4e92ef24d
  700b7e19db54103633c4bf4a6a6b6d55f4d50c03
  63476832d8ec6558cf9bbe3cbe0c757e5cf18043
  13c0170174366b441dc68e8e33757232fa744458
  5686dbbd9fc46cb806599c878d02fe1cb56b83d3
  8365676dbab05860ce0d9110f2af51368b961bbd
  0b2f73f04880d9cb6a5cd8a757f0db0ad01e32c3
  4801a72e5d88cb515b0c7e40fae34180f3f837f2
  10c14a2cc935e1d8c31f9e98587dcf27fb08a6da

Get part of linear run:

  $ hg debuggetbundle repo bundle -H 4801a72e5d88cb515b0c7e40fae34180f3f837f2 -C 8365676dbab05860ce0d9110f2af51368b961bbd
  $ hg debugbundle bundle
  0b2f73f04880d9cb6a5cd8a757f0db0ad01e32c3
  4801a72e5d88cb515b0c7e40fae34180f3f837f2

Get missing branch and merge:

  $ hg debuggetbundle repo bundle -H 4801a72e5d88cb515b0c7e40fae34180f3f837f2 -C 13c0170174366b441dc68e8e33757232fa744458
  $ hg debugbundle bundle
  713346a995c363120712aed1aee7e04afd867638
  d5f6e1ea452285324836a49d7d3c2a63cfed1d31
  ff42371d57168345fdf1a3aac66a51f6a45d41d2
  bac16991d12ff45f9dc43c52da1946dfadb83e80
  6621d79f61b23ec74cf4b69464343d9e0980ec8b
  5686dbbd9fc46cb806599c878d02fe1cb56b83d3
  8365676dbab05860ce0d9110f2af51368b961bbd
  0b2f73f04880d9cb6a5cd8a757f0db0ad01e32c3
  4801a72e5d88cb515b0c7e40fae34180f3f837f2

Get from only one head:

  $ hg debuggetbundle repo bundle -H 928b5f94cdb278bb536eba552de348a4e92ef24d -C 29a4d1f17bd3f0779ca0525bebb1cfb51067c738
  $ hg debugbundle bundle
  8931463777131cd73923e560b760061f2aa8a4bc
  f34414c64173e0ecb61b25dc55e116dbbcc89bee
  928b5f94cdb278bb536eba552de348a4e92ef24d

Get parts of two branches:

  $ hg debuggetbundle repo bundle -H 13c0170174366b441dc68e8e33757232fa744458 -C 700b7e19db54103633c4bf4a6a6b6d55f4d50c03 -H bac16991d12ff45f9dc43c52da1946dfadb83e80 -C d5f6e1ea452285324836a49d7d3c2a63cfed1d31
  $ hg debugbundle bundle
  ff42371d57168345fdf1a3aac66a51f6a45d41d2
  bac16991d12ff45f9dc43c52da1946dfadb83e80
  63476832d8ec6558cf9bbe3cbe0c757e5cf18043
  13c0170174366b441dc68e8e33757232fa744458

Check that we get all needed file changes:

  $ hg debugbundle bundle --all
  format: id, p1, p2, cset, delta base, len(delta)
  
  changelog
  ff42371d57168345fdf1a3aac66a51f6a45d41d2 d5f6e1ea452285324836a49d7d3c2a63cfed1d31 0000000000000000000000000000000000000000 ff42371d57168345fdf1a3aac66a51f6a45d41d2 d5f6e1ea452285324836a49d7d3c2a63cfed1d31 99
  bac16991d12ff45f9dc43c52da1946dfadb83e80 ff42371d57168345fdf1a3aac66a51f6a45d41d2 0000000000000000000000000000000000000000 bac16991d12ff45f9dc43c52da1946dfadb83e80 ff42371d57168345fdf1a3aac66a51f6a45d41d2 99
  63476832d8ec6558cf9bbe3cbe0c757e5cf18043 700b7e19db54103633c4bf4a6a6b6d55f4d50c03 0000000000000000000000000000000000000000 63476832d8ec6558cf9bbe3cbe0c757e5cf18043 bac16991d12ff45f9dc43c52da1946dfadb83e80 102
  13c0170174366b441dc68e8e33757232fa744458 63476832d8ec6558cf9bbe3cbe0c757e5cf18043 0000000000000000000000000000000000000000 13c0170174366b441dc68e8e33757232fa744458 63476832d8ec6558cf9bbe3cbe0c757e5cf18043 102
  
  manifest
  dac7984588fc4eea7acbf39693a9c1b06f5b175d 591f732a3faf1fb903815273f3c199a514a61ccb 0000000000000000000000000000000000000000 ff42371d57168345fdf1a3aac66a51f6a45d41d2 591f732a3faf1fb903815273f3c199a514a61ccb 113
  0772616e6b48a76afb6c1458e193cbb3dae2e4ff dac7984588fc4eea7acbf39693a9c1b06f5b175d 0000000000000000000000000000000000000000 bac16991d12ff45f9dc43c52da1946dfadb83e80 dac7984588fc4eea7acbf39693a9c1b06f5b175d 113
  eb498cd9af6c44108e43041e951ce829e29f6c80 bff2f4817ced57b386caf7c4e3e36a4bc9af7e93 0000000000000000000000000000000000000000 63476832d8ec6558cf9bbe3cbe0c757e5cf18043 0772616e6b48a76afb6c1458e193cbb3dae2e4ff 295
  b15709c071ddd2d93188508ba156196ab4f19620 eb498cd9af6c44108e43041e951ce829e29f6c80 0000000000000000000000000000000000000000 13c0170174366b441dc68e8e33757232fa744458 eb498cd9af6c44108e43041e951ce829e29f6c80 114
  
  mf
  4f73f97080266ab8e0c0561ca8d0da3eaf65b695 301ca08d026bb72cb4258a9d211bdf7ca0bcd810 0000000000000000000000000000000000000000 ff42371d57168345fdf1a3aac66a51f6a45d41d2 301ca08d026bb72cb4258a9d211bdf7ca0bcd810 17
  c7b583de053293870e145f45bd2d61643563fd06 4f73f97080266ab8e0c0561ca8d0da3eaf65b695 0000000000000000000000000000000000000000 bac16991d12ff45f9dc43c52da1946dfadb83e80 4f73f97080266ab8e0c0561ca8d0da3eaf65b695 18
  266ee3c0302a5a18f1cf96817ac79a51836179e9 edc0f6b8db80d68ae6aff2b19f7e5347ab68fa63 0000000000000000000000000000000000000000 63476832d8ec6558cf9bbe3cbe0c757e5cf18043 c7b583de053293870e145f45bd2d61643563fd06 149
  698c6a36220548cd3903ca7dada27c59aa500c52 266ee3c0302a5a18f1cf96817ac79a51836179e9 0000000000000000000000000000000000000000 13c0170174366b441dc68e8e33757232fa744458 266ee3c0302a5a18f1cf96817ac79a51836179e9 19
  
  nf11
  33fbc651630ffa7ccbebfe4eb91320a873e7291c 0000000000000000000000000000000000000000 0000000000000000000000000000000000000000 63476832d8ec6558cf9bbe3cbe0c757e5cf18043 0000000000000000000000000000000000000000 16
  
  nf12
  ddce0544363f037e9fb889faca058f52dc01c0a5 0000000000000000000000000000000000000000 0000000000000000000000000000000000000000 13c0170174366b441dc68e8e33757232fa744458 0000000000000000000000000000000000000000 16
  
  nf4
  3c1407305701051cbed9f9cb9a68bdfb5997c235 0000000000000000000000000000000000000000 0000000000000000000000000000000000000000 ff42371d57168345fdf1a3aac66a51f6a45d41d2 0000000000000000000000000000000000000000 15
  
  nf5
  0dbd89c185f53a1727c54cd1ce256482fa23968e 0000000000000000000000000000000000000000 0000000000000000000000000000000000000000 bac16991d12ff45f9dc43c52da1946dfadb83e80 0000000000000000000000000000000000000000 15

Get branch and merge:

  $ hg debuggetbundle repo bundle -C 7704483d56b2a7b5db54dcee7c62378ac629b348 -H 0b2f73f04880d9cb6a5cd8a757f0db0ad01e32c3
  $ hg debugbundle bundle
  29a4d1f17bd3f0779ca0525bebb1cfb51067c738
  713346a995c363120712aed1aee7e04afd867638
  d5f6e1ea452285324836a49d7d3c2a63cfed1d31
  ff42371d57168345fdf1a3aac66a51f6a45d41d2
  bac16991d12ff45f9dc43c52da1946dfadb83e80
  6621d79f61b23ec74cf4b69464343d9e0980ec8b
  8931463777131cd73923e560b760061f2aa8a4bc
  f34414c64173e0ecb61b25dc55e116dbbcc89bee
  928b5f94cdb278bb536eba552de348a4e92ef24d
  700b7e19db54103633c4bf4a6a6b6d55f4d50c03
  63476832d8ec6558cf9bbe3cbe0c757e5cf18043
  13c0170174366b441dc68e8e33757232fa744458
  5686dbbd9fc46cb806599c878d02fe1cb56b83d3
  8365676dbab05860ce0d9110f2af51368b961bbd
  0b2f73f04880d9cb6a5cd8a757f0db0ad01e32c3

= Test bundle2 =

  $ hg debuggetbundle repo bundle -t bundle2
  $ hg debugbundle bundle
  Stream params: {}
  changegroup -- "{'version': '01'}"
      7704483d56b2a7b5db54dcee7c62378ac629b348
      29a4d1f17bd3f0779ca0525bebb1cfb51067c738
      713346a995c363120712aed1aee7e04afd867638
      d5f6e1ea452285324836a49d7d3c2a63cfed1d31
      ff42371d57168345fdf1a3aac66a51f6a45d41d2
      bac16991d12ff45f9dc43c52da1946dfadb83e80
      6621d79f61b23ec74cf4b69464343d9e0980ec8b
      8931463777131cd73923e560b760061f2aa8a4bc
      f34414c64173e0ecb61b25dc55e116dbbcc89bee
      928b5f94cdb278bb536eba552de348a4e92ef24d
      700b7e19db54103633c4bf4a6a6b6d55f4d50c03
      63476832d8ec6558cf9bbe3cbe0c757e5cf18043
      13c0170174366b441dc68e8e33757232fa744458
      5686dbbd9fc46cb806599c878d02fe1cb56b83d3
      8365676dbab05860ce0d9110f2af51368b961bbd
      0b2f73f04880d9cb6a5cd8a757f0db0ad01e32c3
      4801a72e5d88cb515b0c7e40fae34180f3f837f2
      10c14a2cc935e1d8c31f9e98587dcf27fb08a6da
= Test via HTTP =

Get everything:

  $ hg serve -R repo -p $HGPORT -d --pid-file=hg.pid -E error.log -A access.log
  $ cat hg.pid >> $DAEMON_PIDS
  $ hg debuggetbundle http://localhost:$HGPORT/ bundle
  $ hg debugbundle bundle
  7704483d56b2a7b5db54dcee7c62378ac629b348
  29a4d1f17bd3f0779ca0525bebb1cfb51067c738
  713346a995c363120712aed1aee7e04afd867638
  d5f6e1ea452285324836a49d7d3c2a63cfed1d31
  ff42371d57168345fdf1a3aac66a51f6a45d41d2
  bac16991d12ff45f9dc43c52da1946dfadb83e80
  6621d79f61b23ec74cf4b69464343d9e0980ec8b
  8931463777131cd73923e560b760061f2aa8a4bc
  f34414c64173e0ecb61b25dc55e116dbbcc89bee
  928b5f94cdb278bb536eba552de348a4e92ef24d
  700b7e19db54103633c4bf4a6a6b6d55f4d50c03
  63476832d8ec6558cf9bbe3cbe0c757e5cf18043
  13c0170174366b441dc68e8e33757232fa744458
  5686dbbd9fc46cb806599c878d02fe1cb56b83d3
  8365676dbab05860ce0d9110f2af51368b961bbd
  0b2f73f04880d9cb6a5cd8a757f0db0ad01e32c3
  4801a72e5d88cb515b0c7e40fae34180f3f837f2
  10c14a2cc935e1d8c31f9e98587dcf27fb08a6da

Get parts of two branches:

  $ hg debuggetbundle http://localhost:$HGPORT/ bundle -H 13c0170174366b441dc68e8e33757232fa744458 -C 700b7e19db54103633c4bf4a6a6b6d55f4d50c03 -H bac16991d12ff45f9dc43c52da1946dfadb83e80 -C d5f6e1ea452285324836a49d7d3c2a63cfed1d31
  $ hg debugbundle bundle
  ff42371d57168345fdf1a3aac66a51f6a45d41d2
  bac16991d12ff45f9dc43c52da1946dfadb83e80
  63476832d8ec6558cf9bbe3cbe0c757e5cf18043
  13c0170174366b441dc68e8e33757232fa744458

Check that we get all needed file changes:

  $ hg debugbundle bundle --all
  format: id, p1, p2, cset, delta base, len(delta)
  
  changelog
  ff42371d57168345fdf1a3aac66a51f6a45d41d2 d5f6e1ea452285324836a49d7d3c2a63cfed1d31 0000000000000000000000000000000000000000 ff42371d57168345fdf1a3aac66a51f6a45d41d2 d5f6e1ea452285324836a49d7d3c2a63cfed1d31 99
  bac16991d12ff45f9dc43c52da1946dfadb83e80 ff42371d57168345fdf1a3aac66a51f6a45d41d2 0000000000000000000000000000000000000000 bac16991d12ff45f9dc43c52da1946dfadb83e80 ff42371d57168345fdf1a3aac66a51f6a45d41d2 99
  63476832d8ec6558cf9bbe3cbe0c757e5cf18043 700b7e19db54103633c4bf4a6a6b6d55f4d50c03 0000000000000000000000000000000000000000 63476832d8ec6558cf9bbe3cbe0c757e5cf18043 bac16991d12ff45f9dc43c52da1946dfadb83e80 102
  13c0170174366b441dc68e8e33757232fa744458 63476832d8ec6558cf9bbe3cbe0c757e5cf18043 0000000000000000000000000000000000000000 13c0170174366b441dc68e8e33757232fa744458 63476832d8ec6558cf9bbe3cbe0c757e5cf18043 102
  
  manifest
  dac7984588fc4eea7acbf39693a9c1b06f5b175d 591f732a3faf1fb903815273f3c199a514a61ccb 0000000000000000000000000000000000000000 ff42371d57168345fdf1a3aac66a51f6a45d41d2 591f732a3faf1fb903815273f3c199a514a61ccb 113
  0772616e6b48a76afb6c1458e193cbb3dae2e4ff dac7984588fc4eea7acbf39693a9c1b06f5b175d 0000000000000000000000000000000000000000 bac16991d12ff45f9dc43c52da1946dfadb83e80 dac7984588fc4eea7acbf39693a9c1b06f5b175d 113
  eb498cd9af6c44108e43041e951ce829e29f6c80 bff2f4817ced57b386caf7c4e3e36a4bc9af7e93 0000000000000000000000000000000000000000 63476832d8ec6558cf9bbe3cbe0c757e5cf18043 0772616e6b48a76afb6c1458e193cbb3dae2e4ff 295
  b15709c071ddd2d93188508ba156196ab4f19620 eb498cd9af6c44108e43041e951ce829e29f6c80 0000000000000000000000000000000000000000 13c0170174366b441dc68e8e33757232fa744458 eb498cd9af6c44108e43041e951ce829e29f6c80 114
  
  mf
  4f73f97080266ab8e0c0561ca8d0da3eaf65b695 301ca08d026bb72cb4258a9d211bdf7ca0bcd810 0000000000000000000000000000000000000000 ff42371d57168345fdf1a3aac66a51f6a45d41d2 301ca08d026bb72cb4258a9d211bdf7ca0bcd810 17
  c7b583de053293870e145f45bd2d61643563fd06 4f73f97080266ab8e0c0561ca8d0da3eaf65b695 0000000000000000000000000000000000000000 bac16991d12ff45f9dc43c52da1946dfadb83e80 4f73f97080266ab8e0c0561ca8d0da3eaf65b695 18
  266ee3c0302a5a18f1cf96817ac79a51836179e9 edc0f6b8db80d68ae6aff2b19f7e5347ab68fa63 0000000000000000000000000000000000000000 63476832d8ec6558cf9bbe3cbe0c757e5cf18043 c7b583de053293870e145f45bd2d61643563fd06 149
  698c6a36220548cd3903ca7dada27c59aa500c52 266ee3c0302a5a18f1cf96817ac79a51836179e9 0000000000000000000000000000000000000000 13c0170174366b441dc68e8e33757232fa744458 266ee3c0302a5a18f1cf96817ac79a51836179e9 19
  
  nf11
  33fbc651630ffa7ccbebfe4eb91320a873e7291c 0000000000000000000000000000000000000000 0000000000000000000000000000000000000000 63476832d8ec6558cf9bbe3cbe0c757e5cf18043 0000000000000000000000000000000000000000 16
  
  nf12
  ddce0544363f037e9fb889faca058f52dc01c0a5 0000000000000000000000000000000000000000 0000000000000000000000000000000000000000 13c0170174366b441dc68e8e33757232fa744458 0000000000000000000000000000000000000000 16
  
  nf4
  3c1407305701051cbed9f9cb9a68bdfb5997c235 0000000000000000000000000000000000000000 0000000000000000000000000000000000000000 ff42371d57168345fdf1a3aac66a51f6a45d41d2 0000000000000000000000000000000000000000 15
  
  nf5
  0dbd89c185f53a1727c54cd1ce256482fa23968e 0000000000000000000000000000000000000000 0000000000000000000000000000000000000000 bac16991d12ff45f9dc43c52da1946dfadb83e80 0000000000000000000000000000000000000000 15

Verify we hit the HTTP server:

  $ cat access.log
  * - - [*] "GET /?cmd=capabilities HTTP/1.1" 200 - (glob)
  * - - [*] "GET /?cmd=getbundle HTTP/1.1" 200 - (glob)
  * - - [*] "GET /?cmd=capabilities HTTP/1.1" 200 - (glob)
  * - - [*] "GET /?cmd=getbundle HTTP/1.1" 200 - x-hgarg-1:common=700b7e19db54103633c4bf4a6a6b6d55f4d50c03+d5f6e1ea452285324836a49d7d3c2a63cfed1d31&heads=13c0170174366b441dc68e8e33757232fa744458+bac16991d12ff45f9dc43c52da1946dfadb83e80 (glob)

  $ cat error.log