Mercurial > hg
view tests/test-convert-mtn.t @ 26623:5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Cloning can be an expensive operation for servers because the server
generates a bundle from existing repository data at request time. For
a large repository like mozilla-central, this consumes 4+ minutes
of CPU time on the server. It also results in significant network
utilization. Multiplied by hundreds or even thousands of clients and
the ensuing load can result in difficulties scaling the Mercurial server.
Despite generation of bundles being deterministic until the next
changeset is added, the generation of bundles to service a clone request
is not cached. Each clone thus performs redundant work. This is
wasteful.
This patch introduces the "clonebundles" extension and related
client-side functionality to help alleviate this deficiency. The
client-side feature is behind an experimental flag and is not enabled by
default.
It works as follows:
1) Server operator generates a bundle and makes it available on a
server (likely HTTP).
2) Server operator defines the URL of a bundle file in a
.hg/clonebundles.manifest file.
3) Client `hg clone`ing sees the server is advertising bundle URLs.
4) Client fetches and applies the advertised bundle.
5) Client performs equivalent of `hg pull` to fetch changes made since
the bundle was created.
Essentially, the server performs the expensive work of generating a
bundle once and all subsequent clones fetch a static file from
somewhere. Scaling static file serving is a much more manageable
problem than scaling a Python application like Mercurial. Assuming your
repository grows less than 1% per day, the end result is 99+% of CPU
and network load from clones is eliminated, allowing Mercurial servers
to scale more easily. Serving static files also means data can be
transferred to clients as fast as they can consume it, rather than as
fast as servers can generate it. This makes clones faster.
Mozilla has implemented similar functionality of this patch on
hg.mozilla.org using a custom extension. We are hosting bundle files in
Amazon S3 and CloudFront (a CDN) and have successfully offloaded
>1 TB/day in data transfer from hg.mozilla.org, freeing up significant
bandwidth and CPU resources. The positive impact has been stellar and
I believe it has proved its value to be included in Mercurial core. I
feel it is important for the client-side support to be enabled in core
by default because it means that clients will get faster, more reliable
clones and will enable server operators to reduce load without
requiring any client-side configuration changes (assuming clients are
up to date, of course).
The scope of this feature is narrowly and specifically tailored to
cloning, despite "serve pulls from pre-generated bundles" being a valid
and useful feature. I would eventually like for Mercurial servers to
support transferring *all* repository data via statically hosted files.
You could imagine a server that siphons all pushed data to bundle files
and instructs clients to apply a stream of bundles to reconstruct all
repository data. This feature, while useful and powerful, is
significantly more work to implement because it requires the server
component have awareness of discovery and a mapping of which changesets
are in which files. Full, clone bundles, by contrast, are much simpler.
The wire protocol command is named "clonebundles" instead of something
more generic like "staticbundles" to leave the door open for a new, more
powerful and more generic server-side component with minimal backwards
compatibility implications. The name "bundleclone" is used by Mozilla's
extension and would cause problems since there are subtle differences
in Mozilla's extension.
Mozilla's experience with this idea has taught us that some form of
"content negotiation" is required. Not all clients will support all
bundle formats or even URLs (advanced TLS requirements, etc). To ensure
the highest uptake possible, a server needs to advertise multiple
versions of bundles and clients need to be able to choose the most
appropriate from that list one. The "attributes" in each
server-advertised entry facilitate this filtering and sorting. Their
use will become apparent in subsequent patches.
Initial inspiration and credit for the idea of cloning from static files
belongs to Augie Fackler and his "lookaside clone" extension proof of
concept.
author | Gregory Szorc <gregory.szorc@gmail.com> |
---|---|
date | Fri, 09 Oct 2015 11:22:01 -0700 |
parents | 4d2b9b304ad0 |
children | c2e4e59aaea6 |
line wrap: on
line source
#require mtn Monotone directory is called .monotone on *nix and monotone on Windows. #if windows $ mtndir=monotone #else $ mtndir=.monotone #endif $ echo "[extensions]" >> $HGRCPATH $ echo "convert=" >> $HGRCPATH Windows version of monotone home $ APPDATA=$HOME; export APPDATA tedious monotone keys configuration The /dev/null redirection is necessary under Windows, or it complains about home directory permissions $ mtn --quiet genkey test@selenic.com 1>/dev/null 2>&1 <<EOF > passphrase > passphrase > EOF $ cat >> $HOME/$mtndir/monotonerc <<EOF > function get_passphrase(keypair_id) > return "passphrase" > end > EOF create monotone repository $ mtn db init --db=repo.mtn $ mtn --db=repo.mtn --branch=com.selenic.test setup workingdir $ cd workingdir $ echo a > a $ mkdir dir $ echo b > dir/b $ echo d > dir/d $ $PYTHON -c 'file("bin", "wb").write("a\\x00b")' $ echo c > c $ mtn add a dir/b dir/d c bin mtn: adding 'a' to workspace manifest mtn: adding 'bin' to workspace manifest mtn: adding 'c' to workspace manifest mtn: adding 'dir' to workspace manifest mtn: adding 'dir/b' to workspace manifest mtn: adding 'dir/d' to workspace manifest $ mtn ci -m initialize mtn: beginning commit on branch 'com.selenic.test' mtn: committed revision 0f6e5e4f2e7d2a8ef312408f57618abf026afd90 update monotone working directory $ mtn mv a dir/a mtn: skipping 'dir', already accounted for in workspace mtn: renaming 'a' to 'dir/a' in workspace manifest $ echo a >> dir/a $ echo b >> dir/b $ mtn drop c mtn: dropping 'c' from workspace manifest $ $PYTHON -c 'file("bin", "wb").write("b\\x00c")' $ mtn ci -m update1 mtn: beginning commit on branch 'com.selenic.test' mtn: committed revision 51d0a982464573a2a2cf5ee2c9219c652aaebeff $ cd .. convert once $ hg convert -s mtn repo.mtn assuming destination repo.mtn-hg initializing destination repo.mtn-hg repository scanning source... sorting... converting... 1 initialize 0 update1 $ cd workingdir $ echo e > e $ mtn add e mtn: adding 'e' to workspace manifest $ mtn drop dir/b mtn: dropping 'dir/b' from workspace manifest $ mtn mv bin bin2 mtn: renaming 'bin' to 'bin2' in workspace manifest $ mtn ci -m 'update2 "with" quotes' mtn: beginning commit on branch 'com.selenic.test' mtn: committed revision ebe58335d85d8cb176b6d0a12be04f5314b998da test directory move $ mkdir -p dir1/subdir1 $ mkdir -p dir1/subdir2_other $ echo file1 > dir1/subdir1/file1 $ echo file2 > dir1/subdir2_other/file1 $ mtn add dir1/subdir1/file1 dir1/subdir2_other/file1 mtn: adding 'dir1' to workspace manifest mtn: adding 'dir1/subdir1' to workspace manifest mtn: adding 'dir1/subdir1/file1' to workspace manifest mtn: adding 'dir1/subdir2_other' to workspace manifest mtn: adding 'dir1/subdir2_other/file1' to workspace manifest $ mtn ci -m createdir1 mtn: beginning commit on branch 'com.selenic.test' mtn: committed revision a8d62bc04fee4d2936d28e98bbcc81686dd74306 $ mtn rename dir1/subdir1 dir1/subdir2 mtn: skipping 'dir1', already accounted for in workspace mtn: renaming 'dir1/subdir1' to 'dir1/subdir2' in workspace manifest $ mtn ci -m movedir1 mtn: beginning commit on branch 'com.selenic.test' mtn: committed revision 2c3d241bbbfe538b1b51d910f5676407e3f4d3a6 test subdirectory move $ mtn mv dir dir2 mtn: renaming 'dir' to 'dir2' in workspace manifest $ echo newfile > dir2/newfile $ mtn drop dir2/d mtn: dropping 'dir2/d' from workspace manifest $ mtn add dir2/newfile mtn: adding 'dir2/newfile' to workspace manifest $ mtn ci -m movedir mtn: beginning commit on branch 'com.selenic.test' mtn: committed revision fdb5a02dae8bfce3a79b3393680af471016e1b4c Test directory removal with empty directory $ mkdir dir2/dir $ mkdir dir2/dir/subdir $ echo f > dir2/dir/subdir/f $ mkdir dir2/dir/emptydir $ mtn add --quiet -R dir2/dir $ mtn ci -m emptydir mtn: beginning commit on branch 'com.selenic.test' mtn: committed revision 8bbf76d717001d24964e4604739fdcd0f539fc88 $ mtn drop -R dir2/dir mtn: dropping 'dir2/dir/subdir/f' from workspace manifest mtn: dropping 'dir2/dir/subdir' from workspace manifest mtn: dropping 'dir2/dir/emptydir' from workspace manifest mtn: dropping 'dir2/dir' from workspace manifest $ mtn ci -m dropdirectory mtn: beginning commit on branch 'com.selenic.test' mtn: committed revision 2323d4bc324e6c82628dc04d47a9fd32ad24e322 test directory and file move $ mkdir -p dir3/d1 $ echo a > dir3/a $ mtn add dir3/a dir3/d1 mtn: adding 'dir3' to workspace manifest mtn: adding 'dir3/a' to workspace manifest mtn: adding 'dir3/d1' to workspace manifest $ mtn ci -m dirfilemove mtn: beginning commit on branch 'com.selenic.test' mtn: committed revision 47b192f720faa622f48c68d1eb075b26d405aa8b $ mtn mv dir3/a dir3/d1/a mtn: skipping 'dir3/d1', already accounted for in workspace mtn: renaming 'dir3/a' to 'dir3/d1/a' in workspace manifest $ mtn mv dir3/d1 dir3/d2 mtn: skipping 'dir3', already accounted for in workspace mtn: renaming 'dir3/d1' to 'dir3/d2' in workspace manifest $ mtn ci -m dirfilemove2 mtn: beginning commit on branch 'com.selenic.test' mtn: committed revision 8b543a400d3ee7f6d4bb1835b9b9e3747c8cb632 test directory move into another directory move $ mkdir dir4 $ mkdir dir5 $ echo a > dir4/a $ mtn add dir4/a dir5 mtn: adding 'dir4' to workspace manifest mtn: adding 'dir4/a' to workspace manifest mtn: adding 'dir5' to workspace manifest $ mtn ci -m dirdirmove mtn: beginning commit on branch 'com.selenic.test' mtn: committed revision 466e0b2afc7a55aa2b4ab2f57cb240bb6cd66fc7 $ mtn mv dir5 dir6 mtn: renaming 'dir5' to 'dir6' in workspace manifest $ mtn mv dir4 dir6/dir4 mtn: skipping 'dir6', already accounted for in workspace mtn: renaming 'dir4' to 'dir6/dir4' in workspace manifest $ mtn ci -m dirdirmove2 mtn: beginning commit on branch 'com.selenic.test' mtn: committed revision 3d1f77ebad0c23a5d14911be3b670f990991b749 test diverging directory moves $ mkdir -p dir7/dir9/dir8 $ echo a > dir7/dir9/dir8/a $ echo b > dir7/dir9/b $ echo c > dir7/c $ mtn add -R dir7 mtn: adding 'dir7' to workspace manifest mtn: adding 'dir7/c' to workspace manifest mtn: adding 'dir7/dir9' to workspace manifest mtn: adding 'dir7/dir9/b' to workspace manifest mtn: adding 'dir7/dir9/dir8' to workspace manifest mtn: adding 'dir7/dir9/dir8/a' to workspace manifest $ mtn ci -m divergentdirmove mtn: beginning commit on branch 'com.selenic.test' mtn: committed revision 08a08511f18b428d840199b062de90d0396bc2ed $ mtn mv dir7 dir7-2 mtn: renaming 'dir7' to 'dir7-2' in workspace manifest $ mtn mv dir7-2/dir9 dir9-2 mtn: renaming 'dir7-2/dir9' to 'dir9-2' in workspace manifest $ mtn mv dir9-2/dir8 dir8-2 mtn: renaming 'dir9-2/dir8' to 'dir8-2' in workspace manifest $ mtn ci -m divergentdirmove2 mtn: beginning commit on branch 'com.selenic.test' mtn: committed revision 4a736634505795f17786fffdf2c9cbf5b11df6f6 test large file support (> 32kB) >>> fp = file('large-file', 'wb') >>> for x in xrange(10000): fp.write('%d\n' % x) >>> fp.close() $ md5sum.py large-file 5d6de8a95c3b6bf9e0ffb808ba5299c1 large-file $ mtn add large-file mtn: adding 'large-file' to workspace manifest $ mtn ci -m largefile mtn: beginning commit on branch 'com.selenic.test' mtn: committed revision f0a20fecd10dc4392d18fe69a03f1f4919d3387b test suspending (closing a branch) $ mtn suspend f0a20fecd10dc4392d18fe69a03f1f4919d3387b 2> /dev/null $ cd .. convert incrementally $ hg convert -s mtn repo.mtn assuming destination repo.mtn-hg scanning source... sorting... converting... 12 update2 "with" quotes 11 createdir1 10 movedir1 9 movedir 8 emptydir 7 dropdirectory 6 dirfilemove 5 dirfilemove2 4 dirdirmove 3 dirdirmove2 2 divergentdirmove 1 divergentdirmove2 0 largefile $ glog() > { > hg log -G --template '{rev} "{desc|firstline}" files: {files}\n' "$@" > } $ cd repo.mtn-hg $ hg up -C 12 files updated, 0 files merged, 0 files removed, 0 files unresolved $ glog @ 14 "largefile" files: large-file | o 13 "divergentdirmove2" files: dir7-2/c dir7/c dir7/dir9/b dir7/dir9/dir8/a dir8-2/a dir9-2/b | o 12 "divergentdirmove" files: dir7/c dir7/dir9/b dir7/dir9/dir8/a | o 11 "dirdirmove2" files: dir4/a dir6/dir4/a | o 10 "dirdirmove" files: dir4/a | o 9 "dirfilemove2" files: dir3/a dir3/d2/a | o 8 "dirfilemove" files: dir3/a | o 7 "dropdirectory" files: dir2/dir/subdir/f | o 6 "emptydir" files: dir2/dir/subdir/f | o 5 "movedir" files: dir/a dir/d dir2/a dir2/newfile | o 4 "movedir1" files: dir1/subdir1/file1 dir1/subdir2/file1 | o 3 "createdir1" files: dir1/subdir1/file1 dir1/subdir2_other/file1 | o 2 "update2 "with" quotes" files: bin bin2 dir/b e | o 1 "update1" files: a bin c dir/a dir/b | o 0 "initialize" files: a bin c dir/b dir/d manifest $ hg manifest bin2 dir1/subdir2/file1 dir1/subdir2_other/file1 dir2/a dir2/newfile dir3/d2/a dir6/dir4/a dir7-2/c dir8-2/a dir9-2/b e large-file contents $ cat dir2/a a a $ test -d dir2/dir && echo 'removed dir2/dir is still there!' [1] file move $ hg log -v -C -r 1 | grep copies copies: dir/a (a) check directory move $ hg manifest -r 4 bin2 dir/a dir/d dir1/subdir2/file1 dir1/subdir2_other/file1 e $ test -d dir1/subdir2 || echo 'new dir1/subdir2 does not exist!' $ test -d dir1/subdir1 && echo 'renamed dir1/subdir1 is still there!' [1] $ hg log -v -C -r 4 | grep copies copies: dir1/subdir2/file1 (dir1/subdir1/file1) check file remove with directory move $ hg manifest -r 5 bin2 dir1/subdir2/file1 dir1/subdir2_other/file1 dir2/a dir2/newfile e check file move with directory move $ hg manifest -r 9 bin2 dir1/subdir2/file1 dir1/subdir2_other/file1 dir2/a dir2/newfile dir3/d2/a e check file directory directory move $ hg manifest -r 11 bin2 dir1/subdir2/file1 dir1/subdir2_other/file1 dir2/a dir2/newfile dir3/d2/a dir6/dir4/a e check divergent directory moves $ hg manifest -r 13 bin2 dir1/subdir2/file1 dir1/subdir2_other/file1 dir2/a dir2/newfile dir3/d2/a dir6/dir4/a dir7-2/c dir8-2/a dir9-2/b e test large file support (> 32kB) $ md5sum.py large-file 5d6de8a95c3b6bf9e0ffb808ba5299c1 large-file check branch closing $ hg branches -a $ hg branches -c com.selenic.test 14:* (closed) (glob)