Mercurial > hg
view tests/test-histedit-edit.t @ 26623:5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Cloning can be an expensive operation for servers because the server
generates a bundle from existing repository data at request time. For
a large repository like mozilla-central, this consumes 4+ minutes
of CPU time on the server. It also results in significant network
utilization. Multiplied by hundreds or even thousands of clients and
the ensuing load can result in difficulties scaling the Mercurial server.
Despite generation of bundles being deterministic until the next
changeset is added, the generation of bundles to service a clone request
is not cached. Each clone thus performs redundant work. This is
wasteful.
This patch introduces the "clonebundles" extension and related
client-side functionality to help alleviate this deficiency. The
client-side feature is behind an experimental flag and is not enabled by
default.
It works as follows:
1) Server operator generates a bundle and makes it available on a
server (likely HTTP).
2) Server operator defines the URL of a bundle file in a
.hg/clonebundles.manifest file.
3) Client `hg clone`ing sees the server is advertising bundle URLs.
4) Client fetches and applies the advertised bundle.
5) Client performs equivalent of `hg pull` to fetch changes made since
the bundle was created.
Essentially, the server performs the expensive work of generating a
bundle once and all subsequent clones fetch a static file from
somewhere. Scaling static file serving is a much more manageable
problem than scaling a Python application like Mercurial. Assuming your
repository grows less than 1% per day, the end result is 99+% of CPU
and network load from clones is eliminated, allowing Mercurial servers
to scale more easily. Serving static files also means data can be
transferred to clients as fast as they can consume it, rather than as
fast as servers can generate it. This makes clones faster.
Mozilla has implemented similar functionality of this patch on
hg.mozilla.org using a custom extension. We are hosting bundle files in
Amazon S3 and CloudFront (a CDN) and have successfully offloaded
>1 TB/day in data transfer from hg.mozilla.org, freeing up significant
bandwidth and CPU resources. The positive impact has been stellar and
I believe it has proved its value to be included in Mercurial core. I
feel it is important for the client-side support to be enabled in core
by default because it means that clients will get faster, more reliable
clones and will enable server operators to reduce load without
requiring any client-side configuration changes (assuming clients are
up to date, of course).
The scope of this feature is narrowly and specifically tailored to
cloning, despite "serve pulls from pre-generated bundles" being a valid
and useful feature. I would eventually like for Mercurial servers to
support transferring *all* repository data via statically hosted files.
You could imagine a server that siphons all pushed data to bundle files
and instructs clients to apply a stream of bundles to reconstruct all
repository data. This feature, while useful and powerful, is
significantly more work to implement because it requires the server
component have awareness of discovery and a mapping of which changesets
are in which files. Full, clone bundles, by contrast, are much simpler.
The wire protocol command is named "clonebundles" instead of something
more generic like "staticbundles" to leave the door open for a new, more
powerful and more generic server-side component with minimal backwards
compatibility implications. The name "bundleclone" is used by Mozilla's
extension and would cause problems since there are subtle differences
in Mozilla's extension.
Mozilla's experience with this idea has taught us that some form of
"content negotiation" is required. Not all clients will support all
bundle formats or even URLs (advanced TLS requirements, etc). To ensure
the highest uptake possible, a server needs to advertise multiple
versions of bundles and clients need to be able to choose the most
appropriate from that list one. The "attributes" in each
server-advertised entry facilitate this filtering and sorting. Their
use will become apparent in subsequent patches.
Initial inspiration and credit for the idea of cloning from static files
belongs to Augie Fackler and his "lookaside clone" extension proof of
concept.
author | Gregory Szorc <gregory.szorc@gmail.com> |
---|---|
date | Fri, 09 Oct 2015 11:22:01 -0700 |
parents | 56b2bcea2529 |
children | 4414d500604f |
line wrap: on
line source
$ . "$TESTDIR/histedit-helpers.sh" $ cat >> $HGRCPATH <<EOF > [extensions] > histedit= > strip= > EOF $ initrepo () > { > hg init r > cd r > for x in a b c d e f g; do > echo $x > $x > hg add $x > hg ci -m $x > done > } $ initrepo log before edit $ hg log --graph @ changeset: 6:3c6a8ed2ebe8 | tag: tip | user: test | date: Thu Jan 01 00:00:00 1970 +0000 | summary: g | o changeset: 5:652413bf663e | user: test | date: Thu Jan 01 00:00:00 1970 +0000 | summary: f | o changeset: 4:e860deea161a | user: test | date: Thu Jan 01 00:00:00 1970 +0000 | summary: e | o changeset: 3:055a42cdd887 | user: test | date: Thu Jan 01 00:00:00 1970 +0000 | summary: d | o changeset: 2:177f92b77385 | user: test | date: Thu Jan 01 00:00:00 1970 +0000 | summary: c | o changeset: 1:d2ae7f538514 | user: test | date: Thu Jan 01 00:00:00 1970 +0000 | summary: b | o changeset: 0:cb9a9f314b8b user: test date: Thu Jan 01 00:00:00 1970 +0000 summary: a edit the history $ hg histedit 177f92b77385 --commands - 2>&1 << EOF| fixbundle > pick 177f92b77385 c > pick 055a42cdd887 d > edit e860deea161a e > pick 652413bf663e f > pick 3c6a8ed2ebe8 g > EOF 0 files updated, 0 files merged, 3 files removed, 0 files unresolved Make changes as needed, you may commit or record as needed now. When you are finished, run hg histedit --continue to resume. edit the plan via the editor $ cat >> $TESTTMP/editplan.sh <<EOF > cat > \$1 <<EOF2 > drop e860deea161a e > drop 652413bf663e f > drop 3c6a8ed2ebe8 g > EOF2 > EOF $ HGEDITOR="sh $TESTTMP/editplan.sh" hg histedit --edit-plan $ cat .hg/histedit-state v1 055a42cdd88768532f9cf79daa407fc8d138de9b 3c6a8ed2ebe862cc949d2caa30775dd6f16fb799 False 3 drop e860deea161a2f77de56603b340ebbb4536308ae drop 652413bf663ef2a641cab26574e46d5f5a64a55a drop 3c6a8ed2ebe862cc949d2caa30775dd6f16fb799 0 strip-backup/177f92b77385-0ebe6a8f-histedit.hg edit the plan via --commands $ hg histedit --edit-plan --commands - 2>&1 << EOF > edit e860deea161a e > pick 652413bf663e f > drop 3c6a8ed2ebe8 g > EOF $ cat .hg/histedit-state v1 055a42cdd88768532f9cf79daa407fc8d138de9b 3c6a8ed2ebe862cc949d2caa30775dd6f16fb799 False 3 edit e860deea161a2f77de56603b340ebbb4536308ae pick 652413bf663ef2a641cab26574e46d5f5a64a55a drop 3c6a8ed2ebe862cc949d2caa30775dd6f16fb799 0 strip-backup/177f92b77385-0ebe6a8f-histedit.hg Go at a random point and try to continue $ hg id -n 3+ $ hg up 0 abort: histedit in progress (use 'hg histedit --continue' or 'hg histedit --abort') [255] Try to delete necessary commit $ hg strip -r 652413b abort: histedit in progress, can't strip 652413bf663e [255] commit, then edit the revision $ hg ci -m 'wat' created new head $ echo a > e qnew should fail while we're in the middle of the edit step $ hg --config extensions.mq= qnew please-fail abort: histedit in progress (use 'hg histedit --continue' or 'hg histedit --abort') [255] $ HGEDITOR='echo foobaz > ' hg histedit --continue 2>&1 | fixbundle 0 files updated, 0 files merged, 0 files removed, 0 files unresolved 0 files updated, 0 files merged, 0 files removed, 0 files unresolved $ hg log --graph @ changeset: 6:b5f70786f9b0 | tag: tip | user: test | date: Thu Jan 01 00:00:00 1970 +0000 | summary: f | o changeset: 5:a5e1ba2f7afb | user: test | date: Thu Jan 01 00:00:00 1970 +0000 | summary: foobaz | o changeset: 4:1a60820cd1f6 | user: test | date: Thu Jan 01 00:00:00 1970 +0000 | summary: wat | o changeset: 3:055a42cdd887 | user: test | date: Thu Jan 01 00:00:00 1970 +0000 | summary: d | o changeset: 2:177f92b77385 | user: test | date: Thu Jan 01 00:00:00 1970 +0000 | summary: c | o changeset: 1:d2ae7f538514 | user: test | date: Thu Jan 01 00:00:00 1970 +0000 | summary: b | o changeset: 0:cb9a9f314b8b user: test date: Thu Jan 01 00:00:00 1970 +0000 summary: a $ hg cat e a Stripping necessary commits should not break --abort $ hg histedit 1a60820cd1f6 --commands - 2>&1 << EOF| fixbundle > edit 1a60820cd1f6 wat > pick a5e1ba2f7afb foobaz > pick b5f70786f9b0 g > EOF 0 files updated, 0 files merged, 2 files removed, 0 files unresolved Make changes as needed, you may commit or record as needed now. When you are finished, run hg histedit --continue to resume. $ mv .hg/histedit-state .hg/histedit-state.bak $ hg strip -q -r b5f70786f9b0 $ mv .hg/histedit-state.bak .hg/histedit-state $ hg histedit --abort adding changesets adding manifests adding file changes added 1 changesets with 1 changes to 3 files 2 files updated, 0 files merged, 0 files removed, 0 files unresolved $ hg log -r . changeset: 6:b5f70786f9b0 tag: tip user: test date: Thu Jan 01 00:00:00 1970 +0000 summary: f check histedit_source $ hg log --debug --rev 5 changeset: 5:a5e1ba2f7afb899ef1581cea528fd885d2fca70d phase: draft parent: 4:1a60820cd1f6004a362aa622ebc47d59bc48eb34 parent: -1:0000000000000000000000000000000000000000 manifest: 5:5ad3be8791f39117565557781f5464363b918a45 user: test date: Thu Jan 01 00:00:00 1970 +0000 files: e extra: branch=default extra: histedit_source=e860deea161a2f77de56603b340ebbb4536308ae description: foobaz $ hg histedit tip --commands - 2>&1 <<EOF| fixbundle > edit b5f70786f9b0 f > EOF 0 files updated, 0 files merged, 1 files removed, 0 files unresolved Make changes as needed, you may commit or record as needed now. When you are finished, run hg histedit --continue to resume. $ hg status A f $ hg summary parent: 5:a5e1ba2f7afb foobaz branch: default commit: 1 added (new branch head) update: 1 new changesets (update) phases: 7 draft hist: 1 remaining (histedit --continue) (test also that editor is invoked if histedit is continued for "edit" action) $ HGEDITOR='cat' hg histedit --continue f HG: Enter commit message. Lines beginning with 'HG:' are removed. HG: Leave message empty to abort commit. HG: -- HG: user: test HG: branch 'default' HG: added f 0 files updated, 0 files merged, 0 files removed, 0 files unresolved saved backup bundle to $TESTTMP/r/.hg/strip-backup/b5f70786f9b0-c28d9c86-backup.hg (glob) $ hg status log after edit $ hg log --limit 1 changeset: 6:a107ee126658 tag: tip user: test date: Thu Jan 01 00:00:00 1970 +0000 summary: f say we'll change the message, but don't. $ cat > ../edit.sh <<EOF > cat "\$1" | sed s/pick/mess/ > tmp > mv tmp "\$1" > EOF $ HGEDITOR="sh ../edit.sh" hg histedit tip 2>&1 | fixbundle 0 files updated, 0 files merged, 1 files removed, 0 files unresolved 0 files updated, 0 files merged, 0 files removed, 0 files unresolved $ hg status $ hg log --limit 1 changeset: 6:1fd3b2fe7754 tag: tip user: test date: Thu Jan 01 00:00:00 1970 +0000 summary: f modify the message check saving last-message.txt, at first $ cat > $TESTTMP/commitfailure.py <<EOF > from mercurial import error > def reposetup(ui, repo): > class commitfailure(repo.__class__): > def commit(self, *args, **kwargs): > raise error.Abort('emulating unexpected abort') > repo.__class__ = commitfailure > EOF $ cat >> .hg/hgrc <<EOF > [extensions] > # this failure occurs before editor invocation > commitfailure = $TESTTMP/commitfailure.py > EOF $ cat > $TESTTMP/editor.sh <<EOF > echo "==== before editing" > cat \$1 > echo "====" > echo "check saving last-message.txt" >> \$1 > EOF (test that editor is not invoked before transaction starting) $ rm -f .hg/last-message.txt $ HGEDITOR="sh $TESTTMP/editor.sh" hg histedit tip --commands - 2>&1 << EOF | fixbundle > mess 1fd3b2fe7754 f > EOF 0 files updated, 0 files merged, 1 files removed, 0 files unresolved abort: emulating unexpected abort $ test -f .hg/last-message.txt [1] $ cat >> .hg/hgrc <<EOF > [extensions] > commitfailure = ! > EOF $ hg histedit --abort -q (test that editor is invoked and commit message is saved into "last-message.txt") $ cat >> .hg/hgrc <<EOF > [hooks] > # this failure occurs after editor invocation > pretxncommit.unexpectedabort = false > EOF $ hg status --rev '1fd3b2fe7754^1' --rev 1fd3b2fe7754 A f $ rm -f .hg/last-message.txt $ HGEDITOR="sh $TESTTMP/editor.sh" hg histedit tip --commands - 2>&1 << EOF > mess 1fd3b2fe7754 f > EOF 0 files updated, 0 files merged, 1 files removed, 0 files unresolved adding f ==== before editing f HG: Enter commit message. Lines beginning with 'HG:' are removed. HG: Leave message empty to abort commit. HG: -- HG: user: test HG: branch 'default' HG: added f ==== transaction abort! rollback completed note: commit message saved in .hg/last-message.txt abort: pretxncommit.unexpectedabort hook exited with status 1 [255] $ cat .hg/last-message.txt f check saving last-message.txt (test also that editor is invoked if histedit is continued for "message" action) $ HGEDITOR=cat hg histedit --continue f HG: Enter commit message. Lines beginning with 'HG:' are removed. HG: Leave message empty to abort commit. HG: -- HG: user: test HG: branch 'default' HG: added f transaction abort! rollback completed note: commit message saved in .hg/last-message.txt abort: pretxncommit.unexpectedabort hook exited with status 1 [255] $ cat >> .hg/hgrc <<EOF > [hooks] > pretxncommit.unexpectedabort = > EOF $ hg histedit --abort -q then, check "modify the message" itself $ hg histedit tip --commands - 2>&1 << EOF | fixbundle > mess 1fd3b2fe7754 f > EOF 0 files updated, 0 files merged, 1 files removed, 0 files unresolved 0 files updated, 0 files merged, 0 files removed, 0 files unresolved $ hg status $ hg log --limit 1 changeset: 6:62feedb1200e tag: tip user: test date: Thu Jan 01 00:00:00 1970 +0000 summary: f rollback should not work after a histedit $ hg rollback no rollback information available [1] $ cd .. $ hg clone -qr0 r r0 $ cd r0 $ hg phase -fdr0 $ hg histedit --commands - 0 2>&1 << EOF > edit cb9a9f314b8b a > $EDITED > EOF 0 files updated, 0 files merged, 1 files removed, 0 files unresolved adding a Make changes as needed, you may commit or record as needed now. When you are finished, run hg histedit --continue to resume. [1] $ HGEDITOR=true hg histedit --continue 0 files updated, 0 files merged, 0 files removed, 0 files unresolved saved backup bundle to $TESTTMP/r0/.hg/strip-backup/cb9a9f314b8b-cc5ccb0b-backup.hg (glob) $ hg log -G @ changeset: 0:0efcea34f18a tag: tip user: test date: Thu Jan 01 00:00:00 1970 +0000 summary: a $ echo foo >> b $ hg addr adding b $ hg ci -m 'add b' $ echo foo >> a $ hg ci -m 'extend a' $ hg phase --public 1 Attempting to fold a change into a public change should not work: $ cat > ../edit.sh <<EOF > cat "\$1" | sed s/pick/fold/ > tmp > mv tmp "\$1" > EOF $ HGEDITOR="sh ../edit.sh" hg histedit 2 1 files updated, 0 files merged, 0 files removed, 0 files unresolved reverting a 1 files updated, 0 files merged, 1 files removed, 0 files unresolved abort: cannot fold into public change 18aa70c8ad22 [255] TODO: this abort shouldn't be required, but it is for now to leave the repo in a clean state. $ hg histedit --abort