streamclone: also stream caches to the client
When stream clone is used over bundle2, relevant cache files are also streamed.
This is expected to be a massive performance win for clone since no important
cache will have to be recomputed.
Some performance numbers:
(All times are wall-clock times in seconds, 2 attempts per case.)
# Mozilla-Central
## Clone over ssh over lan
V1 streaming: 234.3 239.6
V2 streaming: 248.4 243.7
## Clone over ssh over Internet
V1 streaming: 175.5 110.9
V2 streaming: 109.1 111.0
## Clone over HTTP over lan
V1 streaming: 105.3 105.6
V2 streaming: 112.7 111.4
## Clone over HTTP over internet
V1 streaming: 105.6 114.6
V2 streaming: 226.7 225.9
## Hg tags
V1 streaming (no cache): 1.084 1.071
V2 streaming (cache): 0.312 0.325
## Hg branches
V1 streaming (no cache): 14.047 14.148
V2 streaming (with cache): 0.312 0.333
# Pypy
## Clone over ssh over internet
V1 streaming: 29.4 30.1
V2 streaming: 31.2 30.1
## Clone over http over internet
V1 streaming: 29.7 29.7
V2 streaming: 75.2 72.9
(since ssh and lan are not affected, there seems to be an issue with how we
read/write the http stream on connection with latency, unrelated to the format)
## Hg tags
V1 streaming (no cache): 1.752 1.664
V2 streaming (with cache): 0.274 0.260
## Hg branches
V1 streaming (no cache): 4.469 4.728
V2 streaming (with cache): 0.318 0.321
# Private repository:
* 500K revision revisions
* 11K topological heads
* 28K branch heads
## hg tags
no cache: 1543.332
with cache: 4.900
## hg branches
no cache: 91.828
with cache: 2.955
#require killdaemons
$ hg init test
$ cd test
$ echo a > a
$ hg ci -Ama
adding a
$ cd ..
$ hg clone test test2
updating to branch default
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ cd test2
$ echo a >> a
$ hg ci -mb
$ req() {
> hg serve -p $HGPORT -d --pid-file=hg.pid -E errors.log
> cat hg.pid >> $DAEMON_PIDS
> hg --cwd ../test2 push http://localhost:$HGPORT/
> exitstatus=$?
> killdaemons.py
> echo % serve errors
> cat errors.log
> return $exitstatus
> }
$ cd ../test
expect ssl error
$ req
pushing to http://localhost:$HGPORT/
searching for changes
abort: HTTP Error 403: ssl required
% serve errors
[255]
expect authorization error
$ echo '[web]' > .hg/hgrc
$ echo 'push_ssl = false' >> .hg/hgrc
$ req
pushing to http://localhost:$HGPORT/
searching for changes
abort: authorization failed
% serve errors
[255]
expect authorization error: must have authorized user
$ echo 'allow_push = unperson' >> .hg/hgrc
$ req
pushing to http://localhost:$HGPORT/
searching for changes
abort: authorization failed
% serve errors
[255]
expect success
$ cat > $TESTTMP/hook.sh <<'EOF'
> echo "phase-move: $HG_NODE: $HG_OLDPHASE -> $HG_PHASE"
> EOF
$ cat >> .hg/hgrc <<EOF
> allow_push = *
> [hooks]
> changegroup = sh -c "printenv.py changegroup 0"
> pushkey = sh -c "printenv.py pushkey 0"
> txnclose-phase.test = sh $TESTTMP/hook.sh
> EOF
$ req
pushing to http://localhost:$HGPORT/
searching for changes
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 1 changesets with 1 changes to 1 files
remote: phase-move: cb9a9f314b8b07ba71012fcdbc544b5a4d82ff5b: draft -> public
remote: phase-move: ba677d0156c1196c1a699fa53f390dcfc3ce3872: -> public
remote: changegroup hook: HG_BUNDLE2=1 HG_HOOKNAME=changegroup HG_HOOKTYPE=changegroup HG_NODE=ba677d0156c1196c1a699fa53f390dcfc3ce3872 HG_NODE_LAST=ba677d0156c1196c1a699fa53f390dcfc3ce3872 HG_SOURCE=serve HG_TXNID=TXN:$ID$ HG_URL=remote:http:$LOCALIP: (glob)
% serve errors
$ hg rollback
repository tip rolled back to revision 0 (undo serve)
expect success, server lacks the httpheader capability
$ CAP=httpheader
$ . "$TESTDIR/notcapable"
$ req
pushing to http://localhost:$HGPORT/
searching for changes
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 1 changesets with 1 changes to 1 files
remote: phase-move: cb9a9f314b8b07ba71012fcdbc544b5a4d82ff5b: draft -> public
remote: phase-move: ba677d0156c1196c1a699fa53f390dcfc3ce3872: -> public
remote: changegroup hook: HG_BUNDLE2=1 HG_HOOKNAME=changegroup HG_HOOKTYPE=changegroup HG_NODE=ba677d0156c1196c1a699fa53f390dcfc3ce3872 HG_NODE_LAST=ba677d0156c1196c1a699fa53f390dcfc3ce3872 HG_SOURCE=serve HG_TXNID=TXN:$ID$ HG_URL=remote:http:$LOCALIP: (glob)
% serve errors
$ hg rollback
repository tip rolled back to revision 0 (undo serve)
expect success, server lacks the unbundlehash capability
$ CAP=unbundlehash
$ . "$TESTDIR/notcapable"
$ req
pushing to http://localhost:$HGPORT/
searching for changes
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 1 changesets with 1 changes to 1 files
remote: phase-move: cb9a9f314b8b07ba71012fcdbc544b5a4d82ff5b: draft -> public
remote: phase-move: ba677d0156c1196c1a699fa53f390dcfc3ce3872: -> public
remote: changegroup hook: HG_BUNDLE2=1 HG_HOOKNAME=changegroup HG_HOOKTYPE=changegroup HG_NODE=ba677d0156c1196c1a699fa53f390dcfc3ce3872 HG_NODE_LAST=ba677d0156c1196c1a699fa53f390dcfc3ce3872 HG_SOURCE=serve HG_TXNID=TXN:$ID$ HG_URL=remote:http:$LOCALIP: (glob)
% serve errors
$ hg rollback
repository tip rolled back to revision 0 (undo serve)
expect push success, phase change failure
$ cat > .hg/hgrc <<EOF
> [web]
> push_ssl = false
> allow_push = *
> [hooks]
> prepushkey = sh -c "printenv.py prepushkey 1"
> [devel]
> legacy.exchange=phases
> EOF
$ req
pushing to http://localhost:$HGPORT/
searching for changes
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 1 changesets with 1 changes to 1 files
remote: prepushkey hook: HG_BUNDLE2=1 HG_HOOKNAME=prepushkey HG_HOOKTYPE=prepushkey HG_KEY=ba677d0156c1196c1a699fa53f390dcfc3ce3872 HG_NAMESPACE=phases HG_NEW=0 HG_NODE=ba677d0156c1196c1a699fa53f390dcfc3ce3872 HG_NODE_LAST=ba677d0156c1196c1a699fa53f390dcfc3ce3872 HG_OLD=1 HG_PENDING=$TESTTMP/test HG_PHASES_MOVED=1 HG_SOURCE=serve HG_TXNID=TXN:$ID$ HG_URL=remote:http:$LOCALIP: (glob)
remote: pushkey-abort: prepushkey hook exited with status 1
remote: transaction abort!
remote: rollback completed
abort: updating ba677d0156c1 to public failed
% serve errors
[255]
expect phase change success
$ cat >> .hg/hgrc <<EOF
> prepushkey = sh -c "printenv.py prepushkey 0"
> [devel]
> legacy.exchange=
> EOF
$ req
pushing to http://localhost:$HGPORT/
searching for changes
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 1 changesets with 1 changes to 1 files
% serve errors
$ hg rollback
repository tip rolled back to revision 0 (undo serve)
expect authorization error: all users denied
$ echo '[web]' > .hg/hgrc
$ echo 'push_ssl = false' >> .hg/hgrc
$ echo 'deny_push = *' >> .hg/hgrc
$ req
pushing to http://localhost:$HGPORT/
searching for changes
abort: authorization failed
% serve errors
[255]
expect authorization error: some users denied, users must be authenticated
$ echo 'deny_push = unperson' >> .hg/hgrc
$ req
pushing to http://localhost:$HGPORT/
searching for changes
abort: authorization failed
% serve errors
[255]
$ cat > .hg/hgrc <<EOF
> [web]
> push_ssl = false
> allow_push = *
> [experimental]
> httppostargs=true
> EOF
$ req
pushing to http://localhost:$HGPORT/
searching for changes
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 1 changesets with 1 changes to 1 files
% serve errors
$ cd ..