view tests/test-rollback.t @ 30442:41a8106789ca

util: implement zstd compression engine Now that zstd is vendored and being built (in some configurations), we can implement a compression engine for zstd! The zstd engine is a little different from existing engines. Because it may not always be present, we have to defer load the module in case importing it fails. We facilitate this via a cached property that holds a reference to the module or None. The "available" method is implemented to reflect reality. The zstd engine declares its ability to handle bundles using the "zstd" human name and the "ZS" internal name. The latter was chosen because internal names are 2 characters (by only convention I think) and "ZS" seems reasonable. The engine, like others, supports specifying the compression level. However, there are no consumers of this API that yet pass in that argument. I have plans to change that, so stay tuned. Since all we need to do to support bundle generation with a new compression engine is implement and register the compression engine, bundle generation with zstd "just works!" Tests demonstrating this have been added. How does performance of zstd for bundle generation compare? On the mozilla-unified repo, `hg bundle --all -t <engine>-v2` yields the following on my i7-6700K on Linux: engine CPU time bundle size vs orig size throughput none 97.0s 4,054,405,584 100.0% 41.8 MB/s bzip2 (l=9) 393.6s 975,343,098 24.0% 10.3 MB/s gzip (l=6) 184.0s 1,140,533,074 28.1% 22.0 MB/s zstd (l=1) 108.2s 1,119,434,718 27.6% 37.5 MB/s zstd (l=2) 111.3s 1,078,328,002 26.6% 36.4 MB/s zstd (l=3) 113.7s 1,011,823,727 25.0% 35.7 MB/s zstd (l=4) 116.0s 1,008,965,888 24.9% 35.0 MB/s zstd (l=5) 121.0s 977,203,148 24.1% 33.5 MB/s zstd (l=6) 131.7s 927,360,198 22.9% 30.8 MB/s zstd (l=7) 139.0s 912,808,505 22.5% 29.2 MB/s zstd (l=12) 198.1s 854,527,714 21.1% 20.5 MB/s zstd (l=18) 681.6s 789,750,690 19.5% 5.9 MB/s On compression, zstd for bundle generation delivers: * better compression than gzip with significantly less CPU utilization * better than bzip2 compression ratios while still being significantly faster than gzip * ability to aggressively tune compression level to achieve significantly smaller bundles That last point is important. With clone bundles, a server can pre-generate a bundle file, upload it to a static file server, and redirect clients to transparently download it during clone. The server could choose to produce a zstd bundle with the highest compression settings possible. This would take a very long time - a magnitude longer than a typical zstd bundle generation - but the result would be hundreds of megabytes smaller! For the clone volume we do at Mozilla, this could translate to petabytes of bandwidth savings per year and faster clones (due to smaller transfer size). I don't have detailed numbers to report on decompression. However, zstd decompression is fast: >1 GB/s output throughput on this machine, even through the Python bindings. And it can do that regardless of the compression level of the input. By the time you have enough data to worry about overhead of decompression, you have plenty of other things to worry about performance wise. zstd is wins all around. I can't wait to implement support for it on the wire protocol and in revlogs.
author Gregory Szorc <gregory.szorc@gmail.com>
date Fri, 11 Nov 2016 01:10:07 -0800
parents fe50341de1ff
children 3ed26ba54685
line wrap: on
line source

setup repo
  $ hg init t
  $ cd t
  $ echo a > a
  $ hg commit -Am'add a'
  adding a
  $ hg verify
  checking changesets
  checking manifests
  crosschecking files in changesets and manifests
  checking files
  1 files, 1 changesets, 1 total revisions
  $ hg parents
  changeset:   0:1f0dee641bb7
  tag:         tip
  user:        test
  date:        Thu Jan 01 00:00:00 1970 +0000
  summary:     add a
  

rollback to null revision
  $ hg status
  $ hg rollback
  repository tip rolled back to revision -1 (undo commit)
  working directory now based on revision -1
  $ hg verify
  checking changesets
  checking manifests
  crosschecking files in changesets and manifests
  checking files
  0 files, 0 changesets, 0 total revisions
  $ hg parents
  $ hg status
  A a

Two changesets this time so we rollback to a real changeset
  $ hg commit -m'add a again'
  $ echo a >> a
  $ hg commit -m'modify a'

Test issue 902 (current branch is preserved)
  $ hg branch test
  marked working directory as branch test
  (branches are permanent and global, did you want a bookmark?)
  $ hg rollback
  repository tip rolled back to revision 0 (undo commit)
  working directory now based on revision 0
  $ hg branch
  default

Test issue 1635 (commit message saved)
  $ cat .hg/last-message.txt ; echo
  modify a

Test rollback of hg before issue 902 was fixed

  $ hg commit -m "test3"
  $ hg branch test
  marked working directory as branch test
  (branches are permanent and global, did you want a bookmark?)
  $ rm .hg/undo.branch
  $ hg rollback
  repository tip rolled back to revision 0 (undo commit)
  named branch could not be reset: current branch is still 'test'
  working directory now based on revision 0
  $ hg branch
  test

working dir unaffected by rollback: do not restore dirstate et. al.
  $ hg log --template '{rev}  {branch}  {desc|firstline}\n'
  0  default  add a again
  $ hg status
  M a
  $ hg bookmark foo
  $ hg commit -m'modify a again'
  $ echo b > b
  $ hg bookmark bar -r default #making bar active, before the transaction
  $ hg commit -Am'add b'
  adding b
  $ hg log --template '{rev}  {branch}  {desc|firstline}\n'
  2  test  add b
  1  test  modify a again
  0  default  add a again
  $ hg update bar
  1 files updated, 0 files merged, 1 files removed, 0 files unresolved
  (activating bookmark bar)
  $ cat .hg/undo.branch ; echo
  test
  $ hg rollback -f
  repository tip rolled back to revision 1 (undo commit)
  $ hg id -n
  0
  $ hg branch
  default
  $ cat .hg/bookmarks.current ; echo
  bar
  $ hg bookmark --delete foo bar

rollback by pretxncommit saves commit message (issue1635)

  $ echo a >> a
  $ hg --config hooks.pretxncommit=false commit -m"precious commit message"
  transaction abort!
  rollback completed
  abort: pretxncommit hook exited with status * (glob)
  [255]
  $ cat .hg/last-message.txt ; echo
  precious commit message

same thing, but run $EDITOR

  $ cat > editor.sh << '__EOF__'
  > echo "another precious commit message" > "$1"
  > __EOF__
  $ HGEDITOR="\"sh\" \"`pwd`/editor.sh\"" hg --config hooks.pretxncommit=false commit 2>&1
  note: commit message saved in .hg/last-message.txt
  transaction abort!
  rollback completed
  abort: pretxncommit hook exited with status * (glob)
  [255]
  $ cat .hg/last-message.txt
  another precious commit message

test rollback on served repository

#if serve
  $ hg commit -m "precious commit message"
  $ hg serve -p $HGPORT -d --pid-file=hg.pid -A access.log -E errors.log
  $ cat hg.pid >> $DAEMON_PIDS
  $ cd ..
  $ hg clone http://localhost:$HGPORT u
  requesting all changes
  adding changesets
  adding manifests
  adding file changes
  added 3 changesets with 2 changes to 1 files (+1 heads)
  updating to branch default
  1 files updated, 0 files merged, 0 files removed, 0 files unresolved
  $ cd u
  $ hg id default
  068774709090

now rollback and observe that 'hg serve' reloads the repository and
presents the correct tip changeset:

  $ hg -R ../t rollback
  repository tip rolled back to revision 1 (undo commit)
  working directory now based on revision 0
  $ hg id default
  791dd2169706
#endif

update to older changeset and then refuse rollback, because
that would lose data (issue2998)
  $ cd ../t
  $ hg -q update
  $ rm `hg status -un`
  $ template='{rev}:{node|short}  [{branch}]  {desc|firstline}\n'
  $ echo 'valuable new file' > b
  $ echo 'valuable modification' >> a
  $ hg commit -A -m'a valuable change'
  adding b
  $ hg update 0
  1 files updated, 0 files merged, 1 files removed, 0 files unresolved
  $ hg rollback
  abort: rollback of last commit while not checked out may lose data
  (use -f to force)
  [255]
  $ hg tip -q
  2:4d9cd3795eea
  $ hg rollback -f
  repository tip rolled back to revision 1 (undo commit)
  $ hg status
  $ hg log --removed b   # yep, it's gone

same again, but emulate an old client that doesn't write undo.desc
  $ hg -q update
  $ echo 'valuable modification redux' >> a
  $ hg commit -m'a valuable change redux'
  $ rm .hg/undo.desc
  $ hg update 0
  1 files updated, 0 files merged, 0 files removed, 0 files unresolved
  $ hg rollback
  rolling back unknown transaction
  $ cat a
  a

corrupt journal test
  $ echo "foo" > .hg/store/journal
  $ hg recover
  rolling back interrupted transaction
  couldn't read journal entry 'foo\n'!
  checking changesets
  checking manifests
  crosschecking files in changesets and manifests
  checking files
  1 files, 2 changesets, 2 total revisions

rollback disabled by config
  $ cat >> $HGRCPATH <<EOF
  > [ui]
  > rollback = false
  > EOF
  $ echo narf >> pinky-sayings.txt
  $ hg add pinky-sayings.txt
  $ hg ci -m 'First one.'
  $ hg rollback
  abort: rollback is disabled because it is unsafe
  (see `hg help -v rollback` for information)
  [255]