changelog: disable delta chains
This patch disables delta chains on changelogs. After this patch, new
entries on changelogs - including existing changelogs - will be stored
as the fulltext of that data (likely compressed). No delta computation
will be performed.
An overview of delta chains and data justifying this change follows.
Revlogs try to store entries as a delta against a previous entry (either
a parent revision in the case of generaldelta or the previous physical
revision when not using generaldelta). Most of the time this is the
correct thing to do: it frequently results in less CPU usage and smaller
storage.
Delta chains are most effective when the base revision being deltad
against is similar to the current data. This tends to occur naturally
for manifests and file data, since only small parts of each tend to
change with each revision. Changelogs, however, are a different story.
Changelog entries represent changesets/commits. And unless commits in a
repository are homogonous (same author, changing same files, similar
commit messages, etc), a delta from one entry to the next tends to be
relatively large compared to the size of the entry. This means that
delta chains tend to be short. How short? Here is the full vs delta
revision breakdown on some real world repos:
Repo % Full % Delta Max Length
hg 45.8 54.2 6
mozilla-central 42.4 57.6 8
mozilla-unified 42.5 57.5 17
pypy 46.1 53.9 6
python-zstandard 46.1 53.9 3
(I threw in python-zstandard as an example of a repo that is homogonous.
It contains a small Python project with changes all from the same
author.)
Contrast this with the manifest revlog for these repos, where 99+% of
revisions are deltas and delta chains run into the thousands.
So delta chains aren't as useful on changelogs. But even a short delta
chain may provide benefits. Let's measure that.
Delta chains may require less CPU to read revisions if the CPU time
spent reading smaller deltas is less than the CPU time used to
decompress larger individual entries. We can measure this via
`hg perfrevlog -c -d 1` to iterate a revlog to resolve each revision's
fulltext. Here are the results of that command on a repo using delta
chains in its changelog and on a repo without delta chains:
hg (forward)
! wall 0.407008 comb 0.410000 user 0.410000 sys 0.000000 (best of 25)
! wall 0.390061 comb 0.390000 user 0.390000 sys 0.000000 (best of 26)
hg (reverse)
! wall 0.515221 comb 0.520000 user 0.520000 sys 0.000000 (best of 19)
! wall 0.400018 comb 0.400000 user 0.390000 sys 0.010000 (best of 25)
mozilla-central (forward)
! wall 4.508296 comb 4.490000 user 4.490000 sys 0.000000 (best of 3)
! wall 4.370222 comb 4.370000 user 4.350000 sys 0.020000 (best of 3)
mozilla-central (reverse)
! wall 5.758995 comb 5.760000 user 5.720000 sys 0.040000 (best of 3)
! wall 4.346503 comb 4.340000 user 4.320000 sys 0.020000 (best of 3)
mozilla-unified (forward)
! wall 4.957088 comb 4.950000 user 4.940000 sys 0.010000 (best of 3)
! wall 4.660528 comb 4.650000 user 4.630000 sys 0.020000 (best of 3)
mozilla-unified (reverse)
! wall 6.119827 comb 6.110000 user 6.090000 sys 0.020000 (best of 3)
! wall 4.675136 comb 4.670000 user 4.670000 sys 0.000000 (best of 3)
pypy (forward)
! wall 1.231122 comb 1.240000 user 1.230000 sys 0.010000 (best of 8)
! wall 1.164896 comb 1.160000 user 1.160000 sys 0.000000 (best of 9)
pypy (reverse)
! wall 1.467049 comb 1.460000 user 1.460000 sys 0.000000 (best of 7)
! wall 1.160200 comb 1.170000 user 1.160000 sys 0.010000 (best of 9)
The data clearly shows that it takes less wall and CPU time to resolve
revisions when there are no delta chains in the changelogs, regardless
of the direction of traversal. Furthermore, not using a delta chain
means that fulltext resolution in reverse is as fast as iterating
forward. So not using delta chains on the changelog is a clear CPU win
for reading operations.
An example of a user-visible operation showing this speed-up is revset
evaluation. Here are results for
`hg perfrevset 'author(gps) or author(mpm)'`:
hg
! wall 1.655506 comb 1.660000 user 1.650000 sys 0.010000 (best of 6)
! wall 1.612723 comb 1.610000 user 1.600000 sys 0.010000 (best of 7)
mozilla-central
! wall 17.629826 comb 17.640000 user 17.600000 sys 0.040000 (best of 3)
! wall 17.311033 comb 17.300000 user 17.260000 sys 0.040000 (best of 3)
What about 00changelog.i size?
Repo Delta Chains No Delta Chains
hg 7,033,250 6,976,771
mozilla-central 82,978,748 81,574,623
mozilla-unified 88,112,349 86,702,162
pypy 20,740,699 20,659,741
The data shows that removing delta chains from the changelog makes the
changelog smaller.
Delta chains are also used during changegroup generation. This
operation essentially converts a series of revisions to one large
delta chain. And changegroup generation is smart: if the delta in
the revlog matches what the changegroup is emitting, it will reuse
the delta instead of recalculating it. We can measure the impact
removing changelog delta chains has on changegroup generation via
`hg perfchangegroupchangelog`:
hg
! wall 1.589245 comb 1.590000 user 1.590000 sys 0.000000 (best of 7)
! wall 1.788060 comb 1.790000 user 1.790000 sys 0.000000 (best of 6)
mozilla-central
! wall 17.382585 comb 17.380000 user 17.340000 sys 0.040000 (best of 3)
! wall 20.161357 comb 20.160000 user 20.120000 sys 0.040000 (best of 3)
mozilla-unified
! wall 18.722839 comb 18.720000 user 18.680000 sys 0.040000 (best of 3)
! wall 21.168075 comb 21.170000 user 21.130000 sys 0.040000 (best of 3)
pypy
! wall 4.828317 comb 4.830000 user 4.820000 sys 0.010000 (best of 3)
! wall 5.415455 comb 5.420000 user 5.410000 sys 0.010000 (best of 3)
The data shows eliminating delta chains makes the changelog part of
changegroup generation slower. This is expected since we now have to
compute deltas for revisions where we could recycle the delta before.
It is worth putting this regression into context of overall changegroup
times. Here is the rough total CPU time spent in changegroup generation
for various repos while using delta chains on the changelog:
Repo CPU Time (s) CPU Time w/ compression
hg 4.50 7.05
mozilla-central 111.1 222.0
pypy 28.68 75.5
Before compression, removing delta chains from the changegroup adds
~4.4% overhead to hg changegroup generation, 1.3% to mozilla-central,
and 2.0% to pypy. When you factor in zlib compression, these percentages
are roughly divided by 2.
While the increased CPU usage for changegroup generation is unfortunate,
I think it is acceptable because the percentage is small, server
operators (those likely impacted most by this) have other mechanisms
to mitigate CPU consumption (namely reducing zlib compression level and
pre-generated clone bundles), and because there is room to optimize this
in the future. For example, we could use the nullid as the base revision,
effectively encoding the full revision for each entry in the changegroup.
When doing this, `hg perfchangegroupchangelog` nearly halves:
mozilla-unified
! wall 21.168075 comb 21.170000 user 21.130000 sys 0.040000 (best of 3)
! wall 11.196461 comb 11.200000 user 11.190000 sys 0.010000 (best of 3)
This looks very promising as a future optimization opportunity.
It's worth that the changes in test-acl.t to the changegroup part size.
This is because revision 6 in the changegroup had a delta chain of
length 2 before and after this patch the base revision is nullrev.
When the base revision is nullrev, cg2packer.deltaparent() hardcodes
the *previous* revision from the changegroup as the delta parent.
This caused the delta in the changegroup to switch base revisions,
the delta to change, and the size to change accordingly. While the
size increased in this case, I think sizes will remain the same
on average, as the delta base for changelog revisions doesn't matter
too much (as this patch shows). So, I don't consider this a regression.
commit hooks can see env vars
(and post-transaction one are run unlocked)
$ cat > $TESTTMP/txnabort.checkargs.py <<EOF
> def showargs(ui, repo, hooktype, **kwargs):
> ui.write('%s python hook: %s\n' % (hooktype, ','.join(sorted(kwargs))))
> EOF
$ hg init a
$ cd a
$ cat > .hg/hgrc <<EOF
> [hooks]
> commit = sh -c "HG_LOCAL= HG_TAG= printenv.py commit"
> commit.b = sh -c "HG_LOCAL= HG_TAG= printenv.py commit.b"
> precommit = sh -c "HG_LOCAL= HG_NODE= HG_TAG= printenv.py precommit"
> pretxncommit = sh -c "HG_LOCAL= HG_TAG= printenv.py pretxncommit"
> pretxncommit.tip = hg -q tip
> pre-identify = printenv.py pre-identify 1
> pre-cat = printenv.py pre-cat
> post-cat = printenv.py post-cat
> pretxnopen = sh -c "HG_LOCAL= HG_TAG= printenv.py pretxnopen"
> pretxnclose = sh -c "HG_LOCAL= HG_TAG= printenv.py pretxnclose"
> txnclose = sh -c "HG_LOCAL= HG_TAG= printenv.py txnclose"
> txnabort.0 = python:$TESTTMP/txnabort.checkargs.py:showargs
> txnabort.1 = sh -c "HG_LOCAL= HG_TAG= printenv.py txnabort"
> txnclose.checklock = sh -c "hg debuglock > /dev/null"
> EOF
$ echo a > a
$ hg add a
$ hg commit -m a
precommit hook: HG_PARENT1=0000000000000000000000000000000000000000
pretxnopen hook: HG_TXNID=TXN:* HG_TXNNAME=commit (glob)
pretxncommit hook: HG_NODE=cb9a9f314b8b07ba71012fcdbc544b5a4d82ff5b HG_PARENT1=0000000000000000000000000000000000000000 HG_PENDING=$TESTTMP/a
0:cb9a9f314b8b
pretxnclose hook: HG_PENDING=$TESTTMP/a HG_PHASES_MOVED=1 HG_TXNID=TXN:* HG_TXNNAME=commit (glob)
txnclose hook: HG_PHASES_MOVED=1 HG_TXNID=TXN:* HG_TXNNAME=commit (glob)
commit hook: HG_NODE=cb9a9f314b8b07ba71012fcdbc544b5a4d82ff5b HG_PARENT1=0000000000000000000000000000000000000000
commit.b hook: HG_NODE=cb9a9f314b8b07ba71012fcdbc544b5a4d82ff5b HG_PARENT1=0000000000000000000000000000000000000000
$ hg clone . ../b
updating to branch default
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ cd ../b
changegroup hooks can see env vars
$ cat > .hg/hgrc <<EOF
> [hooks]
> prechangegroup = printenv.py prechangegroup
> changegroup = printenv.py changegroup
> incoming = printenv.py incoming
> EOF
pretxncommit and commit hooks can see both parents of merge
$ cd ../a
$ echo b >> a
$ hg commit -m a1 -d "1 0"
precommit hook: HG_PARENT1=cb9a9f314b8b07ba71012fcdbc544b5a4d82ff5b
pretxnopen hook: HG_TXNID=TXN:* HG_TXNNAME=commit (glob)
pretxncommit hook: HG_NODE=ab228980c14deea8b9555d91c9581127383e40fd HG_PARENT1=cb9a9f314b8b07ba71012fcdbc544b5a4d82ff5b HG_PENDING=$TESTTMP/a
1:ab228980c14d
pretxnclose hook: HG_PENDING=$TESTTMP/a HG_TXNID=TXN:* HG_TXNNAME=commit (glob)
txnclose hook: HG_TXNID=TXN:* HG_TXNNAME=commit (glob)
commit hook: HG_NODE=ab228980c14deea8b9555d91c9581127383e40fd HG_PARENT1=cb9a9f314b8b07ba71012fcdbc544b5a4d82ff5b
commit.b hook: HG_NODE=ab228980c14deea8b9555d91c9581127383e40fd HG_PARENT1=cb9a9f314b8b07ba71012fcdbc544b5a4d82ff5b
$ hg update -C 0
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ echo b > b
$ hg add b
$ hg commit -m b -d '1 0'
precommit hook: HG_PARENT1=cb9a9f314b8b07ba71012fcdbc544b5a4d82ff5b
pretxnopen hook: HG_TXNID=TXN:* HG_TXNNAME=commit (glob)
pretxncommit hook: HG_NODE=ee9deb46ab31e4cc3310f3cf0c3d668e4d8fffc2 HG_PARENT1=cb9a9f314b8b07ba71012fcdbc544b5a4d82ff5b HG_PENDING=$TESTTMP/a
2:ee9deb46ab31
pretxnclose hook: HG_PENDING=$TESTTMP/a HG_TXNID=TXN:* HG_TXNNAME=commit (glob)
created new head
txnclose hook: HG_TXNID=TXN:* HG_TXNNAME=commit (glob)
commit hook: HG_NODE=ee9deb46ab31e4cc3310f3cf0c3d668e4d8fffc2 HG_PARENT1=cb9a9f314b8b07ba71012fcdbc544b5a4d82ff5b
commit.b hook: HG_NODE=ee9deb46ab31e4cc3310f3cf0c3d668e4d8fffc2 HG_PARENT1=cb9a9f314b8b07ba71012fcdbc544b5a4d82ff5b
$ hg merge 1
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
(branch merge, don't forget to commit)
$ hg commit -m merge -d '2 0'
precommit hook: HG_PARENT1=ee9deb46ab31e4cc3310f3cf0c3d668e4d8fffc2 HG_PARENT2=ab228980c14deea8b9555d91c9581127383e40fd
pretxnopen hook: HG_TXNID=TXN:* HG_TXNNAME=commit (glob)
pretxncommit hook: HG_NODE=07f3376c1e655977439df2a814e3cc14b27abac2 HG_PARENT1=ee9deb46ab31e4cc3310f3cf0c3d668e4d8fffc2 HG_PARENT2=ab228980c14deea8b9555d91c9581127383e40fd HG_PENDING=$TESTTMP/a
3:07f3376c1e65
pretxnclose hook: HG_PENDING=$TESTTMP/a HG_TXNID=TXN:* HG_TXNNAME=commit (glob)
txnclose hook: HG_TXNID=TXN:* HG_TXNNAME=commit (glob)
commit hook: HG_NODE=07f3376c1e655977439df2a814e3cc14b27abac2 HG_PARENT1=ee9deb46ab31e4cc3310f3cf0c3d668e4d8fffc2 HG_PARENT2=ab228980c14deea8b9555d91c9581127383e40fd
commit.b hook: HG_NODE=07f3376c1e655977439df2a814e3cc14b27abac2 HG_PARENT1=ee9deb46ab31e4cc3310f3cf0c3d668e4d8fffc2 HG_PARENT2=ab228980c14deea8b9555d91c9581127383e40fd
test generic hooks
$ hg id
pre-identify hook: HG_ARGS=id HG_OPTS={'bookmarks': None, 'branch': None, 'id': None, 'insecure': None, 'num': None, 'remotecmd': '', 'rev': '', 'ssh': '', 'tags': None} HG_PATS=[]
abort: pre-identify hook exited with status 1
[255]
$ hg cat b
pre-cat hook: HG_ARGS=cat b HG_OPTS={'decode': None, 'exclude': [], 'include': [], 'output': '', 'rev': ''} HG_PATS=['b']
b
post-cat hook: HG_ARGS=cat b HG_OPTS={'decode': None, 'exclude': [], 'include': [], 'output': '', 'rev': ''} HG_PATS=['b'] HG_RESULT=0
$ cd ../b
$ hg pull ../a
pulling from ../a
searching for changes
prechangegroup hook: HG_SOURCE=pull HG_TXNID=TXN:* HG_URL=file:$TESTTMP/a (glob)
adding changesets
adding manifests
adding file changes
added 3 changesets with 2 changes to 2 files
changegroup hook: HG_NODE=ab228980c14deea8b9555d91c9581127383e40fd HG_NODE_LAST=07f3376c1e655977439df2a814e3cc14b27abac2 HG_SOURCE=pull HG_TXNID=TXN:* HG_URL=file:$TESTTMP/a (glob)
incoming hook: HG_NODE=ab228980c14deea8b9555d91c9581127383e40fd HG_SOURCE=pull HG_TXNID=TXN:* HG_URL=file:$TESTTMP/a (glob)
incoming hook: HG_NODE=ee9deb46ab31e4cc3310f3cf0c3d668e4d8fffc2 HG_SOURCE=pull HG_TXNID=TXN:* HG_URL=file:$TESTTMP/a (glob)
incoming hook: HG_NODE=07f3376c1e655977439df2a814e3cc14b27abac2 HG_SOURCE=pull HG_TXNID=TXN:* HG_URL=file:$TESTTMP/a (glob)
(run 'hg update' to get a working copy)
tag hooks can see env vars
$ cd ../a
$ cat >> .hg/hgrc <<EOF
> pretag = printenv.py pretag
> tag = sh -c "HG_PARENT1= HG_PARENT2= printenv.py tag"
> EOF
$ hg tag -d '3 0' a
pretag hook: HG_LOCAL=0 HG_NODE=07f3376c1e655977439df2a814e3cc14b27abac2 HG_TAG=a
precommit hook: HG_PARENT1=07f3376c1e655977439df2a814e3cc14b27abac2
pretxnopen hook: HG_TXNID=TXN:* HG_TXNNAME=commit (glob)
pretxncommit hook: HG_NODE=539e4b31b6dc99b3cfbaa6b53cbc1c1f9a1e3a10 HG_PARENT1=07f3376c1e655977439df2a814e3cc14b27abac2 HG_PENDING=$TESTTMP/a
4:539e4b31b6dc
pretxnclose hook: HG_PENDING=$TESTTMP/a HG_TXNID=TXN:* HG_TXNNAME=commit (glob)
tag hook: HG_LOCAL=0 HG_NODE=07f3376c1e655977439df2a814e3cc14b27abac2 HG_TAG=a
txnclose hook: HG_TXNID=TXN:* HG_TXNNAME=commit (glob)
commit hook: HG_NODE=539e4b31b6dc99b3cfbaa6b53cbc1c1f9a1e3a10 HG_PARENT1=07f3376c1e655977439df2a814e3cc14b27abac2
commit.b hook: HG_NODE=539e4b31b6dc99b3cfbaa6b53cbc1c1f9a1e3a10 HG_PARENT1=07f3376c1e655977439df2a814e3cc14b27abac2
$ hg tag -l la
pretag hook: HG_LOCAL=1 HG_NODE=539e4b31b6dc99b3cfbaa6b53cbc1c1f9a1e3a10 HG_TAG=la
tag hook: HG_LOCAL=1 HG_NODE=539e4b31b6dc99b3cfbaa6b53cbc1c1f9a1e3a10 HG_TAG=la
pretag hook can forbid tagging
$ echo "pretag.forbid = printenv.py pretag.forbid 1" >> .hg/hgrc
$ hg tag -d '4 0' fa
pretag hook: HG_LOCAL=0 HG_NODE=539e4b31b6dc99b3cfbaa6b53cbc1c1f9a1e3a10 HG_TAG=fa
pretag.forbid hook: HG_LOCAL=0 HG_NODE=539e4b31b6dc99b3cfbaa6b53cbc1c1f9a1e3a10 HG_TAG=fa
abort: pretag.forbid hook exited with status 1
[255]
$ hg tag -l fla
pretag hook: HG_LOCAL=1 HG_NODE=539e4b31b6dc99b3cfbaa6b53cbc1c1f9a1e3a10 HG_TAG=fla
pretag.forbid hook: HG_LOCAL=1 HG_NODE=539e4b31b6dc99b3cfbaa6b53cbc1c1f9a1e3a10 HG_TAG=fla
abort: pretag.forbid hook exited with status 1
[255]
pretxncommit hook can see changeset, can roll back txn, changeset no
more there after
$ echo "pretxncommit.forbid0 = hg tip -q" >> .hg/hgrc
$ echo "pretxncommit.forbid1 = printenv.py pretxncommit.forbid 1" >> .hg/hgrc
$ echo z > z
$ hg add z
$ hg -q tip
4:539e4b31b6dc
$ hg commit -m 'fail' -d '4 0'
precommit hook: HG_PARENT1=539e4b31b6dc99b3cfbaa6b53cbc1c1f9a1e3a10
pretxnopen hook: HG_TXNID=TXN:* HG_TXNNAME=commit (glob)
pretxncommit hook: HG_NODE=6f611f8018c10e827fee6bd2bc807f937e761567 HG_PARENT1=539e4b31b6dc99b3cfbaa6b53cbc1c1f9a1e3a10 HG_PENDING=$TESTTMP/a
5:6f611f8018c1
5:6f611f8018c1
pretxncommit.forbid hook: HG_NODE=6f611f8018c10e827fee6bd2bc807f937e761567 HG_PARENT1=539e4b31b6dc99b3cfbaa6b53cbc1c1f9a1e3a10 HG_PENDING=$TESTTMP/a
transaction abort!
txnabort python hook: txnid,txnname
txnabort hook: HG_TXNID=TXN:* HG_TXNNAME=commit (glob)
rollback completed
abort: pretxncommit.forbid1 hook exited with status 1
[255]
$ hg -q tip
4:539e4b31b6dc
(Check that no 'changelog.i.a' file were left behind)
$ ls -1 .hg/store/
00changelog.i
00manifest.i
data
fncache
journal.phaseroots
phaseroots
undo
undo.backup.fncache
undo.backupfiles
undo.phaseroots
precommit hook can prevent commit
$ echo "precommit.forbid = printenv.py precommit.forbid 1" >> .hg/hgrc
$ hg commit -m 'fail' -d '4 0'
precommit hook: HG_PARENT1=539e4b31b6dc99b3cfbaa6b53cbc1c1f9a1e3a10
precommit.forbid hook: HG_PARENT1=539e4b31b6dc99b3cfbaa6b53cbc1c1f9a1e3a10
abort: precommit.forbid hook exited with status 1
[255]
$ hg -q tip
4:539e4b31b6dc
preupdate hook can prevent update
$ echo "preupdate = printenv.py preupdate" >> .hg/hgrc
$ hg update 1
preupdate hook: HG_PARENT1=ab228980c14d
0 files updated, 0 files merged, 2 files removed, 0 files unresolved
update hook
$ echo "update = printenv.py update" >> .hg/hgrc
$ hg update
preupdate hook: HG_PARENT1=539e4b31b6dc
update hook: HG_ERROR=0 HG_PARENT1=539e4b31b6dc
2 files updated, 0 files merged, 0 files removed, 0 files unresolved
pushkey hook
$ echo "pushkey = printenv.py pushkey" >> .hg/hgrc
$ cd ../b
$ hg bookmark -r null foo
$ hg push -B foo ../a
pushing to ../a
searching for changes
no changes found
pretxnopen hook: HG_TXNID=TXN:* HG_TXNNAME=push (glob)
pretxnclose hook: HG_BOOKMARK_MOVED=1 HG_BUNDLE2=1 HG_PENDING=$TESTTMP/a HG_SOURCE=push HG_TXNID=TXN:* HG_TXNNAME=push HG_URL=file:$TESTTMP/a (glob)
pushkey hook: HG_KEY=foo HG_NAMESPACE=bookmarks HG_NEW=0000000000000000000000000000000000000000 HG_RET=1
txnclose hook: HG_BOOKMARK_MOVED=1 HG_BUNDLE2=1 HG_SOURCE=push HG_TXNID=TXN:* HG_TXNNAME=push HG_URL=file:$TESTTMP/a (glob)
exporting bookmark foo
[1]
$ cd ../a
listkeys hook
$ echo "listkeys = printenv.py listkeys" >> .hg/hgrc
$ hg bookmark -r null bar
pretxnopen hook: HG_TXNID=TXN:* HG_TXNNAME=bookmark (glob)
pretxnclose hook: HG_BOOKMARK_MOVED=1 HG_PENDING=$TESTTMP/a HG_TXNID=TXN:* HG_TXNNAME=bookmark (glob)
txnclose hook: HG_BOOKMARK_MOVED=1 HG_TXNID=TXN:* HG_TXNNAME=bookmark (glob)
$ cd ../b
$ hg pull -B bar ../a
pulling from ../a
listkeys hook: HG_NAMESPACE=bookmarks HG_VALUES={'bar': '0000000000000000000000000000000000000000', 'foo': '0000000000000000000000000000000000000000'}
no changes found
listkeys hook: HG_NAMESPACE=phases HG_VALUES={'cb9a9f314b8b07ba71012fcdbc544b5a4d82ff5b': '1', 'publishing': 'True'}
adding remote bookmark bar
$ cd ../a
test that prepushkey can prevent incoming keys
$ echo "prepushkey = printenv.py prepushkey.forbid 1" >> .hg/hgrc
$ cd ../b
$ hg bookmark -r null baz
$ hg push -B baz ../a
pushing to ../a
searching for changes
listkeys hook: HG_NAMESPACE=phases HG_VALUES={'cb9a9f314b8b07ba71012fcdbc544b5a4d82ff5b': '1', 'publishing': 'True'}
listkeys hook: HG_NAMESPACE=bookmarks HG_VALUES={'bar': '0000000000000000000000000000000000000000', 'foo': '0000000000000000000000000000000000000000'}
no changes found
pretxnopen hook: HG_TXNID=TXN:* HG_TXNNAME=push (glob)
prepushkey.forbid hook: HG_BUNDLE2=1 HG_KEY=baz HG_NAMESPACE=bookmarks HG_NEW=0000000000000000000000000000000000000000 HG_SOURCE=push HG_TXNID=TXN:* HG_URL=file:$TESTTMP/a (glob)
pushkey-abort: prepushkey hook exited with status 1
abort: exporting bookmark baz failed!
[255]
$ cd ../a
test that prelistkeys can prevent listing keys
$ echo "prelistkeys = printenv.py prelistkeys.forbid 1" >> .hg/hgrc
$ hg bookmark -r null quux
pretxnopen hook: HG_TXNID=TXN:* HG_TXNNAME=bookmark (glob)
pretxnclose hook: HG_BOOKMARK_MOVED=1 HG_PENDING=$TESTTMP/a HG_TXNID=TXN:* HG_TXNNAME=bookmark (glob)
txnclose hook: HG_BOOKMARK_MOVED=1 HG_TXNID=TXN:* HG_TXNNAME=bookmark (glob)
$ cd ../b
$ hg pull -B quux ../a
pulling from ../a
prelistkeys.forbid hook: HG_NAMESPACE=bookmarks
abort: prelistkeys hook exited with status 1
[255]
$ cd ../a
$ rm .hg/hgrc
prechangegroup hook can prevent incoming changes
$ cd ../b
$ hg -q tip
3:07f3376c1e65
$ cat > .hg/hgrc <<EOF
> [hooks]
> prechangegroup.forbid = printenv.py prechangegroup.forbid 1
> EOF
$ hg pull ../a
pulling from ../a
searching for changes
prechangegroup.forbid hook: HG_SOURCE=pull HG_TXNID=TXN:* HG_URL=file:$TESTTMP/a (glob)
abort: prechangegroup.forbid hook exited with status 1
[255]
pretxnchangegroup hook can see incoming changes, can roll back txn,
incoming changes no longer there after
$ cat > .hg/hgrc <<EOF
> [hooks]
> pretxnchangegroup.forbid0 = hg tip -q
> pretxnchangegroup.forbid1 = printenv.py pretxnchangegroup.forbid 1
> EOF
$ hg pull ../a
pulling from ../a
searching for changes
adding changesets
adding manifests
adding file changes
added 1 changesets with 1 changes to 1 files
4:539e4b31b6dc
pretxnchangegroup.forbid hook: HG_NODE=539e4b31b6dc99b3cfbaa6b53cbc1c1f9a1e3a10 HG_NODE_LAST=539e4b31b6dc99b3cfbaa6b53cbc1c1f9a1e3a10 HG_PENDING=$TESTTMP/b HG_SOURCE=pull HG_TXNID=TXN:* HG_URL=file:$TESTTMP/a (glob)
transaction abort!
rollback completed
abort: pretxnchangegroup.forbid1 hook exited with status 1
[255]
$ hg -q tip
3:07f3376c1e65
outgoing hooks can see env vars
$ rm .hg/hgrc
$ cat > ../a/.hg/hgrc <<EOF
> [hooks]
> preoutgoing = printenv.py preoutgoing
> outgoing = printenv.py outgoing
> EOF
$ hg pull ../a
pulling from ../a
searching for changes
preoutgoing hook: HG_SOURCE=pull
outgoing hook: HG_NODE=539e4b31b6dc99b3cfbaa6b53cbc1c1f9a1e3a10 HG_SOURCE=pull
adding changesets
adding manifests
adding file changes
added 1 changesets with 1 changes to 1 files
adding remote bookmark quux
(run 'hg update' to get a working copy)
$ hg rollback
repository tip rolled back to revision 3 (undo pull)
preoutgoing hook can prevent outgoing changes
$ echo "preoutgoing.forbid = printenv.py preoutgoing.forbid 1" >> ../a/.hg/hgrc
$ hg pull ../a
pulling from ../a
searching for changes
preoutgoing hook: HG_SOURCE=pull
preoutgoing.forbid hook: HG_SOURCE=pull
abort: preoutgoing.forbid hook exited with status 1
[255]
outgoing hooks work for local clones
$ cd ..
$ cat > a/.hg/hgrc <<EOF
> [hooks]
> preoutgoing = printenv.py preoutgoing
> outgoing = printenv.py outgoing
> EOF
$ hg clone a c
preoutgoing hook: HG_SOURCE=clone
outgoing hook: HG_NODE=0000000000000000000000000000000000000000 HG_SOURCE=clone
updating to branch default
3 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ rm -rf c
preoutgoing hook can prevent outgoing changes for local clones
$ echo "preoutgoing.forbid = printenv.py preoutgoing.forbid 1" >> a/.hg/hgrc
$ hg clone a zzz
preoutgoing hook: HG_SOURCE=clone
preoutgoing.forbid hook: HG_SOURCE=clone
abort: preoutgoing.forbid hook exited with status 1
[255]
$ cd "$TESTTMP/b"
$ cat > hooktests.py <<EOF
> from mercurial import error
>
> uncallable = 0
>
> def printargs(args):
> args.pop('ui', None)
> args.pop('repo', None)
> a = list(args.items())
> a.sort()
> print 'hook args:'
> for k, v in a:
> print ' ', k, v
>
> def passhook(**args):
> printargs(args)
>
> def failhook(**args):
> printargs(args)
> return True
>
> class LocalException(Exception):
> pass
>
> def raisehook(**args):
> raise LocalException('exception from hook')
>
> def aborthook(**args):
> raise error.Abort('raise abort from hook')
>
> def brokenhook(**args):
> return 1 + {}
>
> def verbosehook(ui, **args):
> ui.note('verbose output from hook\n')
>
> def printtags(ui, repo, **args):
> print sorted(repo.tags())
>
> class container:
> unreachable = 1
> EOF
$ cat > syntaxerror.py << EOF
> (foo
> EOF
test python hooks
#if windows
$ PYTHONPATH="$TESTTMP/b;$PYTHONPATH"
#else
$ PYTHONPATH="$TESTTMP/b:$PYTHONPATH"
#endif
$ export PYTHONPATH
$ echo '[hooks]' > ../a/.hg/hgrc
$ echo 'preoutgoing.broken = python:hooktests.brokenhook' >> ../a/.hg/hgrc
$ hg pull ../a 2>&1 | grep 'raised an exception'
error: preoutgoing.broken hook raised an exception: unsupported operand type(s) for +: 'int' and 'dict'
$ echo '[hooks]' > ../a/.hg/hgrc
$ echo 'preoutgoing.raise = python:hooktests.raisehook' >> ../a/.hg/hgrc
$ hg pull ../a 2>&1 | grep 'raised an exception'
error: preoutgoing.raise hook raised an exception: exception from hook
$ echo '[hooks]' > ../a/.hg/hgrc
$ echo 'preoutgoing.abort = python:hooktests.aborthook' >> ../a/.hg/hgrc
$ hg pull ../a
pulling from ../a
searching for changes
error: preoutgoing.abort hook failed: raise abort from hook
abort: raise abort from hook
[255]
$ echo '[hooks]' > ../a/.hg/hgrc
$ echo 'preoutgoing.fail = python:hooktests.failhook' >> ../a/.hg/hgrc
$ hg pull ../a
pulling from ../a
searching for changes
hook args:
hooktype preoutgoing
source pull
abort: preoutgoing.fail hook failed
[255]
$ echo '[hooks]' > ../a/.hg/hgrc
$ echo 'preoutgoing.uncallable = python:hooktests.uncallable' >> ../a/.hg/hgrc
$ hg pull ../a
pulling from ../a
searching for changes
abort: preoutgoing.uncallable hook is invalid: "hooktests.uncallable" is not callable
[255]
$ echo '[hooks]' > ../a/.hg/hgrc
$ echo 'preoutgoing.nohook = python:hooktests.nohook' >> ../a/.hg/hgrc
$ hg pull ../a
pulling from ../a
searching for changes
abort: preoutgoing.nohook hook is invalid: "hooktests.nohook" is not defined
[255]
$ echo '[hooks]' > ../a/.hg/hgrc
$ echo 'preoutgoing.nomodule = python:nomodule' >> ../a/.hg/hgrc
$ hg pull ../a
pulling from ../a
searching for changes
abort: preoutgoing.nomodule hook is invalid: "nomodule" not in a module
[255]
$ echo '[hooks]' > ../a/.hg/hgrc
$ echo 'preoutgoing.badmodule = python:nomodule.nowhere' >> ../a/.hg/hgrc
$ hg pull ../a
pulling from ../a
searching for changes
abort: preoutgoing.badmodule hook is invalid: import of "nomodule" failed
(run with --traceback for stack trace)
[255]
$ echo '[hooks]' > ../a/.hg/hgrc
$ echo 'preoutgoing.unreachable = python:hooktests.container.unreachable' >> ../a/.hg/hgrc
$ hg pull ../a
pulling from ../a
searching for changes
abort: preoutgoing.unreachable hook is invalid: import of "hooktests.container" failed
(run with --traceback for stack trace)
[255]
$ echo '[hooks]' > ../a/.hg/hgrc
$ echo 'preoutgoing.syntaxerror = python:syntaxerror.syntaxerror' >> ../a/.hg/hgrc
$ hg pull ../a
pulling from ../a
searching for changes
abort: preoutgoing.syntaxerror hook is invalid: import of "syntaxerror" failed
(run with --traceback for stack trace)
[255]
The second egrep is to filter out lines like ' ^', which are slightly
different between Python 2.6 and Python 2.7.
$ hg pull ../a --traceback 2>&1 | egrep -v '^( +File| [_a-zA-Z*(])' | egrep -v '^( )+(\^)?$'
pulling from ../a
searching for changes
exception from first failed import attempt:
Traceback (most recent call last):
SyntaxError: * (glob)
exception from second failed import attempt:
Traceback (most recent call last):
ImportError: No module named hgext_syntaxerror
Traceback (most recent call last):
HookLoadError: preoutgoing.syntaxerror hook is invalid: import of "syntaxerror" failed
abort: preoutgoing.syntaxerror hook is invalid: import of "syntaxerror" failed
$ echo '[hooks]' > ../a/.hg/hgrc
$ echo 'preoutgoing.pass = python:hooktests.passhook' >> ../a/.hg/hgrc
$ hg pull ../a
pulling from ../a
searching for changes
hook args:
hooktype preoutgoing
source pull
adding changesets
adding manifests
adding file changes
added 1 changesets with 1 changes to 1 files
adding remote bookmark quux
(run 'hg update' to get a working copy)
post- python hooks that fail to *run* don't cause an abort
$ rm ../a/.hg/hgrc
$ echo '[hooks]' > .hg/hgrc
$ echo 'post-pull.broken = python:hooktests.brokenhook' >> .hg/hgrc
$ hg pull ../a
pulling from ../a
searching for changes
no changes found
error: post-pull.broken hook raised an exception: unsupported operand type(s) for +: 'int' and 'dict'
(run with --traceback for stack trace)
but post- python hooks that fail to *load* do
$ echo '[hooks]' > .hg/hgrc
$ echo 'post-pull.nomodule = python:nomodule' >> .hg/hgrc
$ hg pull ../a
pulling from ../a
searching for changes
no changes found
abort: post-pull.nomodule hook is invalid: "nomodule" not in a module
[255]
$ echo '[hooks]' > .hg/hgrc
$ echo 'post-pull.badmodule = python:nomodule.nowhere' >> .hg/hgrc
$ hg pull ../a
pulling from ../a
searching for changes
no changes found
abort: post-pull.badmodule hook is invalid: import of "nomodule" failed
(run with --traceback for stack trace)
[255]
$ echo '[hooks]' > .hg/hgrc
$ echo 'post-pull.nohook = python:hooktests.nohook' >> .hg/hgrc
$ hg pull ../a
pulling from ../a
searching for changes
no changes found
abort: post-pull.nohook hook is invalid: "hooktests.nohook" is not defined
[255]
make sure --traceback works
$ echo '[hooks]' > .hg/hgrc
$ echo 'commit.abort = python:hooktests.aborthook' >> .hg/hgrc
$ echo aa > a
$ hg --traceback commit -d '0 0' -ma 2>&1 | grep '^Traceback'
Traceback (most recent call last):
$ cd ..
$ hg init c
$ cd c
$ cat > hookext.py <<EOF
> def autohook(**args):
> print "Automatically installed hook"
>
> def reposetup(ui, repo):
> repo.ui.setconfig("hooks", "commit.auto", autohook)
> EOF
$ echo '[extensions]' >> .hg/hgrc
$ echo 'hookext = hookext.py' >> .hg/hgrc
$ touch foo
$ hg add foo
$ hg ci -d '0 0' -m 'add foo'
Automatically installed hook
$ echo >> foo
$ hg ci --debug -d '0 0' -m 'change foo'
committing files:
foo
committing manifest
committing changelog
committed changeset 1:52998019f6252a2b893452765fcb0a47351a5708
calling hook commit.auto: hgext_hookext.autohook
Automatically installed hook
$ hg showconfig hooks
hooks.commit.auto=<function autohook at *> (glob)
test python hook configured with python:[file]:[hook] syntax
$ cd ..
$ mkdir d
$ cd d
$ hg init repo
$ mkdir hooks
$ cd hooks
$ cat > testhooks.py <<EOF
> def testhook(**args):
> print 'hook works'
> EOF
$ echo '[hooks]' > ../repo/.hg/hgrc
$ echo "pre-commit.test = python:`pwd`/testhooks.py:testhook" >> ../repo/.hg/hgrc
$ cd ../repo
$ hg commit -d '0 0'
hook works
nothing changed
[1]
$ echo '[hooks]' > .hg/hgrc
$ echo "update.ne = python:`pwd`/nonexistent.py:testhook" >> .hg/hgrc
$ echo "pre-identify.npmd = python:`pwd`/:no_python_module_dir" >> .hg/hgrc
$ hg up null
loading update.ne hook failed:
abort: No such file or directory: $TESTTMP/d/repo/nonexistent.py
[255]
$ hg id
loading pre-identify.npmd hook failed:
abort: No module named repo!
[255]
$ cd ../../b
make sure --traceback works on hook import failure
$ cat > importfail.py <<EOF
> import somebogusmodule
> # dereference something in the module to force demandimport to load it
> somebogusmodule.whatever
> EOF
$ echo '[hooks]' > .hg/hgrc
$ echo 'precommit.importfail = python:importfail.whatever' >> .hg/hgrc
$ echo a >> a
$ hg --traceback commit -ma 2>&1 | egrep -v '^( +File| [a-zA-Z(])'
exception from first failed import attempt:
Traceback (most recent call last):
ImportError: No module named somebogusmodule
exception from second failed import attempt:
Traceback (most recent call last):
ImportError: No module named hgext_importfail
Traceback (most recent call last):
HookLoadError: precommit.importfail hook is invalid: import of "importfail" failed
abort: precommit.importfail hook is invalid: import of "importfail" failed
Issue1827: Hooks Update & Commit not completely post operation
commit and update hooks should run after command completion. The largefiles
use demonstrates a recursive wlock, showing the hook doesn't run until the
final release (and dirstate flush).
$ echo '[hooks]' > .hg/hgrc
$ echo 'commit = hg id' >> .hg/hgrc
$ echo 'update = hg id' >> .hg/hgrc
$ echo bb > a
$ hg ci -ma
223eafe2750c tip
$ hg up 0 --config extensions.largefiles=
cb9a9f314b8b
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
make sure --verbose (and --quiet/--debug etc.) are propagated to the local ui
that is passed to pre/post hooks
$ echo '[hooks]' > .hg/hgrc
$ echo 'pre-identify = python:hooktests.verbosehook' >> .hg/hgrc
$ hg id
cb9a9f314b8b
$ hg id --verbose
calling hook pre-identify: hooktests.verbosehook
verbose output from hook
cb9a9f314b8b
Ensure hooks can be prioritized
$ echo '[hooks]' > .hg/hgrc
$ echo 'pre-identify.a = python:hooktests.verbosehook' >> .hg/hgrc
$ echo 'pre-identify.b = python:hooktests.verbosehook' >> .hg/hgrc
$ echo 'priority.pre-identify.b = 1' >> .hg/hgrc
$ echo 'pre-identify.c = python:hooktests.verbosehook' >> .hg/hgrc
$ hg id --verbose
calling hook pre-identify.b: hooktests.verbosehook
verbose output from hook
calling hook pre-identify.a: hooktests.verbosehook
verbose output from hook
calling hook pre-identify.c: hooktests.verbosehook
verbose output from hook
cb9a9f314b8b
new tags must be visible in pretxncommit (issue3210)
$ echo 'pretxncommit.printtags = python:hooktests.printtags' >> .hg/hgrc
$ hg tag -f foo
['a', 'foo', 'tip']
post-init hooks must not crash (issue4983)
This also creates the `to` repo for the next test block.
$ cd ..
$ cat << EOF >> hgrc-with-post-init-hook
> [hooks]
> post-init = printenv.py post-init
> EOF
$ HGRCPATH=hgrc-with-post-init-hook hg init to
post-init hook: HG_ARGS=init to HG_OPTS={'insecure': None, 'remotecmd': '', 'ssh': ''} HG_PATS=['to'] HG_RESULT=0
new commits must be visible in pretxnchangegroup (issue3428)
$ echo '[hooks]' >> to/.hg/hgrc
$ echo 'prechangegroup = hg --traceback tip' >> to/.hg/hgrc
$ echo 'pretxnchangegroup = hg --traceback tip' >> to/.hg/hgrc
$ echo a >> to/a
$ hg --cwd to ci -Ama
adding a
$ hg clone to from
updating to branch default
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ echo aa >> from/a
$ hg --cwd from ci -mb
$ hg --cwd from push
pushing to $TESTTMP/to (glob)
searching for changes
changeset: 0:cb9a9f314b8b
tag: tip
user: test
date: Thu Jan 01 00:00:00 1970 +0000
summary: a
adding changesets
adding manifests
adding file changes
added 1 changesets with 1 changes to 1 files
changeset: 1:9836a07b9b9d
tag: tip
user: test
date: Thu Jan 01 00:00:00 1970 +0000
summary: b
pretxnclose hook failure should abort the transaction
$ hg init txnfailure
$ cd txnfailure
$ touch a && hg commit -Aqm a
$ cat >> .hg/hgrc <<EOF
> [hooks]
> pretxnclose.error = exit 1
> EOF
$ hg strip -r 0 --config extensions.strip=
0 files updated, 0 files merged, 1 files removed, 0 files unresolved
saved backup bundle to * (glob)
transaction abort!
rollback completed
strip failed, backup bundle stored in * (glob)
abort: pretxnclose.error hook exited with status 1
[255]
$ hg recover
no interrupted transaction available
[1]
$ cd ..
Hook from untrusted hgrc are reported as failure
================================================
$ cat << EOF > $TESTTMP/untrusted.py
> from mercurial import scmutil, util
> def uisetup(ui):
> class untrustedui(ui.__class__):
> def _trusted(self, fp, f):
> if util.normpath(fp.name).endswith('untrusted/.hg/hgrc'):
> return False
> return super(untrustedui, self)._trusted(fp, f)
> ui.__class__ = untrustedui
> EOF
$ cat << EOF >> $HGRCPATH
> [extensions]
> untrusted=$TESTTMP/untrusted.py
> EOF
$ hg init untrusted
$ cd untrusted
Non-blocking hook
-----------------
$ cat << EOF >> .hg/hgrc
> [hooks]
> txnclose.testing=echo txnclose hook called
> EOF
$ touch a && hg commit -Aqm a
warning: untrusted hook txnclose not executed
$ hg log
changeset: 0:3903775176ed
tag: tip
user: test
date: Thu Jan 01 00:00:00 1970 +0000
summary: a
Non-blocking hook
-----------------
$ cat << EOF >> .hg/hgrc
> [hooks]
> pretxnclose.testing=echo pre-txnclose hook called
> EOF
$ touch b && hg commit -Aqm a
transaction abort!
rollback completed
abort: untrusted hook pretxnclose not executed
(see 'hg help config.trusted')
[255]
$ hg log
changeset: 0:3903775176ed
tag: tip
user: test
date: Thu Jan 01 00:00:00 1970 +0000
summary: a