view tests/test-hook.t @ 30155:b7a966ce89ed

changelog: disable delta chains

This patch disables delta chains on changelogs. After this patch, new
entries on changelogs - including existing changelogs - will be stored
as the fulltext of that data (likely compressed). No delta computation
will be performed.

An overview of delta chains and data justifying this change follows.

Revlogs try to store entries as a delta against a previous entry (either
a parent revision in the case of generaldelta or the previous physical
revision when not using generaldelta). Most of the time this is the
correct thing to do: it frequently results in less CPU usage and smaller
storage.

Delta chains are most effective when the base revision being deltad
against is similar to the current data. This tends to occur naturally
for manifests and file data, since only small parts of each tend to
change with each revision. Changelogs, however, are a different story.

Changelog entries represent changesets/commits. And unless commits in a
repository are homogonous (same author, changing same files, similar
commit messages, etc), a delta from one entry to the next tends to be
relatively large compared to the size of the entry. This means that
delta chains tend to be short. How short? Here is the full vs delta
revision breakdown on some real world repos:

Repo             % Full    % Delta   Max Length
hg                45.8       54.2        6
mozilla-central   42.4       57.6        8
mozilla-unified   42.5       57.5       17
pypy              46.1       53.9        6
python-zstandard  46.1       53.9        3

(I threw in python-zstandard as an example of a repo that is homogonous.
It contains a small Python project with changes all from the same
author.)

Contrast this with the manifest revlog for these repos, where 99+% of
revisions are deltas and delta chains run into the thousands.

So delta chains aren't as useful on changelogs. But even a short delta
chain may provide benefits. Let's measure that.

Delta chains may require less CPU to read revisions if the CPU time
spent reading smaller deltas is less than the CPU time used to
decompress larger individual entries. We can measure this via
`hg perfrevlog -c -d 1` to iterate a revlog to resolve each revision's
fulltext. Here are the results of that command on a repo using delta
chains in its changelog and on a repo without delta chains:

hg (forward)
! wall 0.407008 comb 0.410000 user 0.410000 sys 0.000000 (best of 25)
! wall 0.390061 comb 0.390000 user 0.390000 sys 0.000000 (best of 26)

hg (reverse)
! wall 0.515221 comb 0.520000 user 0.520000 sys 0.000000 (best of 19)
! wall 0.400018 comb 0.400000 user 0.390000 sys 0.010000 (best of 25)

mozilla-central (forward)
! wall 4.508296 comb 4.490000 user 4.490000 sys 0.000000 (best of 3)
! wall 4.370222 comb 4.370000 user 4.350000 sys 0.020000 (best of 3)

mozilla-central (reverse)
! wall 5.758995 comb 5.760000 user 5.720000 sys 0.040000 (best of 3)
! wall 4.346503 comb 4.340000 user 4.320000 sys 0.020000 (best of 3)

mozilla-unified (forward)
! wall 4.957088 comb 4.950000 user 4.940000 sys 0.010000 (best of 3)
! wall 4.660528 comb 4.650000 user 4.630000 sys 0.020000 (best of 3)

mozilla-unified (reverse)
! wall 6.119827 comb 6.110000 user 6.090000 sys 0.020000 (best of 3)
! wall 4.675136 comb 4.670000 user 4.670000 sys 0.000000 (best of 3)

pypy (forward)
! wall 1.231122 comb 1.240000 user 1.230000 sys 0.010000 (best of 8)
! wall 1.164896 comb 1.160000 user 1.160000 sys 0.000000 (best of 9)

pypy (reverse)
! wall 1.467049 comb 1.460000 user 1.460000 sys 0.000000 (best of 7)
! wall 1.160200 comb 1.170000 user 1.160000 sys 0.010000 (best of 9)

The data clearly shows that it takes less wall and CPU time to resolve
revisions when there are no delta chains in the changelogs, regardless
of the direction of traversal. Furthermore, not using a delta chain
means that fulltext resolution in reverse is as fast as iterating
forward. So not using delta chains on the changelog is a clear CPU win
for reading operations.

An example of a user-visible operation showing this speed-up is revset
evaluation. Here are results for
`hg perfrevset 'author(gps) or author(mpm)'`:

hg
! wall 1.655506 comb 1.660000 user 1.650000 sys 0.010000 (best of 6)
! wall 1.612723 comb 1.610000 user 1.600000 sys 0.010000 (best of 7)

mozilla-central
! wall 17.629826 comb 17.640000 user 17.600000 sys 0.040000 (best of 3)
! wall 17.311033 comb 17.300000 user 17.260000 sys 0.040000 (best of 3)

What about 00changelog.i size?

Repo                Delta Chains     No Delta Chains
hg                    7,033,250         6,976,771
mozilla-central      82,978,748        81,574,623
mozilla-unified      88,112,349        86,702,162
pypy                 20,740,699        20,659,741

The data shows that removing delta chains from the changelog makes the
changelog smaller.

Delta chains are also used during changegroup generation. This
operation essentially converts a series of revisions to one large
delta chain. And changegroup generation is smart: if the delta in
the revlog matches what the changegroup is emitting, it will reuse
the delta instead of recalculating it. We can measure the impact
removing changelog delta chains has on changegroup generation via
`hg perfchangegroupchangelog`:

hg
! wall 1.589245 comb 1.590000 user 1.590000 sys 0.000000 (best of 7)
! wall 1.788060 comb 1.790000 user 1.790000 sys 0.000000 (best of 6)

mozilla-central
! wall 17.382585 comb 17.380000 user 17.340000 sys 0.040000 (best of 3)
! wall 20.161357 comb 20.160000 user 20.120000 sys 0.040000 (best of 3)

mozilla-unified
! wall 18.722839 comb 18.720000 user 18.680000 sys 0.040000 (best of 3)
! wall 21.168075 comb 21.170000 user 21.130000 sys 0.040000 (best of 3)

pypy
! wall 4.828317 comb 4.830000 user 4.820000 sys 0.010000 (best of 3)
! wall 5.415455 comb 5.420000 user 5.410000 sys 0.010000 (best of 3)

The data shows eliminating delta chains makes the changelog part of
changegroup generation slower. This is expected since we now have to
compute deltas for revisions where we could recycle the delta before.

It is worth putting this regression into context of overall changegroup
times. Here is the rough total CPU time spent in changegroup generation
for various repos while using delta chains on the changelog:

Repo              CPU Time (s)    CPU Time w/ compression
hg                  4.50              7.05
mozilla-central   111.1             222.0
pypy               28.68             75.5

Before compression, removing delta chains from the changegroup adds
~4.4% overhead to hg changegroup generation, 1.3% to mozilla-central,
and 2.0% to pypy. When you factor in zlib compression, these percentages
are roughly divided by 2.

While the increased CPU usage for changegroup generation is unfortunate,
I think it is acceptable because the percentage is small, server
operators (those likely impacted most by this) have other mechanisms
to mitigate CPU consumption (namely reducing zlib compression level and
pre-generated clone bundles), and because there is room to optimize this
in the future. For example, we could use the nullid as the base revision,
effectively encoding the full revision for each entry in the changegroup.
When doing this, `hg perfchangegroupchangelog` nearly halves:

mozilla-unified
! wall 21.168075 comb 21.170000 user 21.130000 sys 0.040000 (best of 3)
! wall 11.196461 comb 11.200000 user 11.190000 sys 0.010000 (best of 3)

This looks very promising as a future optimization opportunity.

It's worth that the changes in test-acl.t to the changegroup part size.
This is because revision 6 in the changegroup had a delta chain of
length 2 before and after this patch the base revision is nullrev.
When the base revision is nullrev, cg2packer.deltaparent() hardcodes
the *previous* revision from the changegroup as the delta parent.
This caused the delta in the changegroup to switch base revisions,
the delta to change, and the size to change accordingly. While the
size increased in this case, I think sizes will remain the same
on average, as the delta base for changelog revisions doesn't matter
too much (as this patch shows). So, I don't consider this a regression.

commit hooks can see env vars
(and post-transaction one are run unlocked)


  $ cat > $TESTTMP/txnabort.checkargs.py <<EOF
  > def showargs(ui, repo, hooktype, **kwargs):
  >     ui.write('%s python hook: %s\n' % (hooktype, ','.join(sorted(kwargs))))
  > EOF

  $ hg init a
  $ cd a
  $ cat > .hg/hgrc <<EOF
  > [hooks]
  > commit = sh -c "HG_LOCAL= HG_TAG= printenv.py commit"
  > commit.b = sh -c "HG_LOCAL= HG_TAG= printenv.py commit.b"
  > precommit = sh -c  "HG_LOCAL= HG_NODE= HG_TAG= printenv.py precommit"
  > pretxncommit = sh -c "HG_LOCAL= HG_TAG= printenv.py pretxncommit"
  > pretxncommit.tip = hg -q tip
  > pre-identify = printenv.py pre-identify 1
  > pre-cat = printenv.py pre-cat
  > post-cat = printenv.py post-cat
  > pretxnopen = sh -c "HG_LOCAL= HG_TAG= printenv.py pretxnopen"
  > pretxnclose = sh -c "HG_LOCAL= HG_TAG= printenv.py pretxnclose"
  > txnclose = sh -c "HG_LOCAL= HG_TAG= printenv.py txnclose"
  > txnabort.0 = python:$TESTTMP/txnabort.checkargs.py:showargs
  > txnabort.1 = sh -c "HG_LOCAL= HG_TAG= printenv.py txnabort"
  > txnclose.checklock = sh -c "hg debuglock > /dev/null"
  > EOF
  $ echo a > a
  $ hg add a
  $ hg commit -m a
  precommit hook: HG_PARENT1=0000000000000000000000000000000000000000
  pretxnopen hook: HG_TXNID=TXN:* HG_TXNNAME=commit (glob)
  pretxncommit hook: HG_NODE=cb9a9f314b8b07ba71012fcdbc544b5a4d82ff5b HG_PARENT1=0000000000000000000000000000000000000000 HG_PENDING=$TESTTMP/a
  0:cb9a9f314b8b
  pretxnclose hook: HG_PENDING=$TESTTMP/a HG_PHASES_MOVED=1 HG_TXNID=TXN:* HG_TXNNAME=commit (glob)
  txnclose hook: HG_PHASES_MOVED=1 HG_TXNID=TXN:* HG_TXNNAME=commit (glob)
  commit hook: HG_NODE=cb9a9f314b8b07ba71012fcdbc544b5a4d82ff5b HG_PARENT1=0000000000000000000000000000000000000000
  commit.b hook: HG_NODE=cb9a9f314b8b07ba71012fcdbc544b5a4d82ff5b HG_PARENT1=0000000000000000000000000000000000000000

  $ hg clone . ../b
  updating to branch default
  1 files updated, 0 files merged, 0 files removed, 0 files unresolved
  $ cd ../b

changegroup hooks can see env vars

  $ cat > .hg/hgrc <<EOF
  > [hooks]
  > prechangegroup = printenv.py prechangegroup
  > changegroup = printenv.py changegroup
  > incoming = printenv.py incoming
  > EOF

pretxncommit and commit hooks can see both parents of merge

  $ cd ../a
  $ echo b >> a
  $ hg commit -m a1 -d "1 0"
  precommit hook: HG_PARENT1=cb9a9f314b8b07ba71012fcdbc544b5a4d82ff5b
  pretxnopen hook: HG_TXNID=TXN:* HG_TXNNAME=commit (glob)
  pretxncommit hook: HG_NODE=ab228980c14deea8b9555d91c9581127383e40fd HG_PARENT1=cb9a9f314b8b07ba71012fcdbc544b5a4d82ff5b HG_PENDING=$TESTTMP/a
  1:ab228980c14d
  pretxnclose hook: HG_PENDING=$TESTTMP/a HG_TXNID=TXN:* HG_TXNNAME=commit (glob)
  txnclose hook: HG_TXNID=TXN:* HG_TXNNAME=commit (glob)
  commit hook: HG_NODE=ab228980c14deea8b9555d91c9581127383e40fd HG_PARENT1=cb9a9f314b8b07ba71012fcdbc544b5a4d82ff5b
  commit.b hook: HG_NODE=ab228980c14deea8b9555d91c9581127383e40fd HG_PARENT1=cb9a9f314b8b07ba71012fcdbc544b5a4d82ff5b
  $ hg update -C 0
  1 files updated, 0 files merged, 0 files removed, 0 files unresolved
  $ echo b > b
  $ hg add b
  $ hg commit -m b -d '1 0'
  precommit hook: HG_PARENT1=cb9a9f314b8b07ba71012fcdbc544b5a4d82ff5b
  pretxnopen hook: HG_TXNID=TXN:* HG_TXNNAME=commit (glob)
  pretxncommit hook: HG_NODE=ee9deb46ab31e4cc3310f3cf0c3d668e4d8fffc2 HG_PARENT1=cb9a9f314b8b07ba71012fcdbc544b5a4d82ff5b HG_PENDING=$TESTTMP/a
  2:ee9deb46ab31
  pretxnclose hook: HG_PENDING=$TESTTMP/a HG_TXNID=TXN:* HG_TXNNAME=commit (glob)
  created new head
  txnclose hook: HG_TXNID=TXN:* HG_TXNNAME=commit (glob)
  commit hook: HG_NODE=ee9deb46ab31e4cc3310f3cf0c3d668e4d8fffc2 HG_PARENT1=cb9a9f314b8b07ba71012fcdbc544b5a4d82ff5b
  commit.b hook: HG_NODE=ee9deb46ab31e4cc3310f3cf0c3d668e4d8fffc2 HG_PARENT1=cb9a9f314b8b07ba71012fcdbc544b5a4d82ff5b
  $ hg merge 1
  1 files updated, 0 files merged, 0 files removed, 0 files unresolved
  (branch merge, don't forget to commit)
  $ hg commit -m merge -d '2 0'
  precommit hook: HG_PARENT1=ee9deb46ab31e4cc3310f3cf0c3d668e4d8fffc2 HG_PARENT2=ab228980c14deea8b9555d91c9581127383e40fd
  pretxnopen hook: HG_TXNID=TXN:* HG_TXNNAME=commit (glob)
  pretxncommit hook: HG_NODE=07f3376c1e655977439df2a814e3cc14b27abac2 HG_PARENT1=ee9deb46ab31e4cc3310f3cf0c3d668e4d8fffc2 HG_PARENT2=ab228980c14deea8b9555d91c9581127383e40fd HG_PENDING=$TESTTMP/a
  3:07f3376c1e65
  pretxnclose hook: HG_PENDING=$TESTTMP/a HG_TXNID=TXN:* HG_TXNNAME=commit (glob)
  txnclose hook: HG_TXNID=TXN:* HG_TXNNAME=commit (glob)
  commit hook: HG_NODE=07f3376c1e655977439df2a814e3cc14b27abac2 HG_PARENT1=ee9deb46ab31e4cc3310f3cf0c3d668e4d8fffc2 HG_PARENT2=ab228980c14deea8b9555d91c9581127383e40fd
  commit.b hook: HG_NODE=07f3376c1e655977439df2a814e3cc14b27abac2 HG_PARENT1=ee9deb46ab31e4cc3310f3cf0c3d668e4d8fffc2 HG_PARENT2=ab228980c14deea8b9555d91c9581127383e40fd

test generic hooks

  $ hg id
  pre-identify hook: HG_ARGS=id HG_OPTS={'bookmarks': None, 'branch': None, 'id': None, 'insecure': None, 'num': None, 'remotecmd': '', 'rev': '', 'ssh': '', 'tags': None} HG_PATS=[]
  abort: pre-identify hook exited with status 1
  [255]
  $ hg cat b
  pre-cat hook: HG_ARGS=cat b HG_OPTS={'decode': None, 'exclude': [], 'include': [], 'output': '', 'rev': ''} HG_PATS=['b']
  b
  post-cat hook: HG_ARGS=cat b HG_OPTS={'decode': None, 'exclude': [], 'include': [], 'output': '', 'rev': ''} HG_PATS=['b'] HG_RESULT=0

  $ cd ../b
  $ hg pull ../a
  pulling from ../a
  searching for changes
  prechangegroup hook: HG_SOURCE=pull HG_TXNID=TXN:* HG_URL=file:$TESTTMP/a (glob)
  adding changesets
  adding manifests
  adding file changes
  added 3 changesets with 2 changes to 2 files
  changegroup hook: HG_NODE=ab228980c14deea8b9555d91c9581127383e40fd HG_NODE_LAST=07f3376c1e655977439df2a814e3cc14b27abac2 HG_SOURCE=pull HG_TXNID=TXN:* HG_URL=file:$TESTTMP/a (glob)
  incoming hook: HG_NODE=ab228980c14deea8b9555d91c9581127383e40fd HG_SOURCE=pull HG_TXNID=TXN:* HG_URL=file:$TESTTMP/a (glob)
  incoming hook: HG_NODE=ee9deb46ab31e4cc3310f3cf0c3d668e4d8fffc2 HG_SOURCE=pull HG_TXNID=TXN:* HG_URL=file:$TESTTMP/a (glob)
  incoming hook: HG_NODE=07f3376c1e655977439df2a814e3cc14b27abac2 HG_SOURCE=pull HG_TXNID=TXN:* HG_URL=file:$TESTTMP/a (glob)
  (run 'hg update' to get a working copy)

tag hooks can see env vars

  $ cd ../a
  $ cat >> .hg/hgrc <<EOF
  > pretag = printenv.py pretag
  > tag = sh -c "HG_PARENT1= HG_PARENT2= printenv.py tag"
  > EOF
  $ hg tag -d '3 0' a
  pretag hook: HG_LOCAL=0 HG_NODE=07f3376c1e655977439df2a814e3cc14b27abac2 HG_TAG=a
  precommit hook: HG_PARENT1=07f3376c1e655977439df2a814e3cc14b27abac2
  pretxnopen hook: HG_TXNID=TXN:* HG_TXNNAME=commit (glob)
  pretxncommit hook: HG_NODE=539e4b31b6dc99b3cfbaa6b53cbc1c1f9a1e3a10 HG_PARENT1=07f3376c1e655977439df2a814e3cc14b27abac2 HG_PENDING=$TESTTMP/a
  4:539e4b31b6dc
  pretxnclose hook: HG_PENDING=$TESTTMP/a HG_TXNID=TXN:* HG_TXNNAME=commit (glob)
  tag hook: HG_LOCAL=0 HG_NODE=07f3376c1e655977439df2a814e3cc14b27abac2 HG_TAG=a
  txnclose hook: HG_TXNID=TXN:* HG_TXNNAME=commit (glob)
  commit hook: HG_NODE=539e4b31b6dc99b3cfbaa6b53cbc1c1f9a1e3a10 HG_PARENT1=07f3376c1e655977439df2a814e3cc14b27abac2
  commit.b hook: HG_NODE=539e4b31b6dc99b3cfbaa6b53cbc1c1f9a1e3a10 HG_PARENT1=07f3376c1e655977439df2a814e3cc14b27abac2
  $ hg tag -l la
  pretag hook: HG_LOCAL=1 HG_NODE=539e4b31b6dc99b3cfbaa6b53cbc1c1f9a1e3a10 HG_TAG=la
  tag hook: HG_LOCAL=1 HG_NODE=539e4b31b6dc99b3cfbaa6b53cbc1c1f9a1e3a10 HG_TAG=la

pretag hook can forbid tagging

  $ echo "pretag.forbid = printenv.py pretag.forbid 1" >> .hg/hgrc
  $ hg tag -d '4 0' fa
  pretag hook: HG_LOCAL=0 HG_NODE=539e4b31b6dc99b3cfbaa6b53cbc1c1f9a1e3a10 HG_TAG=fa
  pretag.forbid hook: HG_LOCAL=0 HG_NODE=539e4b31b6dc99b3cfbaa6b53cbc1c1f9a1e3a10 HG_TAG=fa
  abort: pretag.forbid hook exited with status 1
  [255]
  $ hg tag -l fla
  pretag hook: HG_LOCAL=1 HG_NODE=539e4b31b6dc99b3cfbaa6b53cbc1c1f9a1e3a10 HG_TAG=fla
  pretag.forbid hook: HG_LOCAL=1 HG_NODE=539e4b31b6dc99b3cfbaa6b53cbc1c1f9a1e3a10 HG_TAG=fla
  abort: pretag.forbid hook exited with status 1
  [255]

pretxncommit hook can see changeset, can roll back txn, changeset no
more there after

  $ echo "pretxncommit.forbid0 = hg tip -q" >> .hg/hgrc
  $ echo "pretxncommit.forbid1 = printenv.py pretxncommit.forbid 1" >> .hg/hgrc
  $ echo z > z
  $ hg add z
  $ hg -q tip
  4:539e4b31b6dc
  $ hg commit -m 'fail' -d '4 0'
  precommit hook: HG_PARENT1=539e4b31b6dc99b3cfbaa6b53cbc1c1f9a1e3a10
  pretxnopen hook: HG_TXNID=TXN:* HG_TXNNAME=commit (glob)
  pretxncommit hook: HG_NODE=6f611f8018c10e827fee6bd2bc807f937e761567 HG_PARENT1=539e4b31b6dc99b3cfbaa6b53cbc1c1f9a1e3a10 HG_PENDING=$TESTTMP/a
  5:6f611f8018c1
  5:6f611f8018c1
  pretxncommit.forbid hook: HG_NODE=6f611f8018c10e827fee6bd2bc807f937e761567 HG_PARENT1=539e4b31b6dc99b3cfbaa6b53cbc1c1f9a1e3a10 HG_PENDING=$TESTTMP/a
  transaction abort!
  txnabort python hook: txnid,txnname
  txnabort hook: HG_TXNID=TXN:* HG_TXNNAME=commit (glob)
  rollback completed
  abort: pretxncommit.forbid1 hook exited with status 1
  [255]
  $ hg -q tip
  4:539e4b31b6dc

(Check that no 'changelog.i.a' file were left behind)

  $ ls -1 .hg/store/
  00changelog.i
  00manifest.i
  data
  fncache
  journal.phaseroots
  phaseroots
  undo
  undo.backup.fncache
  undo.backupfiles
  undo.phaseroots


precommit hook can prevent commit

  $ echo "precommit.forbid = printenv.py precommit.forbid 1" >> .hg/hgrc
  $ hg commit -m 'fail' -d '4 0'
  precommit hook: HG_PARENT1=539e4b31b6dc99b3cfbaa6b53cbc1c1f9a1e3a10
  precommit.forbid hook: HG_PARENT1=539e4b31b6dc99b3cfbaa6b53cbc1c1f9a1e3a10
  abort: precommit.forbid hook exited with status 1
  [255]
  $ hg -q tip
  4:539e4b31b6dc

preupdate hook can prevent update

  $ echo "preupdate = printenv.py preupdate" >> .hg/hgrc
  $ hg update 1
  preupdate hook: HG_PARENT1=ab228980c14d
  0 files updated, 0 files merged, 2 files removed, 0 files unresolved

update hook

  $ echo "update = printenv.py update" >> .hg/hgrc
  $ hg update
  preupdate hook: HG_PARENT1=539e4b31b6dc
  update hook: HG_ERROR=0 HG_PARENT1=539e4b31b6dc
  2 files updated, 0 files merged, 0 files removed, 0 files unresolved

pushkey hook

  $ echo "pushkey = printenv.py pushkey" >> .hg/hgrc
  $ cd ../b
  $ hg bookmark -r null foo
  $ hg push -B foo ../a
  pushing to ../a
  searching for changes
  no changes found
  pretxnopen hook: HG_TXNID=TXN:* HG_TXNNAME=push (glob)
  pretxnclose hook: HG_BOOKMARK_MOVED=1 HG_BUNDLE2=1 HG_PENDING=$TESTTMP/a HG_SOURCE=push HG_TXNID=TXN:* HG_TXNNAME=push HG_URL=file:$TESTTMP/a (glob)
  pushkey hook: HG_KEY=foo HG_NAMESPACE=bookmarks HG_NEW=0000000000000000000000000000000000000000 HG_RET=1
  txnclose hook: HG_BOOKMARK_MOVED=1 HG_BUNDLE2=1 HG_SOURCE=push HG_TXNID=TXN:* HG_TXNNAME=push HG_URL=file:$TESTTMP/a (glob)
  exporting bookmark foo
  [1]
  $ cd ../a

listkeys hook

  $ echo "listkeys = printenv.py listkeys" >> .hg/hgrc
  $ hg bookmark -r null bar
  pretxnopen hook: HG_TXNID=TXN:* HG_TXNNAME=bookmark (glob)
  pretxnclose hook: HG_BOOKMARK_MOVED=1 HG_PENDING=$TESTTMP/a HG_TXNID=TXN:* HG_TXNNAME=bookmark (glob)
  txnclose hook: HG_BOOKMARK_MOVED=1 HG_TXNID=TXN:* HG_TXNNAME=bookmark (glob)
  $ cd ../b
  $ hg pull -B bar ../a
  pulling from ../a
  listkeys hook: HG_NAMESPACE=bookmarks HG_VALUES={'bar': '0000000000000000000000000000000000000000', 'foo': '0000000000000000000000000000000000000000'}
  no changes found
  listkeys hook: HG_NAMESPACE=phases HG_VALUES={'cb9a9f314b8b07ba71012fcdbc544b5a4d82ff5b': '1', 'publishing': 'True'}
  adding remote bookmark bar
  $ cd ../a

test that prepushkey can prevent incoming keys

  $ echo "prepushkey = printenv.py prepushkey.forbid 1" >> .hg/hgrc
  $ cd ../b
  $ hg bookmark -r null baz
  $ hg push -B baz ../a
  pushing to ../a
  searching for changes
  listkeys hook: HG_NAMESPACE=phases HG_VALUES={'cb9a9f314b8b07ba71012fcdbc544b5a4d82ff5b': '1', 'publishing': 'True'}
  listkeys hook: HG_NAMESPACE=bookmarks HG_VALUES={'bar': '0000000000000000000000000000000000000000', 'foo': '0000000000000000000000000000000000000000'}
  no changes found
  pretxnopen hook: HG_TXNID=TXN:* HG_TXNNAME=push (glob)
  prepushkey.forbid hook: HG_BUNDLE2=1 HG_KEY=baz HG_NAMESPACE=bookmarks HG_NEW=0000000000000000000000000000000000000000 HG_SOURCE=push HG_TXNID=TXN:* HG_URL=file:$TESTTMP/a (glob)
  pushkey-abort: prepushkey hook exited with status 1
  abort: exporting bookmark baz failed!
  [255]
  $ cd ../a

test that prelistkeys can prevent listing keys

  $ echo "prelistkeys = printenv.py prelistkeys.forbid 1" >> .hg/hgrc
  $ hg bookmark -r null quux
  pretxnopen hook: HG_TXNID=TXN:* HG_TXNNAME=bookmark (glob)
  pretxnclose hook: HG_BOOKMARK_MOVED=1 HG_PENDING=$TESTTMP/a HG_TXNID=TXN:* HG_TXNNAME=bookmark (glob)
  txnclose hook: HG_BOOKMARK_MOVED=1 HG_TXNID=TXN:* HG_TXNNAME=bookmark (glob)
  $ cd ../b
  $ hg pull -B quux ../a
  pulling from ../a
  prelistkeys.forbid hook: HG_NAMESPACE=bookmarks
  abort: prelistkeys hook exited with status 1
  [255]
  $ cd ../a
  $ rm .hg/hgrc

prechangegroup hook can prevent incoming changes

  $ cd ../b
  $ hg -q tip
  3:07f3376c1e65
  $ cat > .hg/hgrc <<EOF
  > [hooks]
  > prechangegroup.forbid = printenv.py prechangegroup.forbid 1
  > EOF
  $ hg pull ../a
  pulling from ../a
  searching for changes
  prechangegroup.forbid hook: HG_SOURCE=pull HG_TXNID=TXN:* HG_URL=file:$TESTTMP/a (glob)
  abort: prechangegroup.forbid hook exited with status 1
  [255]

pretxnchangegroup hook can see incoming changes, can roll back txn,
incoming changes no longer there after

  $ cat > .hg/hgrc <<EOF
  > [hooks]
  > pretxnchangegroup.forbid0 = hg tip -q
  > pretxnchangegroup.forbid1 = printenv.py pretxnchangegroup.forbid 1
  > EOF
  $ hg pull ../a
  pulling from ../a
  searching for changes
  adding changesets
  adding manifests
  adding file changes
  added 1 changesets with 1 changes to 1 files
  4:539e4b31b6dc
  pretxnchangegroup.forbid hook: HG_NODE=539e4b31b6dc99b3cfbaa6b53cbc1c1f9a1e3a10 HG_NODE_LAST=539e4b31b6dc99b3cfbaa6b53cbc1c1f9a1e3a10 HG_PENDING=$TESTTMP/b HG_SOURCE=pull HG_TXNID=TXN:* HG_URL=file:$TESTTMP/a (glob)
  transaction abort!
  rollback completed
  abort: pretxnchangegroup.forbid1 hook exited with status 1
  [255]
  $ hg -q tip
  3:07f3376c1e65

outgoing hooks can see env vars

  $ rm .hg/hgrc
  $ cat > ../a/.hg/hgrc <<EOF
  > [hooks]
  > preoutgoing = printenv.py preoutgoing
  > outgoing = printenv.py outgoing
  > EOF
  $ hg pull ../a
  pulling from ../a
  searching for changes
  preoutgoing hook: HG_SOURCE=pull
  outgoing hook: HG_NODE=539e4b31b6dc99b3cfbaa6b53cbc1c1f9a1e3a10 HG_SOURCE=pull
  adding changesets
  adding manifests
  adding file changes
  added 1 changesets with 1 changes to 1 files
  adding remote bookmark quux
  (run 'hg update' to get a working copy)
  $ hg rollback
  repository tip rolled back to revision 3 (undo pull)

preoutgoing hook can prevent outgoing changes

  $ echo "preoutgoing.forbid = printenv.py preoutgoing.forbid 1" >> ../a/.hg/hgrc
  $ hg pull ../a
  pulling from ../a
  searching for changes
  preoutgoing hook: HG_SOURCE=pull
  preoutgoing.forbid hook: HG_SOURCE=pull
  abort: preoutgoing.forbid hook exited with status 1
  [255]

outgoing hooks work for local clones

  $ cd ..
  $ cat > a/.hg/hgrc <<EOF
  > [hooks]
  > preoutgoing = printenv.py preoutgoing
  > outgoing = printenv.py outgoing
  > EOF
  $ hg clone a c
  preoutgoing hook: HG_SOURCE=clone
  outgoing hook: HG_NODE=0000000000000000000000000000000000000000 HG_SOURCE=clone
  updating to branch default
  3 files updated, 0 files merged, 0 files removed, 0 files unresolved
  $ rm -rf c

preoutgoing hook can prevent outgoing changes for local clones

  $ echo "preoutgoing.forbid = printenv.py preoutgoing.forbid 1" >> a/.hg/hgrc
  $ hg clone a zzz
  preoutgoing hook: HG_SOURCE=clone
  preoutgoing.forbid hook: HG_SOURCE=clone
  abort: preoutgoing.forbid hook exited with status 1
  [255]

  $ cd "$TESTTMP/b"

  $ cat > hooktests.py <<EOF
  > from mercurial import error
  > 
  > uncallable = 0
  > 
  > def printargs(args):
  >     args.pop('ui', None)
  >     args.pop('repo', None)
  >     a = list(args.items())
  >     a.sort()
  >     print 'hook args:'
  >     for k, v in a:
  >        print ' ', k, v
  > 
  > def passhook(**args):
  >     printargs(args)
  > 
  > def failhook(**args):
  >     printargs(args)
  >     return True
  > 
  > class LocalException(Exception):
  >     pass
  > 
  > def raisehook(**args):
  >     raise LocalException('exception from hook')
  > 
  > def aborthook(**args):
  >     raise error.Abort('raise abort from hook')
  > 
  > def brokenhook(**args):
  >     return 1 + {}
  > 
  > def verbosehook(ui, **args):
  >     ui.note('verbose output from hook\n')
  > 
  > def printtags(ui, repo, **args):
  >     print sorted(repo.tags())
  > 
  > class container:
  >     unreachable = 1
  > EOF

  $ cat > syntaxerror.py << EOF
  > (foo
  > EOF

test python hooks

#if windows
  $ PYTHONPATH="$TESTTMP/b;$PYTHONPATH"
#else
  $ PYTHONPATH="$TESTTMP/b:$PYTHONPATH"
#endif
  $ export PYTHONPATH

  $ echo '[hooks]' > ../a/.hg/hgrc
  $ echo 'preoutgoing.broken = python:hooktests.brokenhook' >> ../a/.hg/hgrc
  $ hg pull ../a 2>&1 | grep 'raised an exception'
  error: preoutgoing.broken hook raised an exception: unsupported operand type(s) for +: 'int' and 'dict'

  $ echo '[hooks]' > ../a/.hg/hgrc
  $ echo 'preoutgoing.raise = python:hooktests.raisehook' >> ../a/.hg/hgrc
  $ hg pull ../a 2>&1 | grep 'raised an exception'
  error: preoutgoing.raise hook raised an exception: exception from hook

  $ echo '[hooks]' > ../a/.hg/hgrc
  $ echo 'preoutgoing.abort = python:hooktests.aborthook' >> ../a/.hg/hgrc
  $ hg pull ../a
  pulling from ../a
  searching for changes
  error: preoutgoing.abort hook failed: raise abort from hook
  abort: raise abort from hook
  [255]

  $ echo '[hooks]' > ../a/.hg/hgrc
  $ echo 'preoutgoing.fail = python:hooktests.failhook' >> ../a/.hg/hgrc
  $ hg pull ../a
  pulling from ../a
  searching for changes
  hook args:
    hooktype preoutgoing
    source pull
  abort: preoutgoing.fail hook failed
  [255]

  $ echo '[hooks]' > ../a/.hg/hgrc
  $ echo 'preoutgoing.uncallable = python:hooktests.uncallable' >> ../a/.hg/hgrc
  $ hg pull ../a
  pulling from ../a
  searching for changes
  abort: preoutgoing.uncallable hook is invalid: "hooktests.uncallable" is not callable
  [255]

  $ echo '[hooks]' > ../a/.hg/hgrc
  $ echo 'preoutgoing.nohook = python:hooktests.nohook' >> ../a/.hg/hgrc
  $ hg pull ../a
  pulling from ../a
  searching for changes
  abort: preoutgoing.nohook hook is invalid: "hooktests.nohook" is not defined
  [255]

  $ echo '[hooks]' > ../a/.hg/hgrc
  $ echo 'preoutgoing.nomodule = python:nomodule' >> ../a/.hg/hgrc
  $ hg pull ../a
  pulling from ../a
  searching for changes
  abort: preoutgoing.nomodule hook is invalid: "nomodule" not in a module
  [255]

  $ echo '[hooks]' > ../a/.hg/hgrc
  $ echo 'preoutgoing.badmodule = python:nomodule.nowhere' >> ../a/.hg/hgrc
  $ hg pull ../a
  pulling from ../a
  searching for changes
  abort: preoutgoing.badmodule hook is invalid: import of "nomodule" failed
  (run with --traceback for stack trace)
  [255]

  $ echo '[hooks]' > ../a/.hg/hgrc
  $ echo 'preoutgoing.unreachable = python:hooktests.container.unreachable' >> ../a/.hg/hgrc
  $ hg pull ../a
  pulling from ../a
  searching for changes
  abort: preoutgoing.unreachable hook is invalid: import of "hooktests.container" failed
  (run with --traceback for stack trace)
  [255]

  $ echo '[hooks]' > ../a/.hg/hgrc
  $ echo 'preoutgoing.syntaxerror = python:syntaxerror.syntaxerror' >> ../a/.hg/hgrc
  $ hg pull ../a
  pulling from ../a
  searching for changes
  abort: preoutgoing.syntaxerror hook is invalid: import of "syntaxerror" failed
  (run with --traceback for stack trace)
  [255]

The second egrep is to filter out lines like '    ^', which are slightly
different between Python 2.6 and Python 2.7.
  $ hg pull ../a --traceback 2>&1 | egrep -v '^( +File|    [_a-zA-Z*(])' | egrep -v '^( )+(\^)?$'
  pulling from ../a
  searching for changes
  exception from first failed import attempt:
  Traceback (most recent call last):
  SyntaxError: * (glob)
  exception from second failed import attempt:
  Traceback (most recent call last):
  ImportError: No module named hgext_syntaxerror
  Traceback (most recent call last):
  HookLoadError: preoutgoing.syntaxerror hook is invalid: import of "syntaxerror" failed
  abort: preoutgoing.syntaxerror hook is invalid: import of "syntaxerror" failed

  $ echo '[hooks]' > ../a/.hg/hgrc
  $ echo 'preoutgoing.pass = python:hooktests.passhook' >> ../a/.hg/hgrc
  $ hg pull ../a
  pulling from ../a
  searching for changes
  hook args:
    hooktype preoutgoing
    source pull
  adding changesets
  adding manifests
  adding file changes
  added 1 changesets with 1 changes to 1 files
  adding remote bookmark quux
  (run 'hg update' to get a working copy)

post- python hooks that fail to *run* don't cause an abort
  $ rm ../a/.hg/hgrc
  $ echo '[hooks]' > .hg/hgrc
  $ echo 'post-pull.broken = python:hooktests.brokenhook' >> .hg/hgrc
  $ hg pull ../a
  pulling from ../a
  searching for changes
  no changes found
  error: post-pull.broken hook raised an exception: unsupported operand type(s) for +: 'int' and 'dict'
  (run with --traceback for stack trace)

but post- python hooks that fail to *load* do
  $ echo '[hooks]' > .hg/hgrc
  $ echo 'post-pull.nomodule = python:nomodule' >> .hg/hgrc
  $ hg pull ../a
  pulling from ../a
  searching for changes
  no changes found
  abort: post-pull.nomodule hook is invalid: "nomodule" not in a module
  [255]

  $ echo '[hooks]' > .hg/hgrc
  $ echo 'post-pull.badmodule = python:nomodule.nowhere' >> .hg/hgrc
  $ hg pull ../a
  pulling from ../a
  searching for changes
  no changes found
  abort: post-pull.badmodule hook is invalid: import of "nomodule" failed
  (run with --traceback for stack trace)
  [255]

  $ echo '[hooks]' > .hg/hgrc
  $ echo 'post-pull.nohook = python:hooktests.nohook' >> .hg/hgrc
  $ hg pull ../a
  pulling from ../a
  searching for changes
  no changes found
  abort: post-pull.nohook hook is invalid: "hooktests.nohook" is not defined
  [255]

make sure --traceback works

  $ echo '[hooks]' > .hg/hgrc
  $ echo 'commit.abort = python:hooktests.aborthook' >> .hg/hgrc

  $ echo aa > a
  $ hg --traceback commit -d '0 0' -ma 2>&1 | grep '^Traceback'
  Traceback (most recent call last):

  $ cd ..
  $ hg init c
  $ cd c

  $ cat > hookext.py <<EOF
  > def autohook(**args):
  >     print "Automatically installed hook"
  > 
  > def reposetup(ui, repo):
  >     repo.ui.setconfig("hooks", "commit.auto", autohook)
  > EOF
  $ echo '[extensions]' >> .hg/hgrc
  $ echo 'hookext = hookext.py' >> .hg/hgrc

  $ touch foo
  $ hg add foo
  $ hg ci -d '0 0' -m 'add foo'
  Automatically installed hook
  $ echo >> foo
  $ hg ci --debug -d '0 0' -m 'change foo'
  committing files:
  foo
  committing manifest
  committing changelog
  committed changeset 1:52998019f6252a2b893452765fcb0a47351a5708
  calling hook commit.auto: hgext_hookext.autohook
  Automatically installed hook

  $ hg showconfig hooks
  hooks.commit.auto=<function autohook at *> (glob)

test python hook configured with python:[file]:[hook] syntax

  $ cd ..
  $ mkdir d
  $ cd d
  $ hg init repo
  $ mkdir hooks

  $ cd hooks
  $ cat > testhooks.py <<EOF
  > def testhook(**args):
  >     print 'hook works'
  > EOF
  $ echo '[hooks]' > ../repo/.hg/hgrc
  $ echo "pre-commit.test = python:`pwd`/testhooks.py:testhook" >> ../repo/.hg/hgrc

  $ cd ../repo
  $ hg commit -d '0 0'
  hook works
  nothing changed
  [1]

  $ echo '[hooks]' > .hg/hgrc
  $ echo "update.ne = python:`pwd`/nonexistent.py:testhook" >> .hg/hgrc
  $ echo "pre-identify.npmd = python:`pwd`/:no_python_module_dir" >> .hg/hgrc

  $ hg up null
  loading update.ne hook failed:
  abort: No such file or directory: $TESTTMP/d/repo/nonexistent.py
  [255]

  $ hg id
  loading pre-identify.npmd hook failed:
  abort: No module named repo!
  [255]

  $ cd ../../b

make sure --traceback works on hook import failure

  $ cat > importfail.py <<EOF
  > import somebogusmodule
  > # dereference something in the module to force demandimport to load it
  > somebogusmodule.whatever
  > EOF

  $ echo '[hooks]' > .hg/hgrc
  $ echo 'precommit.importfail = python:importfail.whatever' >> .hg/hgrc

  $ echo a >> a
  $ hg --traceback commit -ma 2>&1 | egrep -v '^( +File|    [a-zA-Z(])'
  exception from first failed import attempt:
  Traceback (most recent call last):
  ImportError: No module named somebogusmodule
  exception from second failed import attempt:
  Traceback (most recent call last):
  ImportError: No module named hgext_importfail
  Traceback (most recent call last):
  HookLoadError: precommit.importfail hook is invalid: import of "importfail" failed
  abort: precommit.importfail hook is invalid: import of "importfail" failed

Issue1827: Hooks Update & Commit not completely post operation

commit and update hooks should run after command completion.  The largefiles
use demonstrates a recursive wlock, showing the hook doesn't run until the
final release (and dirstate flush).

  $ echo '[hooks]' > .hg/hgrc
  $ echo 'commit = hg id' >> .hg/hgrc
  $ echo 'update = hg id' >> .hg/hgrc
  $ echo bb > a
  $ hg ci -ma
  223eafe2750c tip
  $ hg up 0 --config extensions.largefiles=
  cb9a9f314b8b
  1 files updated, 0 files merged, 0 files removed, 0 files unresolved

make sure --verbose (and --quiet/--debug etc.) are propagated to the local ui
that is passed to pre/post hooks

  $ echo '[hooks]' > .hg/hgrc
  $ echo 'pre-identify = python:hooktests.verbosehook' >> .hg/hgrc
  $ hg id
  cb9a9f314b8b
  $ hg id --verbose
  calling hook pre-identify: hooktests.verbosehook
  verbose output from hook
  cb9a9f314b8b

Ensure hooks can be prioritized

  $ echo '[hooks]' > .hg/hgrc
  $ echo 'pre-identify.a = python:hooktests.verbosehook' >> .hg/hgrc
  $ echo 'pre-identify.b = python:hooktests.verbosehook' >> .hg/hgrc
  $ echo 'priority.pre-identify.b = 1' >> .hg/hgrc
  $ echo 'pre-identify.c = python:hooktests.verbosehook' >> .hg/hgrc
  $ hg id --verbose
  calling hook pre-identify.b: hooktests.verbosehook
  verbose output from hook
  calling hook pre-identify.a: hooktests.verbosehook
  verbose output from hook
  calling hook pre-identify.c: hooktests.verbosehook
  verbose output from hook
  cb9a9f314b8b

new tags must be visible in pretxncommit (issue3210)

  $ echo 'pretxncommit.printtags = python:hooktests.printtags' >> .hg/hgrc
  $ hg tag -f foo
  ['a', 'foo', 'tip']

post-init hooks must not crash (issue4983)
This also creates the `to` repo for the next test block.

  $ cd ..
  $ cat << EOF >> hgrc-with-post-init-hook
  > [hooks]
  > post-init = printenv.py post-init
  > EOF
  $ HGRCPATH=hgrc-with-post-init-hook hg init to
  post-init hook: HG_ARGS=init to HG_OPTS={'insecure': None, 'remotecmd': '', 'ssh': ''} HG_PATS=['to'] HG_RESULT=0

new commits must be visible in pretxnchangegroup (issue3428)

  $ echo '[hooks]' >> to/.hg/hgrc
  $ echo 'prechangegroup = hg --traceback tip' >> to/.hg/hgrc
  $ echo 'pretxnchangegroup = hg --traceback tip' >> to/.hg/hgrc
  $ echo a >> to/a
  $ hg --cwd to ci -Ama
  adding a
  $ hg clone to from
  updating to branch default
  1 files updated, 0 files merged, 0 files removed, 0 files unresolved
  $ echo aa >> from/a
  $ hg --cwd from ci -mb
  $ hg --cwd from push
  pushing to $TESTTMP/to (glob)
  searching for changes
  changeset:   0:cb9a9f314b8b
  tag:         tip
  user:        test
  date:        Thu Jan 01 00:00:00 1970 +0000
  summary:     a
  
  adding changesets
  adding manifests
  adding file changes
  added 1 changesets with 1 changes to 1 files
  changeset:   1:9836a07b9b9d
  tag:         tip
  user:        test
  date:        Thu Jan 01 00:00:00 1970 +0000
  summary:     b
  

pretxnclose hook failure should abort the transaction

  $ hg init txnfailure
  $ cd txnfailure
  $ touch a && hg commit -Aqm a
  $ cat >> .hg/hgrc <<EOF
  > [hooks]
  > pretxnclose.error = exit 1
  > EOF
  $ hg strip -r 0 --config extensions.strip=
  0 files updated, 0 files merged, 1 files removed, 0 files unresolved
  saved backup bundle to * (glob)
  transaction abort!
  rollback completed
  strip failed, backup bundle stored in * (glob)
  abort: pretxnclose.error hook exited with status 1
  [255]
  $ hg recover
  no interrupted transaction available
  [1]
  $ cd ..

Hook from untrusted hgrc are reported as failure
================================================

  $ cat << EOF > $TESTTMP/untrusted.py
  > from mercurial import scmutil, util
  > def uisetup(ui):
  >     class untrustedui(ui.__class__):
  >         def _trusted(self, fp, f):
  >             if util.normpath(fp.name).endswith('untrusted/.hg/hgrc'):
  >                 return False
  >             return super(untrustedui, self)._trusted(fp, f)
  >     ui.__class__ = untrustedui
  > EOF
  $ cat << EOF >> $HGRCPATH
  > [extensions]
  > untrusted=$TESTTMP/untrusted.py
  > EOF
  $ hg init untrusted
  $ cd untrusted

Non-blocking hook
-----------------

  $ cat << EOF >> .hg/hgrc
  > [hooks]
  > txnclose.testing=echo txnclose hook called
  > EOF
  $ touch a && hg commit -Aqm a
  warning: untrusted hook txnclose not executed
  $ hg log
  changeset:   0:3903775176ed
  tag:         tip
  user:        test
  date:        Thu Jan 01 00:00:00 1970 +0000
  summary:     a
  

Non-blocking hook
-----------------

  $ cat << EOF >> .hg/hgrc
  > [hooks]
  > pretxnclose.testing=echo pre-txnclose hook called
  > EOF
  $ touch b && hg commit -Aqm a
  transaction abort!
  rollback completed
  abort: untrusted hook pretxnclose not executed
  (see 'hg help config.trusted')
  [255]
  $ hg log
  changeset:   0:3903775176ed
  tag:         tip
  user:        test
  date:        Thu Jan 01 00:00:00 1970 +0000
  summary:     a