tests/test-fix-metadata.t

rebase: add boolean config item rebase.store-source

This allows to use rebase without recording a rebase_source extra
field. This is useful for example to build a mirror converted from
another SCM (such as svn) by converting only new revisions, and
then incrementally add them to the destination by pulling from the
newly converted (unrelated) repo and rebasing the new revisions
onto the last old already stored changeset. Without this patch the
rebased changesets would always receive some rebase_source that
would depend on the particular history of the conversion process,
instead of only depending on the original source revisions.

This is used to implement a hg mirror repo of SvarDOS (a partially
nonfree but completely redistributable DOS distribution) in the
scripts at https://hg.pushbx.org/ecm/svardos.scr/

In particular, cre.sh creates an svn mirror, upd.sh recreates an
entire hg repo from the svn mirror (which takes too long to do in a
regular job), and akt.sh uses hg convert with the config item
convert.svn.startrev to incrementally convert only the two most
recent revisions already found in the mirror destination plus any
possible new revisions. If any are found, the temporary repo's
changesets are pulled into the destination (as changesets from an
unrelated repository). Then the changesets corresponding to the new
revisions are rebased onto the prior final changeset. (Finally, the
two remaining duplicates of the prior head and its parent are
stripped from the destination repository.)

Without this patch, the particular rebase_source extra field would
depend on the order and times at which akt.sh was used, instead of
only depending on the source repository. In other words, whatever
sequence of upd.sh and akt.sh is used at whatever times, it is
desired that the final output repositories always match each other
exactly.

A python hook for "hg fix" that prints out the number of files and revisions
that were affected, along with which fixer tools were applied. Also checks how
many times it sees a specific key generated by one of the fixer tools defined
below.

  $ cat >> $TESTTMP/postfixhook.py <<EOF
  > import collections
  > def file(ui, repo, rev=None, path=b'', metadata=None, **kwargs):
  >   ui.status(b'fixed %s in revision %d using %s\n' %
  >             (path, rev, b', '.join(metadata.keys())))
  > def summarize(ui, repo, replacements=None, wdirwritten=False,
  >               metadata=None, **kwargs):
  >     counts = collections.defaultdict(int)
  >     keys = 0
  >     for fixername, metadatalist in metadata.items():
  >         for metadata in metadatalist:
  >             if metadata is None:
  >                 continue
  >             counts[fixername] += 1
  >             if 'key' in metadata:
  >                 keys += 1
  >     ui.status(b'saw "key" %d times\n' % (keys,))
  >     for name, count in sorted(counts.items()):
  >         ui.status(b'fixed %d files with %s\n' % (count, name))
  >     if replacements:
  >         ui.status(b'fixed %d revisions\n' % (len(replacements),))
  >     if wdirwritten:
  >         ui.status(b'fixed the working copy\n')
  > EOF

Some mock output for fixer tools that demonstrate what could go wrong with
expecting the metadata output format.

  $ printf 'new content\n' > $TESTTMP/missing
  $ printf 'not valid json\0new content\n' > $TESTTMP/invalid
  $ printf '{"key": "value"}\0new content\n' > $TESTTMP/valid

Configure some fixer tools based on the output defined above, and enable the
hooks defined above. Disable parallelism to make output of the parallel file
processing phase stable.

  $ cat >> $HGRCPATH <<EOF
  > [extensions]
  > fix =
  > [fix]
  > metadatafalse:command=cat $TESTTMP/missing
  > metadatafalse:pattern=metadatafalse
  > metadatafalse:metadata=false
  > missing:command=cat $TESTTMP/missing
  > missing:pattern=missing
  > missing:metadata=true
  > invalid:command=cat $TESTTMP/invalid
  > invalid:pattern=invalid
  > invalid:metadata=true
  > valid:command=cat $TESTTMP/valid
  > valid:pattern=valid
  > valid:metadata=true
  > [hooks]
  > postfixfile = python:$TESTTMP/postfixhook.py:file
  > postfix = python:$TESTTMP/postfixhook.py:summarize
  > [worker]
  > enabled=false
  > EOF

See what happens when we execute each of the fixer tools. Some print warnings,
some write back to the file.

  $ hg init repo
  $ cd repo

  $ printf "old content\n" > metadatafalse
  $ printf "old content\n" > invalid
  $ printf "old content\n" > missing
  $ printf "old content\n" > valid
  $ hg add -q

  $ hg fix -w
  ignored invalid output from fixer tool: invalid
  fixed metadatafalse in revision 2147483647 using metadatafalse
  ignored invalid output from fixer tool: missing
  fixed valid in revision 2147483647 using valid
  saw "key" 1 times
  fixed 1 files with valid
  fixed the working copy

  $ cat metadatafalse
  new content
  $ cat missing
  old content
  $ cat invalid
  old content
  $ cat valid
  new content

  $ cd ..