view tests/test-logtoprocess.t @ 47120:7109a38830c9

dirstate-tree: Fold "tracked descendants" counter update in main walk

For the purpose of implementing `has_tracked_dir` (which means "has tracked
descendants) without an expensive sub-tree traversal, we maintaing a counter
of tracked descendants on each "directory" node of the tree-shaped dirstate.

Before this changeset, mutating or inserting a node at a given path would
involve:

* Walking the tree from root through ancestors to find the node or the spot
  where to insert it
* Looking at the previous node if any to decide what counter update is needed
* Performing any node mutation
* Walking the tree *again* to update counters in ancestor nodes

When profiling `hg status` on a large repo, this second walk takes times
while loading a the dirstate from disk.

It turns out we have enough information to decide before he first tree walk
what counter update is needed. This changeset merges the two walks, gaining
~10% of the total time for `hg update` (in the same hyperfine benchmark as
the previous changeset).

---

Profiling was done by compiling with this `.cargo/config`:

    [profile.release]
    debug = true

then running with:

    py-spy record -r 500 -n -o /tmp/hg.json --format speedscope -- \
    ./hg status -R $REPO --config experimental.dirstate-tree.in-memory=1

then visualizing the recorded JSON file in https://www.speedscope.app/

Differential Revision: https://phab.mercurial-scm.org/D10554

#require no-windows

ATTENTION: logtoprocess runs commands asynchronously. Be sure to append "| cat"
to hg commands, to wait for the output, if you want to test its output.
Otherwise the test will be flaky.

Test if logtoprocess correctly captures command-related log calls.

  $ hg init
  $ cat > $TESTTMP/foocommand.py << EOF
  > from __future__ import absolute_import
  > from mercurial import registrar
  > cmdtable = {}
  > command = registrar.command(cmdtable)
  > configtable = {}
  > configitem = registrar.configitem(configtable)
  > configitem(b'logtoprocess', b'foo',
  >     default=None,
  > )
  > @command(b'foobar', [])
  > def foo(ui, repo):
  >     ui.log(b'foo', b'a message: %s\n', b'spam')
  > EOF
  $ cp $HGRCPATH $HGRCPATH.bak
  $ cat >> $HGRCPATH << EOF
  > [extensions]
  > logtoprocess=
  > foocommand=$TESTTMP/foocommand.py
  > [logtoprocess]
  > command=(echo 'logtoprocess command output:';
  >     echo "\$EVENT";
  >     echo "\$MSG1") > $TESTTMP/command.log
  > commandfinish=(echo 'logtoprocess commandfinish output:';
  >     echo "\$EVENT";
  >     echo "\$MSG1";
  >     echo "canonical: \$OPT_CANONICAL_COMMAND") > $TESTTMP/commandfinish.log
  > foo=(echo 'logtoprocess foo output:';
  >     echo "\$EVENT";
  >     echo "\$MSG1") > $TESTTMP/foo.log
  > EOF

Running a command triggers both a ui.log('command') and a
ui.log('commandfinish') call. The foo command also uses ui.log.

Use sort to avoid ordering issues between the various processes we spawn:
  $ hg fooba
  $ sleep 1
  $ cat $TESTTMP/command.log | sort
  
  command
  fooba
  logtoprocess command output:

#if no-chg
  $ cat $TESTTMP/commandfinish.log | sort
  
  canonical: foobar
  commandfinish
  fooba exited 0 after * seconds (glob)
  logtoprocess commandfinish output:
  $ cat $TESTTMP/foo.log | sort
  
  a message: spam
  foo
  logtoprocess foo output:
#endif

Confirm that logging blocked time catches stdio properly:
  $ cp $HGRCPATH.bak $HGRCPATH
  $ cat >> $HGRCPATH << EOF
  > [extensions]
  > logtoprocess=
  > pager=
  > [logtoprocess]
  > uiblocked=echo "\$EVENT stdio \$OPT_STDIO_BLOCKED ms command \$OPT_COMMAND_DURATION ms" > $TESTTMP/uiblocked.log
  > [ui]
  > logblockedtimes=True
  > EOF

  $ hg log
  $ sleep 1
  $ cat $TESTTMP/uiblocked.log
  uiblocked stdio [0-9]+.[0-9]* ms command [0-9]+.[0-9]* ms (re)

Try to confirm that pager wait on logtoprocess:

Add a script that wait on a file to appears for 5 seconds, if it sees it touch
another file or die after 5 seconds. If the scripts is awaited by hg, the
script will die after the timeout before we could touch the file and the
resulting file will not exists. If not, we will touch the file and see it.

  $ cat >> fakepager.py <<EOF
  > import sys
  > printed = False
  > for line in sys.stdin:
  >     sys.stdout.write(line)
  >     printed = True
  > if not printed:
  >     sys.stdout.write('paged empty output!\n')
  > EOF

  $ cat > $TESTTMP/wait-output.sh << EOF
  > #!/bin/sh
  > for i in \`$TESTDIR/seq.py 50\`; do
  >   if [ -f "$TESTTMP/wait-for-touched" ];
  >   then
  >     touch "$TESTTMP/touched";
  >     break;
  >   else
  >     sleep 0.1;
  >   fi
  > done
  > EOF
  $ chmod +x $TESTTMP/wait-output.sh

  $ cat >> $HGRCPATH << EOF
  > [extensions]
  > logtoprocess=
  > pager=
  > [pager]
  > pager = "$PYTHON" $TESTTMP/fakepager.py
  > [logtoprocess]
  > commandfinish=$TESTTMP/wait-output.sh
  > EOF
  $ hg version -q --pager=always
  Mercurial Distributed SCM (version *) (glob)
  $ touch $TESTTMP/wait-for-touched
  $ sleep 0.2
  $ test -f $TESTTMP/touched && echo "SUCCESS Pager is not waiting on ltp" || echo "FAIL Pager is waiting on ltp"
  SUCCESS Pager is not waiting on ltp