view tests/test-fix-pickle.t @ 47120:7109a38830c9

dirstate-tree: Fold "tracked descendants" counter update in main walk

For the purpose of implementing `has_tracked_dir` (which means "has tracked
descendants) without an expensive sub-tree traversal, we maintaing a counter
of tracked descendants on each "directory" node of the tree-shaped dirstate.

Before this changeset, mutating or inserting a node at a given path would
involve:

* Walking the tree from root through ancestors to find the node or the spot
  where to insert it
* Looking at the previous node if any to decide what counter update is needed
* Performing any node mutation
* Walking the tree *again* to update counters in ancestor nodes

When profiling `hg status` on a large repo, this second walk takes times
while loading a the dirstate from disk.

It turns out we have enough information to decide before he first tree walk
what counter update is needed. This changeset merges the two walks, gaining
~10% of the total time for `hg update` (in the same hyperfine benchmark as
the previous changeset).

---

Profiling was done by compiling with this `.cargo/config`:

    [profile.release]
    debug = true

then running with:

    py-spy record -r 500 -n -o /tmp/hg.json --format speedscope -- \
    ./hg status -R $REPO --config experimental.dirstate-tree.in-memory=1

then visualizing the recorded JSON file in https://www.speedscope.app/

Differential Revision: https://phab.mercurial-scm.org/D10554

A script that implements uppercasing all letters in a file.

  $ UPPERCASEPY="$TESTTMP/uppercase.py"
  $ cat > $UPPERCASEPY <<EOF
  > import sys
  > from mercurial.utils.procutil import setbinary
  > setbinary(sys.stdin)
  > setbinary(sys.stdout)
  > sys.stdout.write(sys.stdin.read().upper())
  > EOF
  $ TESTLINES="foo\nbar\nbaz\n"
  $ printf $TESTLINES | "$PYTHON" $UPPERCASEPY
  FOO
  BAR
  BAZ

This file attempts to test our workarounds for pickle's lack of
support for short reads.

  $ cat >> $HGRCPATH <<EOF
  > [extensions]
  > fix =
  > [fix]
  > uppercase-whole-file:command="$PYTHON" $UPPERCASEPY
  > uppercase-whole-file:pattern=set:**
  > EOF

  $ hg init repo
  $ cd repo

# Create a file that's large enough that it seems to not fit in
# pickle's buffer, making it use the code path that expects our
# _blockingreader's read() method to return bytes.
  $ echo "some stuff" > file
  $ for i in $($TESTDIR/seq.py 13); do
  >   cat file file > tmp
  >   mv -f tmp file
  > done
  $ hg commit -Am "add large file"
  adding file

Check that we don't get a crash

  $ hg fix -r .
  saved backup bundle to $TESTTMP/repo/.hg/strip-backup/*-fix.hg (glob)