view tests/test-status-terse.t @ 38732:be4984261611

merge: mark file gets as not thread safe (issue5933)

In default installs, this has the effect of disabling the thread-based
worker on Windows when manifesting files in the working directory. My
measurements have shown that with revlog-based repositories, Mercurial
spends a lot of CPU time in revlog code resolving file data. This ends
up incurring a lot of context switching across threads and slows down
`hg update` operations when going from an empty working directory to
the tip of the repo.

On mozilla-unified (246,351 files) on an i7-6700K (4+4 CPUs):

before: 487s wall
after:  360s wall (equivalent to worker.enabled=false)
cpus=2: 379s wall

Even with only 2 threads, the thread pool is still slower.

The introduction of the thread-based worker (02b36e860e0b) states that
it resulted in a "~50%" speedup for `hg sparse --enable-profile` and
`hg sparse --disable-profile`. This disagrees with my measurement
above. I theorize a few reasons for this:

1) Removal of files from the working directory is I/O - not CPU - bound
   and should benefit from a thread pool (unless I/O is insanely fast
   and the GIL release is near instantaneous). So tests like `hg sparse
   --enable-profile` may exercise deletion throughput and aren't good
   benchmarks for worker tasks that are CPU heavy.
2) The patch was authored by someone at Facebook. The results were
   likely measured against a repository using remotefilelog. And I
   believe that revision retrieval during working directory updates with
   remotefilelog will often use a remote store, thus being I/O and not
   CPU bound. This probably resulted in an overstated performance gain.

Since there appears to be a need to enable the thread-based worker with
some stores, I've made the flagging of file gets as thread safe
configurable. I've made it experimental because I don't want to formalize
a boolean flag for this option and because this attribute is best
captured against the store implementation. But we don't have a proper
store API for this yet. I'd rather cross this bridge later.

It is possible there are revlog-based repositories that do benefit from
a thread-based worker. I didn't do very comprehensive testing. If there
are, we may want to devise a more proper algorithm for whether to use
the thread-based worker, including possibly config options to limit the
number of threads to use. But until I see evidence that justifies
complexity, simplicity wins.

Differential Revision: https://phab.mercurial-scm.org/D3963

  $ mkdir folder
  $ cd folder
  $ hg init
  $ mkdir x x/l x/m x/n x/l/u x/l/u/a
  $ touch a b x/aa.o x/bb.o
  $ hg status
  ? a
  ? b
  ? x/aa.o
  ? x/bb.o

  $ hg status --terse u
  ? a
  ? b
  ? x/
  $ hg status --terse maudric
  ? a
  ? b
  ? x/
  $ hg status --terse madric
  ? a
  ? b
  ? x/aa.o
  ? x/bb.o
  $ hg status --terse f
  abort: 'f' not recognized
  [255]

Add a .hgignore so that we can also have ignored files

  $ echo ".*\.o" > .hgignore
  $ hg status
  ? .hgignore
  ? a
  ? b
  $ hg status -i
  I x/aa.o
  I x/bb.o

Tersing ignored files
  $ hg status -t i --ignored
  I x/

Adding more files
  $ mkdir y
  $ touch x/aa x/bb y/l y/m y/l.o y/m.o
  $ touch x/l/aa x/m/aa x/n/aa x/l/u/bb x/l/u/a/bb

  $ hg status
  ? .hgignore
  ? a
  ? b
  ? x/aa
  ? x/bb
  ? x/l/aa
  ? x/l/u/a/bb
  ? x/l/u/bb
  ? x/m/aa
  ? x/n/aa
  ? y/l
  ? y/m

  $ hg status --terse u
  ? .hgignore
  ? a
  ? b
  ? x/
  ? y/

Run from subdirectory
  $ hg status --terse u --cwd x/l
  ? .hgignore
  ? a
  ? b
  ? x/
  ? y/
  $ relstatus() {
  >   hg status --terse u --config commands.status.relative=1 "$@";
  > }
This should probably have {"l/", "m/", "n/"} instead of {"."}. They should
probably come after "../y/".
  $ relstatus --cwd x
  ? ../.hgignore
  ? ../a
  ? ../b
  ? .
  ? ../y/
This should probably have {"u/", "../m/", "../n/"} instead of {"../"}.
  $ relstatus --cwd x/l
  ? ../../.hgignore
  ? ../../a
  ? ../../b
  ? ../
  ? ../../y/
This should probably have {"a/", "bb", "../aa", "../../m/", "../../n/"}
instead of {"../../"}.
  $ relstatus --cwd x/l/u
  ? ../../../.hgignore
  ? ../../../a
  ? ../../../b
  ? ../../
  ? ../../../y/
This should probably have {"bb", "../bb", "../../aa", "../../../m/",
"../../../n/"} instead of {"../../../"}.
  $ relstatus --cwd x/l/u/a
  ? ../../../../.hgignore
  ? ../../../../a
  ? ../../../../b
  ? ../../../
  ? ../../../../y/

  $ hg add x/aa x/bb .hgignore
  $ hg status --terse au
  A .hgignore
  A x/aa
  A x/bb
  ? a
  ? b
  ? x/l/
  ? x/m/
  ? x/n/
  ? y/

Including ignored files

  $ hg status --terse aui
  A .hgignore
  A x/aa
  A x/bb
  ? a
  ? b
  ? x/l/
  ? x/m/
  ? x/n/
  ? y/l
  ? y/m
  $ hg status --terse au -i
  I x/aa.o
  I x/bb.o
  I y/l.o
  I y/m.o

Committing some of the files

  $ hg commit x/aa x/bb .hgignore -m "First commit"
  $ hg status
  ? a
  ? b
  ? x/l/aa
  ? x/l/u/a/bb
  ? x/l/u/bb
  ? x/m/aa
  ? x/n/aa
  ? y/l
  ? y/m
  $ hg status --terse mardu
  ? a
  ? b
  ? x/l/
  ? x/m/
  ? x/n/
  ? y/

Modifying already committed files

  $ echo "Hello" >> x/aa
  $ echo "World" >> x/bb
  $ hg status --terse maurdc
  M x/aa
  M x/bb
  ? a
  ? b
  ? x/l/
  ? x/m/
  ? x/n/
  ? y/

Respecting other flags

  $ hg status --terse marduic --all
  M x/aa
  M x/bb
  ? a
  ? b
  ? x/l/
  ? x/m/
  ? x/n/
  ? y/l
  ? y/m
  I x/aa.o
  I x/bb.o
  I y/l.o
  I y/m.o
  C .hgignore
  $ hg status --terse marduic -a
  $ hg status --terse marduic -c
  C .hgignore
  $ hg status --terse marduic -m
  M x/aa
  M x/bb

Passing 'i' in terse value will consider the ignored files while tersing

  $ hg status --terse marduic -u
  ? a
  ? b
  ? x/l/
  ? x/m/
  ? x/n/
  ? y/l
  ? y/m

Omitting 'i' in terse value does not consider ignored files while tersing

  $ hg status --terse marduc -u
  ? a
  ? b
  ? x/l/
  ? x/m/
  ? x/n/
  ? y/

Trying with --rev

  $ hg status --terse marduic --rev 0 --rev 1
  abort: cannot use --terse with --rev
  [255]

Config item to set the default terseness
  $ cat <<EOF >> $HGRCPATH
  > [commands]
  > status.terse = u
  > EOF
  $ hg status -mu
  M x/aa
  M x/bb
  ? a
  ? b
  ? x/l/
  ? x/m/
  ? x/n/
  ? y/

Command line flag overrides the default
  $ hg status --terse=
  M x/aa
  M x/bb
  ? a
  ? b
  ? x/l/aa
  ? x/l/u/a/bb
  ? x/l/u/bb
  ? x/m/aa
  ? x/n/aa
  ? y/l
  ? y/m
  $ hg status --terse=mardu
  M x/aa
  M x/bb
  ? a
  ? b
  ? x/l/
  ? x/m/
  ? x/n/
  ? y/

Specifying --rev should still work, with the terseness disabled.
  $ hg status --rev 0
  M x/aa
  M x/bb
  ? a
  ? b
  ? x/l/aa
  ? x/l/u/a/bb
  ? x/l/u/bb
  ? x/m/aa
  ? x/n/aa
  ? y/l
  ? y/m