view tests/test-import-context.t @ 24461:05ccfe6763f1

osutil: use getdirentriesattr on OS X if possible

This is a significant win for large repositories on OS X, especially with a
cold cache. Unfortunately we need to keep the lstat-based implementation around
for two reasons:

- Not all filesystems support this call.
- There's an edge case in which it's best to fall back to avoid a retry loop.
  More about this in the comments.

The below tests are all performed on a Mac with an SSD running OS X 10.9, on a
repository with over 200k files. The results are best of 5 with simulated
best-effort conditions.

The gains with a hot cache are pretty impressive: 'hg status' goes from 5.18
seconds to 3.79 seconds.

However, a repository that large will probably already be using something like
hgwatchman [1], which helps much more (for this repo, 'hg status' with
hgwatchman is approximately 1 second). Where this really helps is when the
cache is cold [2]: hg status goes from 31.0 seconds to 9.66.

See http://lists.apple.com/archives/filesystem-dev/2014/Dec/msg00002.html for
some more discussion about this function.

This is based on a patch by Sean Farley <[e-mail]>.

[1] https://bitbucket.org/facebook/hgwatchman

[2] There appears to be no easy way to clear the file cache (aka "vnodes") on
OS X short of rebooting. purge(8) purportedly does that but in my testing had
little effect. The workaround I came up with was to assume that vnode eviction
was LRU, make sure the kern.maxvnodes sysctl is smaller than the size of the
repository, then make sure we'd always miss the cache by running 'hg status' in
another clone of the repository before running it in the test repository.

Test applying context diffs

  $ cat > writepatterns.py <<EOF
  > import sys
  > 
  > path = sys.argv[1]
  > lasteol = sys.argv[2] == '1'
  > patterns = sys.argv[3:]
  > 
  > fp = file(path, 'wb')
  > for i, pattern in enumerate(patterns):
  >     count = int(pattern[0:-1])
  >     char = pattern[-1] + '\n'
  >     if not lasteol and i == len(patterns) - 1:
  >         fp.write((char*count)[:-1])
  >     else:
  >         fp.write(char*count)
  > fp.close()
  > EOF
  $ cat > cat.py <<EOF
  > import sys
  > sys.stdout.write(repr(file(sys.argv[1], 'rb').read()) + '\n')
  > EOF

Initialize the test repository

  $ hg init repo
  $ cd repo
  $ python ../writepatterns.py a 0 5A 1B 5C 1D
  $ python ../writepatterns.py b 1 1A 1B
  $ python ../writepatterns.py c 1 5A
  $ python ../writepatterns.py d 1 5A 1B
  $ hg add
  adding a
  adding b
  adding c
  adding d
  $ hg ci -m addfiles

Add file, missing a last end of line

  $ hg import --no-commit - <<EOF
  > *** /dev/null	2010-10-16 18:05:49.000000000 +0200
  > --- b/newnoeol	2010-10-16 18:23:26.000000000 +0200
  > ***************
  > *** 0 ****
  > --- 1,2 ----
  > + a
  > + b
  > \ No newline at end of file
  > *** a/a	Sat Oct 16 16:35:51 2010
  > --- b/a	Sat Oct 16 16:35:51 2010
  > ***************
  > *** 3,12 ****
  >   A
  >   A
  >   A
  > ! B
  >   C
  >   C
  >   C
  >   C
  >   C
  > ! D
  > \ No newline at end of file
  > --- 3,13 ----
  >   A
  >   A
  >   A
  > ! E
  >   C
  >   C
  >   C
  >   C
  >   C
  > ! F
  > ! F
  > 
  > *** a/b	2010-10-16 18:40:38.000000000 +0200
  > --- /dev/null	2010-10-16 18:05:49.000000000 +0200
  > ***************
  > *** 1,2 ****
  > - A
  > - B
  > --- 0 ----
  > *** a/c	Sat Oct 16 21:34:26 2010
  > --- b/c	Sat Oct 16 21:34:27 2010
  > ***************
  > *** 3,5 ****
  > --- 3,7 ----
  >   A
  >   A
  >   A
  > + B
  > + B
  > *** a/d	Sat Oct 16 21:47:20 2010
  > --- b/d	Sat Oct 16 21:47:22 2010
  > ***************
  > *** 2,6 ****
  >   A
  >   A
  >   A
  > - A
  > - B
  > --- 2,4 ----
  > EOF
  applying patch from stdin
  $ hg st
  M a
  M c
  M d
  A newnoeol
  R b

What's in a

  $ python ../cat.py a
  'A\nA\nA\nA\nA\nE\nC\nC\nC\nC\nC\nF\nF\n'
  $ python ../cat.py newnoeol
  'a\nb'
  $ python ../cat.py c
  'A\nA\nA\nA\nA\nB\nB\n'
  $ python ../cat.py d
  'A\nA\nA\nA\n'

  $ cd ..