view tests/test-rhg.t @ 51681:522b4d729e89

mmap: populate the mapping by default

Without pre-population, accessing all data through a mmap can result in many
pagefault, reducing performance significantly. If the mmap is prepopulated, the
performance can no longer get slower than a full read.
(See benchmark number below)

In some cases were very few data is read, prepopulating can be overkill and
slower than populating on access (through page fault). So that behavior can be
controlled when the caller can pre-determine the best behavior.
(See benchmark number below)

In addition, testing with populating in a secondary thread yield great result
combining the best of each approach. This might be implemented in later
changesets.

In all cases, using mmap has a great effect on memory usage when many processes
run in parallel on the same machine.

### Benchmarks

# What did I run

A couple of month back I ran a large benchmark campaign to assess the impact of
various approach for using mmap with the revlog (and other files), it
highlighted a few benchmarks that capture the impact of the changes well. So to
validate this change I checked the following:

- log command displaying various revisions
  (read the changelog index)
- log command displaying the patch of listed revisions
  (read the changelog index, the manifest index and a few files indexes)
- unbundling a few revisions
  (read and write changelog, manifest and few files indexes, and walk the graph
  to update some cache)
- pushing a few revisions
  (read and write changelog, manifest and few files indexes, walk the graph to
  update some cache, performs various accesses locally and remotely during
  discovery)

Benchmarks were run using the default module policy (c+py) and the rust one. No
significant difference were found between the two implementation, so we will
present result using the default policy (unless otherwise specified).

I ran them on a few repositories :

- mercurial: a "public changeset only" copy of mercurial from 2018-08-01 using
  zstd compression and sparse-revlog

- pypy: a copy of pypy from 2018-08-01 using zstd compression and sparse-revlog

- netbeans: a copy of netbeans from 2018-08-01 using zstd compression and
  sparse-revlog

- mozilla-try: a copy of mozilla-try from 2019-02-18 using zstd compression and
  sparse-revlog

- mozilla-try persistent-nodemap: Same as the above but with a persistent
  nodemap. Used for the log --patch benchmark only

# Results

For the smaller repositories (mercurial, pypy), the impact of mmap is almost
imperceptible, other cost dominating the operation. The impact of prepopulating
is undiscernible in the benchmark we ran.

For larger repositories the benchmark support explanation given above:

On netbeans, the log can be about 1% faster without repopulation (for a
difference < 100ms) but unbundle becomes a bit slower, even when small.

### data-env-vars.name                             = netbeans-2018-08-01-zstd-sparse-revlog
  # benchmark.name                                 = hg.command.unbundle
  # benchmark.variants.issue6528                   = disabled
  # benchmark.variants.reuse-external-delta-parent = yes
  # benchmark.variants.revs                        = any-1-extra-rev
  # benchmark.variants.source                      = unbundle
  # benchmark.variants.verbosity                   = quiet
with-populate: 0.240157
no-populate:   0.265087  (+10.38%, +0.02)
  # benchmark.variants.revs                        = any-100-extra-rev
with-populate: 1.459518
no-populate:   1.481290   (+1.49%, +0.02)
 ## benchmark.name                                 = hg.command.push
  # benchmark.variants.explicit-rev                = none
  # benchmark.variants.issue6528                   = disabled
  # benchmark.variants.protocol                    = ssh
  # benchmark.variants.reuse-external-delta-parent = yes
  # benchmark.variants.revs                        = any-1-extra-rev
with-populate: 0.771919
no-populate:   0.792025   (+2.60%, +0.02)
  # benchmark.variants.revs                        = any-100-extra-rev
with-populate: 1.459518
no-populate:   1.481290   (+1.49%, +0.02)


For mozilla-try, the "slow down" from pre-populate for small `hg log` is more
visible, but still small in absolute time. (using rust value for the persistent
nodemap value to be relevant).

### data-env-vars.name            = mozilla-try-2019-02-18-ds2-pnm
  # benchmark.name                = hg.command.log
  # bin-env-vars.hg.flavor        = rust
  # benchmark.variants.patch      = yes
  # benchmark.variants.limit-rev  = 1
with-populate: 0.237813
no-populate:   0.229452   (-3.52%, -0.01)
  # benchmark.variants.limit-rev  = 10
  # benchmark.variants.patch      = yes
with-populate: 1.213578
no-populate:   1.205189
### data-env-vars.name            = mozilla-try-2019-02-18-zstd-sparse-revlog
  # benchmark.variants.limit-rev  = 1000
  # benchmark.variants.patch      = no
  # benchmark.variants.rev        = tip
with-populate: 0.198607
no-populate:   0.195038   (-1.80%, -0.00)

However pre-populating provide a significant boost on more complex operations
like unbundle or push:

### data-env-vars.name                             = mozilla-try-2019-02-18-zstd-sparse-revlog
  # benchmark.name                                 = hg.command.push
  # benchmark.variants.explicit-rev                = none
  # benchmark.variants.issue6528                   = disabled
  # benchmark.variants.protocol                    = ssh
  # benchmark.variants.reuse-external-delta-parent = yes
  # benchmark.variants.revs                        = any-1-extra-rev
with-populate: 4.798632
no-populate:   4.953295   (+3.22%, +0.15)
  # benchmark.variants.revs                        = any-100-extra-rev
with-populate: 4.903618
no-populate:   5.014963   (+2.27%, +0.11)
 ## benchmark.name                                 = hg.command.unbundle
  # benchmark.variants.revs                        = any-1-extra-rev
with-populate: 1.423411
no-populate:   1.585365  (+11.38%, +0.16)
  # benchmark.variants.revs                        = any-100-extra-rev
with-populate: 1.537909
no-populate:   1.688489   (+9.79%, +0.15)

#require rhg

  $ NO_FALLBACK="env RHG_ON_UNSUPPORTED=abort"

Unimplemented command
  $ $NO_FALLBACK rhg unimplemented-command
  unsupported feature: error: The subcommand 'unimplemented-command' wasn't recognized
  
  Usage: rhg [OPTIONS] <COMMAND>
  
  For more information try '--help'
  
  [252]
  $ rhg unimplemented-command --config rhg.on-unsupported=abort-silent
  [252]

Finding root
  $ $NO_FALLBACK rhg root
  abort: no repository found in '$TESTTMP' (.hg not found)!
  [255]

  $ hg init repository
  $ cd repository
  $ $NO_FALLBACK rhg root
  $TESTTMP/repository

Reading and setting configuration
  $ echo "[ui]" >> $HGRCPATH
  $ echo "username = user1" >> $HGRCPATH
  $ echo "[extensions]" >> $HGRCPATH
  $ echo "sparse =" >> $HGRCPATH
  $ $NO_FALLBACK rhg config ui.username
  user1
  $ echo "[ui]" >> .hg/hgrc
  $ echo "username = user2" >> .hg/hgrc
  $ $NO_FALLBACK rhg config ui.username
  user2
  $ $NO_FALLBACK rhg --config ui.username=user3 config ui.username
  user3

Unwritable file descriptor
  $ $NO_FALLBACK rhg root > /dev/full
  abort: No space left on device (os error 28)
  [255]

Deleted repository
  $ rm -rf `pwd`
  $ $NO_FALLBACK rhg root
  abort: error getting current working directory: $ENOENT$
  [255]

Listing tracked files
  $ cd $TESTTMP
  $ hg init repository
  $ cd repository
  $ for i in 1 2 3; do
  >   echo $i >> file$i
  >   hg add file$i
  > done
  > hg commit -m "commit $i" -q

Listing tracked files from root
  $ $NO_FALLBACK rhg files
  file1
  file2
  file3

Listing tracked files from subdirectory
  $ mkdir -p path/to/directory
  $ cd path/to/directory
  $ $NO_FALLBACK rhg files
  ../../../file1
  ../../../file2
  ../../../file3

  $ $NO_FALLBACK rhg files --config ui.relative-paths=legacy
  ../../../file1
  ../../../file2
  ../../../file3

  $ $NO_FALLBACK rhg files --config ui.relative-paths=false
  file1
  file2
  file3

  $ $NO_FALLBACK rhg files --config ui.relative-paths=true
  ../../../file1
  ../../../file2
  ../../../file3

Listing tracked files through broken pipe
  $ $NO_FALLBACK rhg files | head -n 1
  ../../../file1

Debuging data in inline index
  $ cd $TESTTMP
  $ rm -rf repository
  $ hg init repository
  $ cd repository
  $ for i in 1 2 3 4 5 6; do
  >   echo $i >> file-$i
  >   hg add file-$i
  >   hg commit -m "Commit $i" -q
  > done
  $ $NO_FALLBACK rhg debugdata -c 2
  8d0267cb034247ebfa5ee58ce59e22e57a492297
  test
  0 0
  file-3
  
  Commit 3 (no-eol)
  $ $NO_FALLBACK rhg debugdata -m 2
  file-1\x00b8e02f6433738021a065f94175c7cd23db5f05be (esc)
  file-2\x005d9299349fc01ddd25d0070d149b124d8f10411e (esc)
  file-3\x002661d26c649684b482d10f91960cc3db683c38b4 (esc)

Debuging with full node id
  $ $NO_FALLBACK rhg debugdata -c `hg log -r 0 -T '{node}'`
  d1d1c679d3053e8926061b6f45ca52009f011e3f
  test
  0 0
  file-1
  
  Commit 1 (no-eol)

Specifying revisions by changeset ID
  $ hg log -T '{node}\n'
  c6ad58c44207b6ff8a4fbbca7045a5edaa7e908b
  d654274993d0149eecc3cc03214f598320211900
  f646af7e96481d3a5470b695cf30ad8e3ab6c575
  cf8b83f14ead62b374b6e91a0e9303b85dfd9ed7
  91c6f6e73e39318534dc415ea4e8a09c99cd74d6
  6ae9681c6d30389694d8701faf24b583cf3ccafe
  $ $NO_FALLBACK rhg files -r cf8b83
  file-1
  file-2
  file-3
  $ $NO_FALLBACK rhg cat -r cf8b83 file-2
  2
  $ $NO_FALLBACK rhg cat --rev cf8b83 file-2
  2
  $ $NO_FALLBACK rhg cat -r c file-2
  abort: ambiguous revision identifier: c
  [255]
  $ $NO_FALLBACK rhg cat -r d file-2
  2
  $ $NO_FALLBACK rhg cat -r 0000 file-2
  file-2: no such file in rev 000000000000
  [1]

Cat files
  $ cd $TESTTMP
  $ rm -rf repository
  $ hg init repository
  $ cd repository
  $ echo "original content" > original
  $ hg add original
  $ hg commit -m "add original" original
Without `--rev`
  $ $NO_FALLBACK rhg cat original
  original content
With `--rev`
  $ $NO_FALLBACK rhg cat -r 0 original
  original content
Cat copied file should not display copy metadata
  $ hg copy original copy_of_original
  $ hg commit -m "add copy of original"
  $ $NO_FALLBACK rhg cat original
  original content
  $ $NO_FALLBACK rhg cat -r 1 copy_of_original
  original content


Fallback to Python
  $ $NO_FALLBACK rhg cat original --exclude="*.rs"
  unsupported feature: error: Found argument '--exclude' which wasn't expected, or isn't valid in this context
  
    If you tried to supply '--exclude' as a value rather than a flag, use '-- --exclude'
  
  Usage: rhg cat <FILE>...
  
  For more information try '--help'
  
  [252]
  $ rhg cat original --exclude="*.rs"
  original content

Check that `fallback-immediately` overrides `$NO_FALLBACK`
  $ $NO_FALLBACK rhg cat original --exclude="*.rs" --config rhg.fallback-immediately=1
  original content

  $ (unset RHG_FALLBACK_EXECUTABLE; rhg cat original --exclude="*.rs")
  abort: 'rhg.on-unsupported=fallback' without 'rhg.fallback-executable' set.
  [255]

  $ (unset RHG_FALLBACK_EXECUTABLE; rhg cat original)
  original content

  $ rhg cat original --exclude="*.rs" --config rhg.fallback-executable=false
  [1]

  $ rhg cat original --exclude="*.rs" --config rhg.fallback-executable=hg-non-existent
  abort: invalid fallback 'hg-non-existent': cannot find binary path
  [253]

  $ rhg cat original --exclude="*.rs" --config rhg.fallback-executable=rhg
  Blocking recursive fallback. The 'rhg.fallback-executable = rhg' config points to `rhg` itself.
  unsupported feature: error: Found argument '--exclude' which wasn't expected, or isn't valid in this context
  
    If you tried to supply '--exclude' as a value rather than a flag, use '-- --exclude'
  
  Usage: rhg cat <FILE>...
  
  For more information try '--help'
  
  [252]

Fallback with shell path segments
  $ $NO_FALLBACK rhg cat .
  unsupported feature: `..` or `.` path segment
  [252]
  $ $NO_FALLBACK rhg cat ..
  unsupported feature: `..` or `.` path segment
  [252]
  $ $NO_FALLBACK rhg cat ../..
  unsupported feature: `..` or `.` path segment
  [252]

Fallback with filesets
  $ $NO_FALLBACK rhg cat "set:c or b"
  unsupported feature: fileset
  [252]

Fallback with generic hooks
  $ $NO_FALLBACK rhg cat original --config hooks.pre-cat=something
  unsupported feature: pre-cat hook defined
  [252]

  $ $NO_FALLBACK rhg cat original --config hooks.post-cat=something
  unsupported feature: post-cat hook defined
  [252]

  $ $NO_FALLBACK rhg cat original --config hooks.fail-cat=something
  unsupported feature: fail-cat hook defined
  [252]

Fallback with [defaults]
  $ $NO_FALLBACK rhg cat original --config "defaults.cat=-r null"
  unsupported feature: `defaults` config set
  [252]


Requirements
  $ $NO_FALLBACK rhg debugrequirements
  dotencode
  fncache
  generaldelta
  persistent-nodemap
  revlog-compression-zstd (zstd !)
  revlogv1
  share-safe
  sparserevlog
  store

  $ echo indoor-pool >> .hg/requires
  $ $NO_FALLBACK rhg files
  unsupported feature: repository requires feature unknown to this Mercurial: indoor-pool
  [252]

  $ $NO_FALLBACK rhg cat -r 1 copy_of_original
  unsupported feature: repository requires feature unknown to this Mercurial: indoor-pool
  [252]

  $ $NO_FALLBACK rhg debugrequirements
  unsupported feature: repository requires feature unknown to this Mercurial: indoor-pool
  [252]

  $ echo -e '\xFF' >> .hg/requires
  $ $NO_FALLBACK rhg debugrequirements
  abort: parse error in 'requires' file
  [255]

Persistent nodemap
  $ cd $TESTTMP
  $ rm -rf repository
  $ hg --config format.use-persistent-nodemap=no init repository
  $ cd repository
  $ $NO_FALLBACK rhg debugrequirements | grep nodemap
  [1]
  $ hg debugbuilddag .+5000 --overwritten-file --config "storage.revlog.nodemap.mode=warn"
  $ hg id -r tip
  c3ae8dec9fad tip
  $ ls .hg/store/00changelog*
  .hg/store/00changelog.d
  .hg/store/00changelog.i
  $ $NO_FALLBACK rhg files -r c3ae8dec9fad
  of

  $ cd $TESTTMP
  $ rm -rf repository
  $ hg --config format.use-persistent-nodemap=True init repository
  $ cd repository
  $ $NO_FALLBACK rhg debugrequirements | grep nodemap
  persistent-nodemap
  $ hg debugbuilddag .+5000 --overwritten-file --config "storage.revlog.nodemap.mode=warn"
  $ hg id -r tip
  c3ae8dec9fad tip
  $ ls .hg/store/00changelog*
  .hg/store/00changelog-*.nd (glob)
  .hg/store/00changelog.d
  .hg/store/00changelog.i
  .hg/store/00changelog.n

Rhg status on a sparse repo with nodemap (this specific combination used to crash in 6.5.2)

  $ hg debugsparse -X excluded-dir
  $ $NO_FALLBACK rhg status

Specifying revisions by changeset ID
  $ $NO_FALLBACK rhg files -r c3ae8dec9fad
  of
  $ $NO_FALLBACK rhg cat -r c3ae8dec9fad of
  r5000

Crate a shared repository

  $ echo "[extensions]"      >> $HGRCPATH
  $ echo "share = "          >> $HGRCPATH

  $ cd $TESTTMP
  $ hg init repo1
  $ echo a > repo1/a
  $ hg -R repo1 commit -A -m'init'
  adding a

  $ hg share repo1 repo2
  updating working directory
  1 files updated, 0 files merged, 0 files removed, 0 files unresolved

And check that basic rhg commands work with sharing

  $ $NO_FALLBACK rhg files -R repo2
  repo2/a
  $ $NO_FALLBACK rhg -R repo2 cat -r 0 repo2/a
  a

Same with relative sharing

  $ hg share repo2 repo3 --relative
  updating working directory
  1 files updated, 0 files merged, 0 files removed, 0 files unresolved

  $ $NO_FALLBACK rhg files -R repo3
  repo3/a
  $ $NO_FALLBACK rhg -R repo3 cat -r 0 repo3/a
  a

Same with share-safe

  $ echo "[format]"         >> $HGRCPATH
  $ echo "use-share-safe = True" >> $HGRCPATH

  $ cd $TESTTMP
  $ hg init repo4
  $ cd repo4
  $ echo a > a
  $ hg commit -A -m'init'
  adding a

  $ cd ..
  $ hg share repo4 repo5
  updating working directory
  1 files updated, 0 files merged, 0 files removed, 0 files unresolved

And check that basic rhg commands work with sharing

  $ cd repo5
  $ $NO_FALLBACK rhg files
  a
  $ $NO_FALLBACK rhg cat -r 0 a
  a

The blackbox extension is supported

  $ echo "[extensions]" >> $HGRCPATH
  $ echo "blackbox =" >> $HGRCPATH
  $ echo "[blackbox]" >> $HGRCPATH
  $ echo "maxsize = 1" >> $HGRCPATH
  $ $NO_FALLBACK rhg files > /dev/null
  $ cat .hg/blackbox.log
  ????-??-?? ??:??:??.??? * @d3873e73d99ef67873dac33fbcc66268d5d2b6f4 (*)> (rust) files exited 0 after * seconds (glob)
  $ cat .hg/blackbox.log.1
  ????-??-?? ??:??:??.??? * @d3873e73d99ef67873dac33fbcc66268d5d2b6f4 (*)> (rust) files (glob)

Subrepos are not supported

  $ touch .hgsub
  $ $NO_FALLBACK rhg files
  unsupported feature: subrepos (.hgsub is present)
  [252]
  $ rhg files
  a
  $ rm .hgsub

The `:required` extension suboptions are correctly ignored

  $ echo "[extensions]" >> $HGRCPATH
  $ echo "blackbox:required = yes" >> $HGRCPATH
  $ rhg files
  a
  $ echo "*:required = yes" >> $HGRCPATH
  $ rhg files
  a

We can ignore all extensions at once

  $ echo "[extensions]" >> $HGRCPATH
  $ echo "thisextensionbetternotexist=" >> $HGRCPATH
  $ echo "thisextensionbetternotexisteither=" >> $HGRCPATH
  $ $NO_FALLBACK rhg files
  unsupported feature: extensions: thisextensionbetternotexist, thisextensionbetternotexisteither (consider adding them to 'rhg.ignored-extensions' config)
  [252]

  $ echo "[rhg]" >> $HGRCPATH
  $ echo "ignored-extensions=*" >> $HGRCPATH
  $ $NO_FALLBACK rhg files
  a