view tests/test-patch-offset.t @ 44363:f7459da77f23

nodemap: introduce an option to use mmap to read the nodemap mapping

The performance and memory benefit is much greater if we don't have to copy all
the data in memory for each information. So we introduce an option (on by
default) to read the data using mmap.

This changeset is the last one definition the API for index support nodemap
data. (they have to be able to use the mmaping).

Below are some benchmark comparing the best we currently have in 5.3 with the
final step of this series (using the persistent nodemap implementation in
Rust). The benchmark run `hg perfindex` with various revset and the following
variants:

Before:

 * do not use the persistent nodemap
 * use the CPython implementation of the index for nodemap
 * use mmapping of the changelog index

After:

 * use the MixedIndex Rust code, with the NodeTree object for nodemap access
   (still in review)
 * use the persistent nodemap data from disk
 * access the persistent nodemap data through mmap
 * use mmapping of the changelog index

The persistent nodemap greatly speed up  most operation on very large
repositories. Some of the previously very fast lookup end up a bit slower because
the persistent nodemap has to be setup. However the absolute slowdown is very
small and won't matters in the big picture.

Here are some numbers (in seconds) for the reference copy of mozilla-try:

Revset            Before   After     abs-change speedup

-10000:           0.004622 0.005532   0.000910  ×   0.83
-10:              0.000050 0.000132   0.000082  ×   0.37
tip               0.000052 0.000085   0.000033  ×   0.61
0 + (-10000:)     0.028222 0.005337  -0.022885  ×   5.29
0                 0.023521 0.000084  -0.023437  × 280.01
(-10000:) + 0     0.235539 0.005308  -0.230231  ×  44.37
(-10:) + :9       0.232883 0.000180  -0.232703  ×1293.79
(-10000:) + (:99) 0.238735 0.005358  -0.233377  ×  44.55
:99 + (-10000:)   0.317942 0.005593  -0.312349  ×  56.84
:9 + (-10:)       0.313372 0.000179  -0.313193  ×1750.68
:9                0.316450 0.000143  -0.316307  ×2212.93


On smaller repositories, the cost of nodemap related operation is not as big, so
the win is much more modest. Yet it helps shaving a handful of millisecond here
and there.

Here are some numbers (in seconds) for the reference copy of mercurial:

Revset            Before   After     abs-change speedup
-10:              0.000065 0.000097   0.000032  × 0.67
tip               0.000063 0.000078   0.000015  × 0.80
0                 0.000561 0.000079  -0.000482  × 7.10
-10000:           0.004609 0.003648  -0.000961  × 1.26
0 + (-10000:)     0.005023 0.003715  -0.001307  × 1.35
(-10:) + :9       0.002187 0.000108  -0.002079  ×20.25
(-10000:) + 0     0.006252 0.003716  -0.002536  × 1.68
(-10000:) + (:99) 0.006367 0.003707  -0.002660  × 1.71
:9 + (-10:)       0.003846 0.000110  -0.003736  ×34.96
:9                0.003854 0.000099  -0.003755  ×38.92
:99 + (-10000:)   0.007644 0.003778  -0.003866  × 2.02

Differential Revision: https://phab.mercurial-scm.org/D7894

  $ cat > writepatterns.py <<EOF
  > import sys
  > 
  > path = sys.argv[1]
  > patterns = sys.argv[2:]
  > 
  > fp = open(path, 'wb')
  > for pattern in patterns:
  >     count = int(pattern[0:-1])
  >     char = pattern[-1].encode('utf8') + b'\n'
  >     fp.write(char * count)
  > fp.close()
  > EOF

prepare repo

  $ hg init a
  $ cd a

These initial lines of Xs were not in the original file used to generate
the patch.  So all the patch hunks need to be applied to a constant offset
within this file.  If the offset isn't tracked then the hunks can be
applied to the wrong lines of this file.

  $ "$PYTHON" ../writepatterns.py a 34X 10A 1B 10A 1C 10A 1B 10A 1D 10A 1B 10A 1E 10A 1B 10A
  $ hg commit -Am adda
  adding a

This is a cleaner patch generated via diff
In this case it reproduces the problem when
the output of hg export does not
import patch

  $ hg import -v -m 'b' -d '2 0' - <<EOF
  > --- a/a	2009-12-08 19:26:17.000000000 -0800
  > +++ b/a	2009-12-08 19:26:17.000000000 -0800
  > @@ -9,7 +9,7 @@
  >  A
  >  A
  >  B
  > -A
  > +a
  >  A
  >  A
  >  A
  > @@ -53,7 +53,7 @@
  >  A
  >  A
  >  B
  > -A
  > +a
  >  A
  >  A
  >  A
  > @@ -75,7 +75,7 @@
  >  A
  >  A
  >  B
  > -A
  > +a
  >  A
  >  A
  >  A
  > EOF
  applying patch from stdin
  patching file a
  Hunk #1 succeeded at 43 (offset 34 lines).
  Hunk #2 succeeded at 87 (offset 34 lines).
  Hunk #3 succeeded at 109 (offset 34 lines).
  committing files:
  a
  committing manifest
  committing changelog
  created 189885cecb41

compare imported changes against reference file

  $ "$PYTHON" ../writepatterns.py aref 34X 10A 1B 1a 9A 1C 10A 1B 10A 1D 10A 1B 1a 9A 1E 10A 1B 1a 9A
  $ diff aref a

  $ cd ..