view tests/test-ui-verbosity.py.out @ 44363:f7459da77f23

nodemap: introduce an option to use mmap to read the nodemap mapping

The performance and memory benefit is much greater if we don't have to copy all
the data in memory for each information. So we introduce an option (on by
default) to read the data using mmap.

This changeset is the last one definition the API for index support nodemap
data. (they have to be able to use the mmaping).

Below are some benchmark comparing the best we currently have in 5.3 with the
final step of this series (using the persistent nodemap implementation in
Rust). The benchmark run `hg perfindex` with various revset and the following
variants:

Before:

 * do not use the persistent nodemap
 * use the CPython implementation of the index for nodemap
 * use mmapping of the changelog index

After:

 * use the MixedIndex Rust code, with the NodeTree object for nodemap access
   (still in review)
 * use the persistent nodemap data from disk
 * access the persistent nodemap data through mmap
 * use mmapping of the changelog index

The persistent nodemap greatly speed up  most operation on very large
repositories. Some of the previously very fast lookup end up a bit slower because
the persistent nodemap has to be setup. However the absolute slowdown is very
small and won't matters in the big picture.

Here are some numbers (in seconds) for the reference copy of mozilla-try:

Revset            Before   After     abs-change speedup

-10000:           0.004622 0.005532   0.000910  ×   0.83
-10:              0.000050 0.000132   0.000082  ×   0.37
tip               0.000052 0.000085   0.000033  ×   0.61
0 + (-10000:)     0.028222 0.005337  -0.022885  ×   5.29
0                 0.023521 0.000084  -0.023437  × 280.01
(-10000:) + 0     0.235539 0.005308  -0.230231  ×  44.37
(-10:) + :9       0.232883 0.000180  -0.232703  ×1293.79
(-10000:) + (:99) 0.238735 0.005358  -0.233377  ×  44.55
:99 + (-10000:)   0.317942 0.005593  -0.312349  ×  56.84
:9 + (-10:)       0.313372 0.000179  -0.313193  ×1750.68
:9                0.316450 0.000143  -0.316307  ×2212.93


On smaller repositories, the cost of nodemap related operation is not as big, so
the win is much more modest. Yet it helps shaving a handful of millisecond here
and there.

Here are some numbers (in seconds) for the reference copy of mercurial:

Revset            Before   After     abs-change speedup
-10:              0.000065 0.000097   0.000032  × 0.67
tip               0.000063 0.000078   0.000015  × 0.80
0                 0.000561 0.000079  -0.000482  × 7.10
-10000:           0.004609 0.003648  -0.000961  × 1.26
0 + (-10000:)     0.005023 0.003715  -0.001307  × 1.35
(-10:) + :9       0.002187 0.000108  -0.002079  ×20.25
(-10000:) + 0     0.006252 0.003716  -0.002536  × 1.68
(-10000:) + (:99) 0.006367 0.003707  -0.002660  × 1.71
:9 + (-10:)       0.003846 0.000110  -0.003736  ×34.96
:9                0.003854 0.000099  -0.003755  ×38.92
:99 + (-10000:)   0.007644 0.003778  -0.003866  × 2.02

Differential Revision: https://phab.mercurial-scm.org/D7894

      hgrc settings    command line options      final result   
    quiet verbo debug   quiet verbo debug      quiet verbo debug
 0  False False False   False False False  ->  False False False
 1   True False False   False False False  ->   True False False
 2  False  True False   False False False  ->  False  True False
 3   True  True False   False False False  ->  False False False
 4  False False  True   False False False  ->  False  True  True
 5   True False  True   False False False  ->  False  True  True
 6  False  True  True   False False False  ->  False  True  True
 7   True  True  True   False False False  ->  False  True  True
 8  False False False    True False False  ->   True False False
 9   True False False    True False False  ->   True False False
10  False  True False    True False False  ->   True False False
11   True  True False    True False False  ->   True False False
12  False False  True    True False False  ->   True False False
13   True False  True    True False False  ->   True False False
14  False  True  True    True False False  ->   True False False
15   True  True  True    True False False  ->   True False False
16  False False False   False  True False  ->  False  True False
17   True False False   False  True False  ->  False  True False
18  False  True False   False  True False  ->  False  True False
19   True  True False   False  True False  ->  False  True False
20  False False  True   False  True False  ->  False  True False
21   True False  True   False  True False  ->  False  True False
22  False  True  True   False  True False  ->  False  True False
23   True  True  True   False  True False  ->  False  True False
24  False False False    True  True False  ->  False False False
25   True False False    True  True False  ->  False False False
26  False  True False    True  True False  ->  False False False
27   True  True False    True  True False  ->  False False False
28  False False  True    True  True False  ->  False False False
29   True False  True    True  True False  ->  False False False
30  False  True  True    True  True False  ->  False False False
31   True  True  True    True  True False  ->  False False False
32  False False False   False False  True  ->  False  True  True
33   True False False   False False  True  ->  False  True  True
34  False  True False   False False  True  ->  False  True  True
35   True  True False   False False  True  ->  False  True  True
36  False False  True   False False  True  ->  False  True  True
37   True False  True   False False  True  ->  False  True  True
38  False  True  True   False False  True  ->  False  True  True
39   True  True  True   False False  True  ->  False  True  True
40  False False False    True False  True  ->  False  True  True
41   True False False    True False  True  ->  False  True  True
42  False  True False    True False  True  ->  False  True  True
43   True  True False    True False  True  ->  False  True  True
44  False False  True    True False  True  ->  False  True  True
45   True False  True    True False  True  ->  False  True  True
46  False  True  True    True False  True  ->  False  True  True
47   True  True  True    True False  True  ->  False  True  True
48  False False False   False  True  True  ->  False  True  True
49   True False False   False  True  True  ->  False  True  True
50  False  True False   False  True  True  ->  False  True  True
51   True  True False   False  True  True  ->  False  True  True
52  False False  True   False  True  True  ->  False  True  True
53   True False  True   False  True  True  ->  False  True  True
54  False  True  True   False  True  True  ->  False  True  True
55   True  True  True   False  True  True  ->  False  True  True
56  False False False    True  True  True  ->  False  True  True
57   True False False    True  True  True  ->  False  True  True
58  False  True False    True  True  True  ->  False  True  True
59   True  True False    True  True  True  ->  False  True  True
60  False False  True    True  True  True  ->  False  True  True
61   True False  True    True  True  True  ->  False  True  True
62  False  True  True    True  True  True  ->  False  True  True
63   True  True  True    True  True  True  ->  False  True  True