annotate tests/test-revlog-mmapindex.t @ 34296:3c9691728237

revlog: add option to mmap revlog index Following on from Jun Wu's patch last October[1], we have found that using mmap for the revlog index in repos with large revlogs gives a noticable performance improvment (~110ms on each hg invocation), particularly for commands that don't touch the index very much. This changeset adds this as an option, activated by a new experimental config option so that it can be enabled on a per-repo basis. The configuration option specifies an index size threshold at which Mercurial will switch to using mmap to access the index. If the configuration option is not specified, the default remains to load the full file, which seems to be the best option for smaller repos. Some initial performance numbers for average of 5 invocations of `hg log -l 5` for different cache states: | Repo: | HG | FB | |---|---|---| | Index size: | 2.3MB | much bigger | | read (warm): | 237ms | 432ms | | mmap (warm): | 227ms | 321ms | | | (-3%) | (-26%) | | read (cold): | 397ms | 696ms | | mmap (cold): | 410ms | 888ms | | | (+3%) | (+28%) | [1] https://www.mercurial-scm.org/pipermail/mercurial-devel/2016-October/088737.html Test Plan: `hg log --config experimental.mmapindex=true` Differential Revision: https://phab.mercurial-scm.org/D477
author Mark Thomas <mbthomas@fb.com>
date Wed, 13 Sep 2017 17:26:26 +0000
parents
children b0c97e44576f
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
34296
3c9691728237 revlog: add option to mmap revlog index
Mark Thomas <mbthomas@fb.com>
parents:
diff changeset
1 create verbosemmap.py
3c9691728237 revlog: add option to mmap revlog index
Mark Thomas <mbthomas@fb.com>
parents:
diff changeset
2 $ cat << EOF > verbosemmap.py
3c9691728237 revlog: add option to mmap revlog index
Mark Thomas <mbthomas@fb.com>
parents:
diff changeset
3 > # extension to make util.mmapread verbose
3c9691728237 revlog: add option to mmap revlog index
Mark Thomas <mbthomas@fb.com>
parents:
diff changeset
4 >
3c9691728237 revlog: add option to mmap revlog index
Mark Thomas <mbthomas@fb.com>
parents:
diff changeset
5 > from __future__ import absolute_import
3c9691728237 revlog: add option to mmap revlog index
Mark Thomas <mbthomas@fb.com>
parents:
diff changeset
6 >
3c9691728237 revlog: add option to mmap revlog index
Mark Thomas <mbthomas@fb.com>
parents:
diff changeset
7 > from mercurial import (
3c9691728237 revlog: add option to mmap revlog index
Mark Thomas <mbthomas@fb.com>
parents:
diff changeset
8 > extensions,
3c9691728237 revlog: add option to mmap revlog index
Mark Thomas <mbthomas@fb.com>
parents:
diff changeset
9 > util,
3c9691728237 revlog: add option to mmap revlog index
Mark Thomas <mbthomas@fb.com>
parents:
diff changeset
10 > )
3c9691728237 revlog: add option to mmap revlog index
Mark Thomas <mbthomas@fb.com>
parents:
diff changeset
11 >
3c9691728237 revlog: add option to mmap revlog index
Mark Thomas <mbthomas@fb.com>
parents:
diff changeset
12 > def mmapread(orig, fp):
3c9691728237 revlog: add option to mmap revlog index
Mark Thomas <mbthomas@fb.com>
parents:
diff changeset
13 > print "mmapping %s" % fp.name
3c9691728237 revlog: add option to mmap revlog index
Mark Thomas <mbthomas@fb.com>
parents:
diff changeset
14 > return orig(fp)
3c9691728237 revlog: add option to mmap revlog index
Mark Thomas <mbthomas@fb.com>
parents:
diff changeset
15 >
3c9691728237 revlog: add option to mmap revlog index
Mark Thomas <mbthomas@fb.com>
parents:
diff changeset
16 > def extsetup(ui):
3c9691728237 revlog: add option to mmap revlog index
Mark Thomas <mbthomas@fb.com>
parents:
diff changeset
17 > extensions.wrapfunction(util, 'mmapread', mmapread)
3c9691728237 revlog: add option to mmap revlog index
Mark Thomas <mbthomas@fb.com>
parents:
diff changeset
18 > EOF
3c9691728237 revlog: add option to mmap revlog index
Mark Thomas <mbthomas@fb.com>
parents:
diff changeset
19
3c9691728237 revlog: add option to mmap revlog index
Mark Thomas <mbthomas@fb.com>
parents:
diff changeset
20 setting up base repo
3c9691728237 revlog: add option to mmap revlog index
Mark Thomas <mbthomas@fb.com>
parents:
diff changeset
21 $ hg init a
3c9691728237 revlog: add option to mmap revlog index
Mark Thomas <mbthomas@fb.com>
parents:
diff changeset
22 $ cd a
3c9691728237 revlog: add option to mmap revlog index
Mark Thomas <mbthomas@fb.com>
parents:
diff changeset
23 $ touch a
3c9691728237 revlog: add option to mmap revlog index
Mark Thomas <mbthomas@fb.com>
parents:
diff changeset
24 $ hg add a
3c9691728237 revlog: add option to mmap revlog index
Mark Thomas <mbthomas@fb.com>
parents:
diff changeset
25 $ hg commit -qm base
3c9691728237 revlog: add option to mmap revlog index
Mark Thomas <mbthomas@fb.com>
parents:
diff changeset
26 $ for i in `$TESTDIR/seq.py 1 100` ; do
3c9691728237 revlog: add option to mmap revlog index
Mark Thomas <mbthomas@fb.com>
parents:
diff changeset
27 > echo $i > a
3c9691728237 revlog: add option to mmap revlog index
Mark Thomas <mbthomas@fb.com>
parents:
diff changeset
28 > hg commit -qm $i
3c9691728237 revlog: add option to mmap revlog index
Mark Thomas <mbthomas@fb.com>
parents:
diff changeset
29 > done
3c9691728237 revlog: add option to mmap revlog index
Mark Thomas <mbthomas@fb.com>
parents:
diff changeset
30
3c9691728237 revlog: add option to mmap revlog index
Mark Thomas <mbthomas@fb.com>
parents:
diff changeset
31 set up verbosemmap extension
3c9691728237 revlog: add option to mmap revlog index
Mark Thomas <mbthomas@fb.com>
parents:
diff changeset
32 $ cat << EOF >> $HGRCPATH
3c9691728237 revlog: add option to mmap revlog index
Mark Thomas <mbthomas@fb.com>
parents:
diff changeset
33 > [extensions]
3c9691728237 revlog: add option to mmap revlog index
Mark Thomas <mbthomas@fb.com>
parents:
diff changeset
34 > verbosemmap=$TESTTMP/verbosemmap.py
3c9691728237 revlog: add option to mmap revlog index
Mark Thomas <mbthomas@fb.com>
parents:
diff changeset
35 > EOF
3c9691728237 revlog: add option to mmap revlog index
Mark Thomas <mbthomas@fb.com>
parents:
diff changeset
36
3c9691728237 revlog: add option to mmap revlog index
Mark Thomas <mbthomas@fb.com>
parents:
diff changeset
37 mmap index which is now more than 4k long
3c9691728237 revlog: add option to mmap revlog index
Mark Thomas <mbthomas@fb.com>
parents:
diff changeset
38 $ hg log -l 5 -T '{rev}\n' --config experimental.mmapindexthreshold=4k
3c9691728237 revlog: add option to mmap revlog index
Mark Thomas <mbthomas@fb.com>
parents:
diff changeset
39 mmapping $TESTTMP/a/.hg/store/00changelog.i (glob)
3c9691728237 revlog: add option to mmap revlog index
Mark Thomas <mbthomas@fb.com>
parents:
diff changeset
40 100
3c9691728237 revlog: add option to mmap revlog index
Mark Thomas <mbthomas@fb.com>
parents:
diff changeset
41 99
3c9691728237 revlog: add option to mmap revlog index
Mark Thomas <mbthomas@fb.com>
parents:
diff changeset
42 98
3c9691728237 revlog: add option to mmap revlog index
Mark Thomas <mbthomas@fb.com>
parents:
diff changeset
43 97
3c9691728237 revlog: add option to mmap revlog index
Mark Thomas <mbthomas@fb.com>
parents:
diff changeset
44 96
3c9691728237 revlog: add option to mmap revlog index
Mark Thomas <mbthomas@fb.com>
parents:
diff changeset
45
3c9691728237 revlog: add option to mmap revlog index
Mark Thomas <mbthomas@fb.com>
parents:
diff changeset
46 do not mmap index which is still less than 32k
3c9691728237 revlog: add option to mmap revlog index
Mark Thomas <mbthomas@fb.com>
parents:
diff changeset
47 $ hg log -l 5 -T '{rev}\n' --config experimental.mmapindexthreshold=32k
3c9691728237 revlog: add option to mmap revlog index
Mark Thomas <mbthomas@fb.com>
parents:
diff changeset
48 100
3c9691728237 revlog: add option to mmap revlog index
Mark Thomas <mbthomas@fb.com>
parents:
diff changeset
49 99
3c9691728237 revlog: add option to mmap revlog index
Mark Thomas <mbthomas@fb.com>
parents:
diff changeset
50 98
3c9691728237 revlog: add option to mmap revlog index
Mark Thomas <mbthomas@fb.com>
parents:
diff changeset
51 97
3c9691728237 revlog: add option to mmap revlog index
Mark Thomas <mbthomas@fb.com>
parents:
diff changeset
52 96
3c9691728237 revlog: add option to mmap revlog index
Mark Thomas <mbthomas@fb.com>
parents:
diff changeset
53
3c9691728237 revlog: add option to mmap revlog index
Mark Thomas <mbthomas@fb.com>
parents:
diff changeset
54 $ cd ..