tests/test-config-env.py
author Pierre-Yves David <pierre-yves.david@octobus.net>
Tue, 11 Feb 2020 11:18:52 +0100
changeset 44363 f7459da77f23
parent 44032 2d4cad94d08a
child 48875 6000f5b25c9b
permissions -rw-r--r--
nodemap: introduce an option to use mmap to read the nodemap mapping The performance and memory benefit is much greater if we don't have to copy all the data in memory for each information. So we introduce an option (on by default) to read the data using mmap. This changeset is the last one definition the API for index support nodemap data. (they have to be able to use the mmaping). Below are some benchmark comparing the best we currently have in 5.3 with the final step of this series (using the persistent nodemap implementation in Rust). The benchmark run `hg perfindex` with various revset and the following variants: Before: * do not use the persistent nodemap * use the CPython implementation of the index for nodemap * use mmapping of the changelog index After: * use the MixedIndex Rust code, with the NodeTree object for nodemap access (still in review) * use the persistent nodemap data from disk * access the persistent nodemap data through mmap * use mmapping of the changelog index The persistent nodemap greatly speed up most operation on very large repositories. Some of the previously very fast lookup end up a bit slower because the persistent nodemap has to be setup. However the absolute slowdown is very small and won't matters in the big picture. Here are some numbers (in seconds) for the reference copy of mozilla-try: Revset Before After abs-change speedup -10000: 0.004622 0.005532 0.000910 × 0.83 -10: 0.000050 0.000132 0.000082 × 0.37 tip 0.000052 0.000085 0.000033 × 0.61 0 + (-10000:) 0.028222 0.005337 -0.022885 × 5.29 0 0.023521 0.000084 -0.023437 × 280.01 (-10000:) + 0 0.235539 0.005308 -0.230231 × 44.37 (-10:) + :9 0.232883 0.000180 -0.232703 ×1293.79 (-10000:) + (:99) 0.238735 0.005358 -0.233377 × 44.55 :99 + (-10000:) 0.317942 0.005593 -0.312349 × 56.84 :9 + (-10:) 0.313372 0.000179 -0.313193 ×1750.68 :9 0.316450 0.000143 -0.316307 ×2212.93 On smaller repositories, the cost of nodemap related operation is not as big, so the win is much more modest. Yet it helps shaving a handful of millisecond here and there. Here are some numbers (in seconds) for the reference copy of mercurial: Revset Before After abs-change speedup -10: 0.000065 0.000097 0.000032 × 0.67 tip 0.000063 0.000078 0.000015 × 0.80 0 0.000561 0.000079 -0.000482 × 7.10 -10000: 0.004609 0.003648 -0.000961 × 1.26 0 + (-10000:) 0.005023 0.003715 -0.001307 × 1.35 (-10:) + :9 0.002187 0.000108 -0.002079 ×20.25 (-10000:) + 0 0.006252 0.003716 -0.002536 × 1.68 (-10000:) + (:99) 0.006367 0.003707 -0.002660 × 1.71 :9 + (-10:) 0.003846 0.000110 -0.003736 ×34.96 :9 0.003854 0.000099 -0.003755 ×38.92 :99 + (-10000:) 0.007644 0.003778 -0.003866 × 2.02 Differential Revision: https://phab.mercurial-scm.org/D7894
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
31685
d83e51654c8a rcutil: let environ override system configs (BC)
Jun Wu <quark@fb.com>
parents:
diff changeset
     1
# Test the config layer generated by environment variables
d83e51654c8a rcutil: let environ override system configs (BC)
Jun Wu <quark@fb.com>
parents:
diff changeset
     2
d83e51654c8a rcutil: let environ override system configs (BC)
Jun Wu <quark@fb.com>
parents:
diff changeset
     3
from __future__ import absolute_import, print_function
d83e51654c8a rcutil: let environ override system configs (BC)
Jun Wu <quark@fb.com>
parents:
diff changeset
     4
d83e51654c8a rcutil: let environ override system configs (BC)
Jun Wu <quark@fb.com>
parents:
diff changeset
     5
import os
d83e51654c8a rcutil: let environ override system configs (BC)
Jun Wu <quark@fb.com>
parents:
diff changeset
     6
d83e51654c8a rcutil: let environ override system configs (BC)
Jun Wu <quark@fb.com>
parents:
diff changeset
     7
from mercurial import (
d83e51654c8a rcutil: let environ override system configs (BC)
Jun Wu <quark@fb.com>
parents:
diff changeset
     8
    encoding,
43872
527eba3013ea tests: make test-config-env.py a little less hacky
Martin von Zweigbergk <martinvonz@google.com>
parents: 43076
diff changeset
     9
    extensions,
31685
d83e51654c8a rcutil: let environ override system configs (BC)
Jun Wu <quark@fb.com>
parents:
diff changeset
    10
    rcutil,
d83e51654c8a rcutil: let environ override system configs (BC)
Jun Wu <quark@fb.com>
parents:
diff changeset
    11
    ui as uimod,
31857
08fbc97d1364 tests: print Unix style paths in *.py tests
Matt Harbison <matt_harbison@yahoo.com>
parents: 31685
diff changeset
    12
    util,
31685
d83e51654c8a rcutil: let environ override system configs (BC)
Jun Wu <quark@fb.com>
parents:
diff changeset
    13
)
d83e51654c8a rcutil: let environ override system configs (BC)
Jun Wu <quark@fb.com>
parents:
diff changeset
    14
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 37119
diff changeset
    15
from mercurial.utils import procutil
37119
d4a2e0d5d042 procutil: bulk-replace util.std* to point to new module
Yuya Nishihara <yuya@tcha.org>
parents: 36730
diff changeset
    16
36730
a22915edc279 py3: byte-stringify test-config.t and test-config-env.py
Yuya Nishihara <yuya@tcha.org>
parents: 31857
diff changeset
    17
testtmp = encoding.environ[b'TESTTMP']
31685
d83e51654c8a rcutil: let environ override system configs (BC)
Jun Wu <quark@fb.com>
parents:
diff changeset
    18
d83e51654c8a rcutil: let environ override system configs (BC)
Jun Wu <quark@fb.com>
parents:
diff changeset
    19
# prepare hgrc files
d83e51654c8a rcutil: let environ override system configs (BC)
Jun Wu <quark@fb.com>
parents:
diff changeset
    20
def join(name):
d83e51654c8a rcutil: let environ override system configs (BC)
Jun Wu <quark@fb.com>
parents:
diff changeset
    21
    return os.path.join(testtmp, name)
d83e51654c8a rcutil: let environ override system configs (BC)
Jun Wu <quark@fb.com>
parents:
diff changeset
    22
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 37119
diff changeset
    23
36730
a22915edc279 py3: byte-stringify test-config.t and test-config-env.py
Yuya Nishihara <yuya@tcha.org>
parents: 31857
diff changeset
    24
with open(join(b'sysrc'), 'wb') as f:
a22915edc279 py3: byte-stringify test-config.t and test-config-env.py
Yuya Nishihara <yuya@tcha.org>
parents: 31857
diff changeset
    25
    f.write(b'[ui]\neditor=e0\n[pager]\npager=p0\n')
31685
d83e51654c8a rcutil: let environ override system configs (BC)
Jun Wu <quark@fb.com>
parents:
diff changeset
    26
36730
a22915edc279 py3: byte-stringify test-config.t and test-config-env.py
Yuya Nishihara <yuya@tcha.org>
parents: 31857
diff changeset
    27
with open(join(b'userrc'), 'wb') as f:
a22915edc279 py3: byte-stringify test-config.t and test-config-env.py
Yuya Nishihara <yuya@tcha.org>
parents: 31857
diff changeset
    28
    f.write(b'[ui]\neditor=e1')
31685
d83e51654c8a rcutil: let environ override system configs (BC)
Jun Wu <quark@fb.com>
parents:
diff changeset
    29
d83e51654c8a rcutil: let environ override system configs (BC)
Jun Wu <quark@fb.com>
parents:
diff changeset
    30
# replace rcpath functions so they point to the files above
d83e51654c8a rcutil: let environ override system configs (BC)
Jun Wu <quark@fb.com>
parents:
diff changeset
    31
def systemrcpath():
36730
a22915edc279 py3: byte-stringify test-config.t and test-config-env.py
Yuya Nishihara <yuya@tcha.org>
parents: 31857
diff changeset
    32
    return [join(b'sysrc')]
31685
d83e51654c8a rcutil: let environ override system configs (BC)
Jun Wu <quark@fb.com>
parents:
diff changeset
    33
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 37119
diff changeset
    34
31685
d83e51654c8a rcutil: let environ override system configs (BC)
Jun Wu <quark@fb.com>
parents:
diff changeset
    35
def userrcpath():
36730
a22915edc279 py3: byte-stringify test-config.t and test-config-env.py
Yuya Nishihara <yuya@tcha.org>
parents: 31857
diff changeset
    36
    return [join(b'userrc')]
31685
d83e51654c8a rcutil: let environ override system configs (BC)
Jun Wu <quark@fb.com>
parents:
diff changeset
    37
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 37119
diff changeset
    38
44031
1864efbe90d9 ui: add the ability to apply `defaultrc` configs from resources
Matt Harbison <matt_harbison@yahoo.com>
parents: 43872
diff changeset
    39
extensions.wrapfunction(rcutil, 'default_rc_resources', lambda orig: [])
43872
527eba3013ea tests: make test-config-env.py a little less hacky
Martin von Zweigbergk <martinvonz@google.com>
parents: 43076
diff changeset
    40
31685
d83e51654c8a rcutil: let environ override system configs (BC)
Jun Wu <quark@fb.com>
parents:
diff changeset
    41
rcutil.systemrcpath = systemrcpath
d83e51654c8a rcutil: let environ override system configs (BC)
Jun Wu <quark@fb.com>
parents:
diff changeset
    42
rcutil.userrcpath = userrcpath
d83e51654c8a rcutil: let environ override system configs (BC)
Jun Wu <quark@fb.com>
parents:
diff changeset
    43
d83e51654c8a rcutil: let environ override system configs (BC)
Jun Wu <quark@fb.com>
parents:
diff changeset
    44
# utility to print configs
d83e51654c8a rcutil: let environ override system configs (BC)
Jun Wu <quark@fb.com>
parents:
diff changeset
    45
def printconfigs(env):
d83e51654c8a rcutil: let environ override system configs (BC)
Jun Wu <quark@fb.com>
parents:
diff changeset
    46
    encoding.environ = env
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 37119
diff changeset
    47
    rcutil._rccomponents = None  # reset cache
31685
d83e51654c8a rcutil: let environ override system configs (BC)
Jun Wu <quark@fb.com>
parents:
diff changeset
    48
    ui = uimod.ui.load()
d83e51654c8a rcutil: let environ override system configs (BC)
Jun Wu <quark@fb.com>
parents:
diff changeset
    49
    for section, name, value in ui.walkconfig():
d83e51654c8a rcutil: let environ override system configs (BC)
Jun Wu <quark@fb.com>
parents:
diff changeset
    50
        source = ui.configsource(section, name)
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 37119
diff changeset
    51
        procutil.stdout.write(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 37119
diff changeset
    52
            b'%s.%s=%s # %s\n' % (section, name, value, util.pconvert(source))
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 37119
diff changeset
    53
        )
37119
d4a2e0d5d042 procutil: bulk-replace util.std* to point to new module
Yuya Nishihara <yuya@tcha.org>
parents: 36730
diff changeset
    54
    procutil.stdout.write(b'\n')
31685
d83e51654c8a rcutil: let environ override system configs (BC)
Jun Wu <quark@fb.com>
parents:
diff changeset
    55
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 37119
diff changeset
    56
31685
d83e51654c8a rcutil: let environ override system configs (BC)
Jun Wu <quark@fb.com>
parents:
diff changeset
    57
# environment variable overrides
d83e51654c8a rcutil: let environ override system configs (BC)
Jun Wu <quark@fb.com>
parents:
diff changeset
    58
printconfigs({})
36730
a22915edc279 py3: byte-stringify test-config.t and test-config-env.py
Yuya Nishihara <yuya@tcha.org>
parents: 31857
diff changeset
    59
printconfigs({b'EDITOR': b'e2', b'PAGER': b'p2'})