Mercurial > hg
annotate tests/test-doctest.py @ 40021:c537144fdbef
wireprotov2: support response caching
One of the things I've learned from managing VCS servers over the
years is that they are hard to scale. It is well known that some
companies have very beefy (read: very expensive) servers to power
their VCS needs. It is also known that specialized servers for
various VCS exist in order to facilitate scaling servers. (Mercurial
is in this boat.)
One of the aspects that make a VCS server hard to scale is the
high CPU load incurred by constant client clone/pull operations.
To alleviate the scaling pain associated with data retrieval
operations, I want to integrate caching into the Mercurial wire
protocol server as robustly as possible such that servers can
aggressively cache responses and defer as much server load as
possible.
This commit represents the initial implementation of a general
caching layer in wire protocol version 2.
We define a new interface and behavior for a wire protocol cacher
in repository.py. (This is probably where a reviewer should look
first to understand what is going on.)
The bulk of the added code is in wireprotov2server.py, where we
define how a command can opt in to being cached and integrate
caching into command dispatching.
From a very high-level:
* A command can declare itself as cacheable by providing a callable
that can be used to derive a cache key.
* At dispatch time, if a command is cacheable, we attempt to
construct a cacher and use it for serving the request and/or
caching the request.
* The dispatch layer handles the bulk of the business logic for
caching, making cachers mostly "dumb content stores."
* The mechanism for invalidating cached entries (one of the harder
parts about caching in general) is by varying the cache key when
state changes. As such, cachers don't need to be concerned with
cache invalidation.
Initially, we've hooked up support for caching "manifestdata" and
"filedata" commands. These are the simplest to cache, as they should
be immutable over time. Caching of commands related to changeset
data is a bit harder (because cache validation is impacted by
changes to bookmarks, phases, etc). This will be implemented later.
(Strictly speaking, censoring a file should invalidate caches. I've
added an inline TODO to track this edge case.)
To prove it works, this commit implements a test-only extension
providing in-memory caching backed by an lrucachedict. A new test
showing this extension behaving properly is added. FWIW, the
cacher is ~50 lines of code, demonstrating the relative ease with
which a cache can be added to a server.
While the test cacher is not suitable for production workloads, just
for kicks I performed a clone of just the changeset and manifest data
for the mozilla-unified repository. With a fully warmed cache (of just
the manifest data since changeset data is not cached), server-side
CPU usage dropped from ~73s to ~28s. That's pretty significant and
demonstrates the potential that response caching has on server
scalability!
Differential Revision: https://phab.mercurial-scm.org/D4773
author | Gregory Szorc <gregory.szorc@gmail.com> |
---|---|
date | Wed, 26 Sep 2018 17:16:56 -0700 |
parents | 740f7d447222 |
children | a3183ca7ce8f |
rev | line source |
---|---|
7041
b856071435f7
tests: fix readline escape characters in output for test-doctest.py
Mads Kiilerich <mads@kiilerich.com>
parents:
5525
diff
changeset
|
1 # this is hack to make sure no escape characters are inserted into the output |
28933
6262f0215d08
tests: make test-doctest use absolute_import
Pulkit Goyal <7895pulkit@gmail.com>
parents:
27432
diff
changeset
|
2 |
6262f0215d08
tests: make test-doctest use absolute_import
Pulkit Goyal <7895pulkit@gmail.com>
parents:
27432
diff
changeset
|
3 from __future__ import absolute_import |
6262f0215d08
tests: make test-doctest use absolute_import
Pulkit Goyal <7895pulkit@gmail.com>
parents:
27432
diff
changeset
|
4 |
6262f0215d08
tests: make test-doctest use absolute_import
Pulkit Goyal <7895pulkit@gmail.com>
parents:
27432
diff
changeset
|
5 import doctest |
6262f0215d08
tests: make test-doctest use absolute_import
Pulkit Goyal <7895pulkit@gmail.com>
parents:
27432
diff
changeset
|
6 import os |
34140
52ec9ac0303b
doctest: normalize b'', u'' and exception output on Python 3
Yuya Nishihara <yuya@tcha.org>
parents:
32485
diff
changeset
|
7 import re |
28933
6262f0215d08
tests: make test-doctest use absolute_import
Pulkit Goyal <7895pulkit@gmail.com>
parents:
27432
diff
changeset
|
8 import sys |
31438
82350f7fa56c
tests: allow running doctests selectively on Python 3
Yuya Nishihara <yuya@tcha.org>
parents:
31024
diff
changeset
|
9 |
82350f7fa56c
tests: allow running doctests selectively on Python 3
Yuya Nishihara <yuya@tcha.org>
parents:
31024
diff
changeset
|
10 ispy3 = (sys.version_info[0] >= 3) |
82350f7fa56c
tests: allow running doctests selectively on Python 3
Yuya Nishihara <yuya@tcha.org>
parents:
31024
diff
changeset
|
11 |
7078
967adcf5910d
test-doctest: remove TERM env variable only if it's there
Patrick Mezard <pmezard@gmail.com>
parents:
7041
diff
changeset
|
12 if 'TERM' in os.environ: |
7184
380fda3eed13
clean up trailing spaces
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
7078
diff
changeset
|
13 del os.environ['TERM'] |
3232
394ac87f3b74
[extendedchangelog] encode/decode function
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents:
diff
changeset
|
14 |
34140
52ec9ac0303b
doctest: normalize b'', u'' and exception output on Python 3
Yuya Nishihara <yuya@tcha.org>
parents:
32485
diff
changeset
|
15 class py3docchecker(doctest.OutputChecker): |
52ec9ac0303b
doctest: normalize b'', u'' and exception output on Python 3
Yuya Nishihara <yuya@tcha.org>
parents:
32485
diff
changeset
|
16 def check_output(self, want, got, optionflags): |
52ec9ac0303b
doctest: normalize b'', u'' and exception output on Python 3
Yuya Nishihara <yuya@tcha.org>
parents:
32485
diff
changeset
|
17 want2 = re.sub(r'''\bu(['"])(.*?)\1''', r'\1\2\1', want) # py2: u'' |
52ec9ac0303b
doctest: normalize b'', u'' and exception output on Python 3
Yuya Nishihara <yuya@tcha.org>
parents:
32485
diff
changeset
|
18 got2 = re.sub(r'''\bb(['"])(.*?)\1''', r'\1\2\1', got) # py3: b'' |
52ec9ac0303b
doctest: normalize b'', u'' and exception output on Python 3
Yuya Nishihara <yuya@tcha.org>
parents:
32485
diff
changeset
|
19 # py3: <exc.name>: b'<msg>' -> <name>: <msg> |
52ec9ac0303b
doctest: normalize b'', u'' and exception output on Python 3
Yuya Nishihara <yuya@tcha.org>
parents:
32485
diff
changeset
|
20 # <exc.name>: <others> -> <name>: <others> |
52ec9ac0303b
doctest: normalize b'', u'' and exception output on Python 3
Yuya Nishihara <yuya@tcha.org>
parents:
32485
diff
changeset
|
21 got2 = re.sub(r'''^mercurial\.\w+\.(\w+): (['"])(.*?)\2''', r'\1: \3', |
52ec9ac0303b
doctest: normalize b'', u'' and exception output on Python 3
Yuya Nishihara <yuya@tcha.org>
parents:
32485
diff
changeset
|
22 got2, re.MULTILINE) |
52ec9ac0303b
doctest: normalize b'', u'' and exception output on Python 3
Yuya Nishihara <yuya@tcha.org>
parents:
32485
diff
changeset
|
23 got2 = re.sub(r'^mercurial\.\w+\.(\w+): ', r'\1: ', got2, re.MULTILINE) |
52ec9ac0303b
doctest: normalize b'', u'' and exception output on Python 3
Yuya Nishihara <yuya@tcha.org>
parents:
32485
diff
changeset
|
24 return any(doctest.OutputChecker.check_output(self, w, g, optionflags) |
52ec9ac0303b
doctest: normalize b'', u'' and exception output on Python 3
Yuya Nishihara <yuya@tcha.org>
parents:
32485
diff
changeset
|
25 for w, g in [(want, got), (want2, got2)]) |
52ec9ac0303b
doctest: normalize b'', u'' and exception output on Python 3
Yuya Nishihara <yuya@tcha.org>
parents:
32485
diff
changeset
|
26 |
34424
e416819d9ebb
doctest: drop hack to run py2/3 tests selectively
Yuya Nishihara <yuya@tcha.org>
parents:
34362
diff
changeset
|
27 def testmod(name, optionflags=0, testtarget=None): |
20047
10a7d2bcb81b
tests: make doctest test runner less verbose
Mads Kiilerich <madski@unity3d.com>
parents:
19098
diff
changeset
|
28 __import__(name) |
10a7d2bcb81b
tests: make doctest test runner less verbose
Mads Kiilerich <madski@unity3d.com>
parents:
19098
diff
changeset
|
29 mod = sys.modules[name] |
10a7d2bcb81b
tests: make doctest test runner less verbose
Mads Kiilerich <madski@unity3d.com>
parents:
19098
diff
changeset
|
30 if testtarget is not None: |
10a7d2bcb81b
tests: make doctest test runner less verbose
Mads Kiilerich <madski@unity3d.com>
parents:
19098
diff
changeset
|
31 mod = getattr(mod, testtarget) |
34140
52ec9ac0303b
doctest: normalize b'', u'' and exception output on Python 3
Yuya Nishihara <yuya@tcha.org>
parents:
32485
diff
changeset
|
32 |
52ec9ac0303b
doctest: normalize b'', u'' and exception output on Python 3
Yuya Nishihara <yuya@tcha.org>
parents:
32485
diff
changeset
|
33 # minimal copy of doctest.testmod() |
52ec9ac0303b
doctest: normalize b'', u'' and exception output on Python 3
Yuya Nishihara <yuya@tcha.org>
parents:
32485
diff
changeset
|
34 finder = doctest.DocTestFinder() |
52ec9ac0303b
doctest: normalize b'', u'' and exception output on Python 3
Yuya Nishihara <yuya@tcha.org>
parents:
32485
diff
changeset
|
35 checker = None |
52ec9ac0303b
doctest: normalize b'', u'' and exception output on Python 3
Yuya Nishihara <yuya@tcha.org>
parents:
32485
diff
changeset
|
36 if ispy3: |
52ec9ac0303b
doctest: normalize b'', u'' and exception output on Python 3
Yuya Nishihara <yuya@tcha.org>
parents:
32485
diff
changeset
|
37 checker = py3docchecker() |
52ec9ac0303b
doctest: normalize b'', u'' and exception output on Python 3
Yuya Nishihara <yuya@tcha.org>
parents:
32485
diff
changeset
|
38 runner = doctest.DocTestRunner(checker=checker, optionflags=optionflags) |
52ec9ac0303b
doctest: normalize b'', u'' and exception output on Python 3
Yuya Nishihara <yuya@tcha.org>
parents:
32485
diff
changeset
|
39 for test in finder.find(mod, name): |
52ec9ac0303b
doctest: normalize b'', u'' and exception output on Python 3
Yuya Nishihara <yuya@tcha.org>
parents:
32485
diff
changeset
|
40 runner.run(test) |
52ec9ac0303b
doctest: normalize b'', u'' and exception output on Python 3
Yuya Nishihara <yuya@tcha.org>
parents:
32485
diff
changeset
|
41 runner.summarize() |
14171
fa2b596db182
ui: add configint function and tests
Sune Foldager <cryo@cyanite.org>
parents:
13949
diff
changeset
|
42 |
27432
77d25b913f80
changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents:
25805
diff
changeset
|
43 testmod('mercurial.changegroup') |
20047
10a7d2bcb81b
tests: make doctest test runner less verbose
Mads Kiilerich <madski@unity3d.com>
parents:
19098
diff
changeset
|
44 testmod('mercurial.changelog') |
36511
aa3294027936
cmdutil: expand filename format string by templater (BC)
Yuya Nishihara <yuya@tcha.org>
parents:
35169
diff
changeset
|
45 testmod('mercurial.cmdutil') |
31518
43d6ef658874
color: insert color code after every "\e[0m" (issue5413)
Yuya Nishihara <yuya@tcha.org>
parents:
31481
diff
changeset
|
46 testmod('mercurial.color') |
31481
a7c687c35119
ui: move configlist parser to config.py
Jun Wu <quark@fb.com>
parents:
31439
diff
changeset
|
47 testmod('mercurial.config') |
32485
05abc47f3746
annotate: add core algorithm to skip a rev
Siddharth Agarwal <sid0@fb.com>
parents:
31808
diff
changeset
|
48 testmod('mercurial.context') |
34212
dfd009e5f9f2
py3: iterate bytes as a byte string in dagparser.py
Yuya Nishihara <yuya@tcha.org>
parents:
34208
diff
changeset
|
49 testmod('mercurial.dagparser', optionflags=doctest.NORMALIZE_WHITESPACE) |
20047
10a7d2bcb81b
tests: make doctest test runner less verbose
Mads Kiilerich <madski@unity3d.com>
parents:
19098
diff
changeset
|
50 testmod('mercurial.dispatch') |
34218
aa877860d4d7
py3: use 'surrogatepass' error handler to process U+DCxx transparently
Yuya Nishihara <yuya@tcha.org>
parents:
34215
diff
changeset
|
51 testmod('mercurial.encoding') |
35169
898c6f812a51
fancyopts: add early-options parser compatible with getopt()
Yuya Nishihara <yuya@tcha.org>
parents:
34424
diff
changeset
|
52 testmod('mercurial.fancyopts') |
34256
ebe3d0095c69
py3: convert system strings to bytes in doctest of formatter.py
Yuya Nishihara <yuya@tcha.org>
parents:
34254
diff
changeset
|
53 testmod('mercurial.formatter') |
20799
069bf1b821c8
clone: add doctest for default destination
Yuya Nishihara <yuya@tcha.org>
parents:
20419
diff
changeset
|
54 testmod('mercurial.hg') |
34358
8cbcee0b923d
py3: remove use of str() in hgwebdir
Yuya Nishihara <yuya@tcha.org>
parents:
34256
diff
changeset
|
55 testmod('mercurial.hgweb.hgwebdir_mod') |
20047
10a7d2bcb81b
tests: make doctest test runner less verbose
Mads Kiilerich <madski@unity3d.com>
parents:
19098
diff
changeset
|
56 testmod('mercurial.match') |
31808
ca3b4a2b7e54
mdiff: add a hunkinrange helper function
Denis Laxalde <denis@laxalde.org>
parents:
31518
diff
changeset
|
57 testmod('mercurial.mdiff') |
20047
10a7d2bcb81b
tests: make doctest test runner less verbose
Mads Kiilerich <madski@unity3d.com>
parents:
19098
diff
changeset
|
58 testmod('mercurial.minirst') |
34253
5ce32fe7df34
py3: fix doctests in patch.py to be compatible with Python 3
Yuya Nishihara <yuya@tcha.org>
parents:
34218
diff
changeset
|
59 testmod('mercurial.patch') |
34254
cd022a11ec83
py3: use bytes os.sep in doctest of pathutil.py
Yuya Nishihara <yuya@tcha.org>
parents:
34253
diff
changeset
|
60 testmod('mercurial.pathutil') |
25306
c87b05925054
parser: add helper to reduce nesting of chained infix operations
Yuya Nishihara <yuya@tcha.org>
parents:
24243
diff
changeset
|
61 testmod('mercurial.parser') |
34141
9b4d7d4855f5
doctest: enable tests by default on Python 3
Yuya Nishihara <yuya@tcha.org>
parents:
34140
diff
changeset
|
62 testmod('mercurial.pycompat') |
38638
740f7d447222
revlog: add a doctest to _trimchunk
Boris Feld <boris.feld@octobus.net>
parents:
37183
diff
changeset
|
63 testmod('mercurial.revlog') |
31024
0b8356705de6
revset: split language services to revsetlang module (API)
Yuya Nishihara <yuya@tcha.org>
parents:
30881
diff
changeset
|
64 testmod('mercurial.revsetlang') |
30881
1be65deb3d54
smartset: move set classes and related functions from revset module (API)
Yuya Nishihara <yuya@tcha.org>
parents:
30560
diff
changeset
|
65 testmod('mercurial.smartset') |
34215
b4abc438a8c9
py3: iterate bytes as a byte string in store.lowerencode()
Yuya Nishihara <yuya@tcha.org>
parents:
34212
diff
changeset
|
66 testmod('mercurial.store') |
20840
308344d80fe5
subrepo: factor out Git version check to add doctests
Siddharth Agarwal <sid0@fb.com>
parents:
20799
diff
changeset
|
67 testmod('mercurial.subrepo') |
20047
10a7d2bcb81b
tests: make doctest test runner less verbose
Mads Kiilerich <madski@unity3d.com>
parents:
19098
diff
changeset
|
68 testmod('mercurial.templatefilters') |
25783
1f6878c87c25
templater: introduce one-pass parsing of nested template strings
Yuya Nishihara <yuya@tcha.org>
parents:
25306
diff
changeset
|
69 testmod('mercurial.templater') |
34208
0a2fd3bfc704
py3: convert function name to bytes in ui.configwith()
Yuya Nishihara <yuya@tcha.org>
parents:
34206
diff
changeset
|
70 testmod('mercurial.ui') |
20047
10a7d2bcb81b
tests: make doctest test runner less verbose
Mads Kiilerich <madski@unity3d.com>
parents:
19098
diff
changeset
|
71 testmod('mercurial.url') |
34362
b76937fafe8a
py3: work around bytes/unicode divergence in parsedate()
Yuya Nishihara <yuya@tcha.org>
parents:
34359
diff
changeset
|
72 testmod('mercurial.util') |
20047
10a7d2bcb81b
tests: make doctest test runner less verbose
Mads Kiilerich <madski@unity3d.com>
parents:
19098
diff
changeset
|
73 testmod('mercurial.util', testtarget='platform') |
37083
f99d64e8a4e4
stringutil: move generic string helpers to new module
Yuya Nishihara <yuya@tcha.org>
parents:
36511
diff
changeset
|
74 testmod('mercurial.utils.stringutil') |
34359
ee10eb665036
py3: replace str(None) with literal in convcmd.py
Yuya Nishihara <yuya@tcha.org>
parents:
34358
diff
changeset
|
75 testmod('hgext.convert.convcmd') |
20047
10a7d2bcb81b
tests: make doctest test runner less verbose
Mads Kiilerich <madski@unity3d.com>
parents:
19098
diff
changeset
|
76 testmod('hgext.convert.cvsps') |
20048
da99ebd35f00
convert: readability and test of rpairs function
Mads Kiilerich <madski@unity3d.com>
parents:
20047
diff
changeset
|
77 testmod('hgext.convert.filemap') |
25788
a36fd0993522
convert: unescape Perforce-escaped special characters in filenames
Eugene Baranov <eug.baranov@gmail.com>
parents:
25783
diff
changeset
|
78 testmod('hgext.convert.p4') |
20419
e61a8395c3c1
convert: make subversion revsplit more stable when meeting revisions without @
Mads Kiilerich <madski@unity3d.com>
parents:
20048
diff
changeset
|
79 testmod('hgext.convert.subversion') |
37183
ded5ea279a93
fix: new extension for automatically modifying file contents
Danny Hooper <hooper@google.com>
parents:
37083
diff
changeset
|
80 testmod('hgext.fix') |
22546
aac5482db318
mq: refactor patchheader header ordering to match export (BC)
Mads Kiilerich <madski@unity3d.com>
parents:
21568
diff
changeset
|
81 testmod('hgext.mq') |
34206
1e71dddc10a2
drawdag: add a couple of doctests to help with python3 porting
Augie Fackler <raf@durin42.com>
parents:
34141
diff
changeset
|
82 # Helper scripts in tests/ that have doctests: |
1e71dddc10a2
drawdag: add a couple of doctests to help with python3 porting
Augie Fackler <raf@durin42.com>
parents:
34141
diff
changeset
|
83 testmod('drawdag') |