Mercurial > hg-stable
annotate tests/get-with-headers.py @ 26379:39d643252b9f
revlog: use existing file handle when reading during _addrevision
_addrevision() may need to read from revlogs as part of computing
deltas. Previously, we would flush existing file handles and open
a new, short-lived file handle to perform the reading.
If we have an existing file handle, it seems logical to reuse it
for reading instead of opening a new file handle. This patch
makes that the new behavior.
After this patch, revlog files are only reopened when adding
revisions if the revlog is switched from inline to non-inline.
On Linux when unbundling a bundle of the mozilla-central repo, this
patch has the following impact on system call counts:
Call Before After Delta
write 827,639 673,390 -154,249
open 700,103 684,089 -16,014
read 74,489 74,489 0
fstat 493,924 461,896 -32,028
close 249,131 233,117 -16,014
stat 242,001 242,001 0
lstat 18,676 18,676 0
lseek 20,268 20,268 0
ioctl 14,652 13,173 -1,479
TOTAL 3,180,758 2,930,679 -250,079
It's worth noting that many of the open() calls fail due to missing
files. That's why there are many more open() calls than close().
Despite the significant system call reduction, this change does not
seem to have a significant performance impact on Linux.
On Windows 10 (not a VM, on a SSD), this patch appears to reduce
unbundle time for mozilla-central from ~960s to ~920s. This isn't
as significant as I was hoping. But a decrease it is nonetheless.
Still, Windows unbundle performance is still >2x slower than Linux.
Despite the lack of significant gains, fewer system calls is fewer
system calls. If nothing else, this will narrow the focus of potential
areas to optimize in the future.
author | Gregory Szorc <gregory.szorc@gmail.com> |
---|---|
date | Sun, 27 Sep 2015 16:08:18 -0700 |
parents | 5a6820f8da4d |
children | 8e86679d8acd |
rev | line source |
---|---|
2532
84655f721f39
Add a test for getting raw files via the web UI.
Eric Hopper <hopper@omnifarious.org>
parents:
diff
changeset
|
1 #!/usr/bin/env python |
84655f721f39
Add a test for getting raw files via the web UI.
Eric Hopper <hopper@omnifarious.org>
parents:
diff
changeset
|
2 |
8447
d5ebcf8f6855
tests: fix doc string in get-with-headers.py
Martin Geisler <mg@lazybytes.net>
parents:
7544
diff
changeset
|
3 """This does HTTP GET requests given a host:port and path and returns |
2532
84655f721f39
Add a test for getting raw files via the web UI.
Eric Hopper <hopper@omnifarious.org>
parents:
diff
changeset
|
4 a subset of the headers plus the body of the result.""" |
84655f721f39
Add a test for getting raw files via the web UI.
Eric Hopper <hopper@omnifarious.org>
parents:
diff
changeset
|
5 |
10905
13a1b2fb7ef2
pylint, pyflakes: remove unused or duplicate imports
Nicolas Dumazet <nicdumz.commits@gmail.com>
parents:
9722
diff
changeset
|
6 import httplib, sys |
7054
e837f2294643
get-with-headers: fix stream modes under Windows
Patrick Mezard <pmezard@gmail.com>
parents:
5561
diff
changeset
|
7 |
e837f2294643
get-with-headers: fix stream modes under Windows
Patrick Mezard <pmezard@gmail.com>
parents:
5561
diff
changeset
|
8 try: |
24543
747401086a38
get-with-headers: support parsing and pretty printing JSON
Gregory Szorc <gregory.szorc@gmail.com>
parents:
23409
diff
changeset
|
9 import json |
747401086a38
get-with-headers: support parsing and pretty printing JSON
Gregory Szorc <gregory.szorc@gmail.com>
parents:
23409
diff
changeset
|
10 except ImportError: |
747401086a38
get-with-headers: support parsing and pretty printing JSON
Gregory Szorc <gregory.szorc@gmail.com>
parents:
23409
diff
changeset
|
11 try: |
747401086a38
get-with-headers: support parsing and pretty printing JSON
Gregory Szorc <gregory.szorc@gmail.com>
parents:
23409
diff
changeset
|
12 import simplejson as json |
747401086a38
get-with-headers: support parsing and pretty printing JSON
Gregory Szorc <gregory.szorc@gmail.com>
parents:
23409
diff
changeset
|
13 except ImportError: |
747401086a38
get-with-headers: support parsing and pretty printing JSON
Gregory Szorc <gregory.szorc@gmail.com>
parents:
23409
diff
changeset
|
14 json = None |
747401086a38
get-with-headers: support parsing and pretty printing JSON
Gregory Szorc <gregory.szorc@gmail.com>
parents:
23409
diff
changeset
|
15 |
747401086a38
get-with-headers: support parsing and pretty printing JSON
Gregory Szorc <gregory.szorc@gmail.com>
parents:
23409
diff
changeset
|
16 try: |
7054
e837f2294643
get-with-headers: fix stream modes under Windows
Patrick Mezard <pmezard@gmail.com>
parents:
5561
diff
changeset
|
17 import msvcrt, os |
e837f2294643
get-with-headers: fix stream modes under Windows
Patrick Mezard <pmezard@gmail.com>
parents:
5561
diff
changeset
|
18 msvcrt.setmode(sys.stdout.fileno(), os.O_BINARY) |
e837f2294643
get-with-headers: fix stream modes under Windows
Patrick Mezard <pmezard@gmail.com>
parents:
5561
diff
changeset
|
19 msvcrt.setmode(sys.stderr.fileno(), os.O_BINARY) |
e837f2294643
get-with-headers: fix stream modes under Windows
Patrick Mezard <pmezard@gmail.com>
parents:
5561
diff
changeset
|
20 except ImportError: |
e837f2294643
get-with-headers: fix stream modes under Windows
Patrick Mezard <pmezard@gmail.com>
parents:
5561
diff
changeset
|
21 pass |
e837f2294643
get-with-headers: fix stream modes under Windows
Patrick Mezard <pmezard@gmail.com>
parents:
5561
diff
changeset
|
22 |
12182
1121af239761
tests: extend get-with-headers to support cache testing
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
10905
diff
changeset
|
23 twice = False |
1121af239761
tests: extend get-with-headers to support cache testing
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
10905
diff
changeset
|
24 if '--twice' in sys.argv: |
1121af239761
tests: extend get-with-headers to support cache testing
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
10905
diff
changeset
|
25 sys.argv.remove('--twice') |
1121af239761
tests: extend get-with-headers to support cache testing
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
10905
diff
changeset
|
26 twice = True |
18400
f1118507174b
get-with-headers: add a --headeronly switch
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents:
18393
diff
changeset
|
27 headeronly = False |
f1118507174b
get-with-headers: add a --headeronly switch
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents:
18393
diff
changeset
|
28 if '--headeronly' in sys.argv: |
f1118507174b
get-with-headers: add a --headeronly switch
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents:
18393
diff
changeset
|
29 sys.argv.remove('--headeronly') |
f1118507174b
get-with-headers: add a --headeronly switch
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents:
18393
diff
changeset
|
30 headeronly = True |
24543
747401086a38
get-with-headers: support parsing and pretty printing JSON
Gregory Szorc <gregory.szorc@gmail.com>
parents:
23409
diff
changeset
|
31 formatjson = False |
747401086a38
get-with-headers: support parsing and pretty printing JSON
Gregory Szorc <gregory.szorc@gmail.com>
parents:
23409
diff
changeset
|
32 if '--json' in sys.argv: |
747401086a38
get-with-headers: support parsing and pretty printing JSON
Gregory Szorc <gregory.szorc@gmail.com>
parents:
23409
diff
changeset
|
33 sys.argv.remove('--json') |
747401086a38
get-with-headers: support parsing and pretty printing JSON
Gregory Szorc <gregory.szorc@gmail.com>
parents:
23409
diff
changeset
|
34 formatjson = True |
12182
1121af239761
tests: extend get-with-headers to support cache testing
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
10905
diff
changeset
|
35 |
1121af239761
tests: extend get-with-headers to support cache testing
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
10905
diff
changeset
|
36 tag = None |
1121af239761
tests: extend get-with-headers to support cache testing
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
10905
diff
changeset
|
37 def request(host, path, show): |
17017
953faba28e91
tests: prepare get-with-headers.py for MSYS
Mads Kiilerich <mads@kiilerich.com>
parents:
12250
diff
changeset
|
38 assert not path.startswith('/'), path |
12182
1121af239761
tests: extend get-with-headers to support cache testing
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
10905
diff
changeset
|
39 global tag |
1121af239761
tests: extend get-with-headers to support cache testing
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
10905
diff
changeset
|
40 headers = {} |
1121af239761
tests: extend get-with-headers to support cache testing
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
10905
diff
changeset
|
41 if tag: |
1121af239761
tests: extend get-with-headers to support cache testing
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
10905
diff
changeset
|
42 headers['If-None-Match'] = tag |
5561
22713dce19f6
hgweb: return meaningful HTTP status codes instead of nonsense
Bryan O'Sullivan <bos@serpentine.com>
parents:
2532
diff
changeset
|
43 |
12182
1121af239761
tests: extend get-with-headers to support cache testing
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
10905
diff
changeset
|
44 conn = httplib.HTTPConnection(host) |
17017
953faba28e91
tests: prepare get-with-headers.py for MSYS
Mads Kiilerich <mads@kiilerich.com>
parents:
12250
diff
changeset
|
45 conn.request("GET", '/' + path, None, headers) |
12182
1121af239761
tests: extend get-with-headers to support cache testing
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
10905
diff
changeset
|
46 response = conn.getresponse() |
25208
5a6820f8da4d
tests: just use 'response.reason'
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
24543
diff
changeset
|
47 print response.status, response.reason |
18380
a4d7fd7ad1f7
serve: don't send any content headers with 304 responses
Mads Kiilerich <madski@unity3d.com>
parents:
17017
diff
changeset
|
48 if show[:1] == ['-']: |
18393
a38039ef7312
tests: make test-hgweb.t output stable
Mads Kiilerich <madski@unity3d.com>
parents:
18380
diff
changeset
|
49 show = sorted(h for h, v in response.getheaders() |
a38039ef7312
tests: make test-hgweb.t output stable
Mads Kiilerich <madski@unity3d.com>
parents:
18380
diff
changeset
|
50 if h.lower() not in show) |
12182
1121af239761
tests: extend get-with-headers to support cache testing
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
10905
diff
changeset
|
51 for h in [h.lower() for h in show]: |
1121af239761
tests: extend get-with-headers to support cache testing
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
10905
diff
changeset
|
52 if response.getheader(h, None) is not None: |
1121af239761
tests: extend get-with-headers to support cache testing
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
10905
diff
changeset
|
53 print "%s: %s" % (h, response.getheader(h)) |
18400
f1118507174b
get-with-headers: add a --headeronly switch
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents:
18393
diff
changeset
|
54 if not headeronly: |
f1118507174b
get-with-headers: add a --headeronly switch
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents:
18393
diff
changeset
|
55 print |
23409
dc4d2cd3aa3e
hgweb: send proper HTTP response after uncaught exception
Gregory Szorc <gregory.szorc@gmail.com>
parents:
19865
diff
changeset
|
56 data = response.read() |
24543
747401086a38
get-with-headers: support parsing and pretty printing JSON
Gregory Szorc <gregory.szorc@gmail.com>
parents:
23409
diff
changeset
|
57 |
747401086a38
get-with-headers: support parsing and pretty printing JSON
Gregory Szorc <gregory.szorc@gmail.com>
parents:
23409
diff
changeset
|
58 # Pretty print JSON. This also has the beneficial side-effect |
747401086a38
get-with-headers: support parsing and pretty printing JSON
Gregory Szorc <gregory.szorc@gmail.com>
parents:
23409
diff
changeset
|
59 # of verifying emitted JSON is well-formed. |
747401086a38
get-with-headers: support parsing and pretty printing JSON
Gregory Szorc <gregory.szorc@gmail.com>
parents:
23409
diff
changeset
|
60 if formatjson: |
747401086a38
get-with-headers: support parsing and pretty printing JSON
Gregory Szorc <gregory.szorc@gmail.com>
parents:
23409
diff
changeset
|
61 if not json: |
747401086a38
get-with-headers: support parsing and pretty printing JSON
Gregory Szorc <gregory.szorc@gmail.com>
parents:
23409
diff
changeset
|
62 print 'no json module not available' |
747401086a38
get-with-headers: support parsing and pretty printing JSON
Gregory Szorc <gregory.szorc@gmail.com>
parents:
23409
diff
changeset
|
63 print 'did you forget a #require json?' |
747401086a38
get-with-headers: support parsing and pretty printing JSON
Gregory Szorc <gregory.szorc@gmail.com>
parents:
23409
diff
changeset
|
64 sys.exit(1) |
747401086a38
get-with-headers: support parsing and pretty printing JSON
Gregory Szorc <gregory.szorc@gmail.com>
parents:
23409
diff
changeset
|
65 |
747401086a38
get-with-headers: support parsing and pretty printing JSON
Gregory Szorc <gregory.szorc@gmail.com>
parents:
23409
diff
changeset
|
66 # json.dumps() will print trailing newlines. Eliminate them |
747401086a38
get-with-headers: support parsing and pretty printing JSON
Gregory Szorc <gregory.szorc@gmail.com>
parents:
23409
diff
changeset
|
67 # to make tests easier to write. |
747401086a38
get-with-headers: support parsing and pretty printing JSON
Gregory Szorc <gregory.szorc@gmail.com>
parents:
23409
diff
changeset
|
68 data = json.loads(data) |
747401086a38
get-with-headers: support parsing and pretty printing JSON
Gregory Szorc <gregory.szorc@gmail.com>
parents:
23409
diff
changeset
|
69 lines = json.dumps(data, sort_keys=True, indent=2).splitlines() |
747401086a38
get-with-headers: support parsing and pretty printing JSON
Gregory Szorc <gregory.szorc@gmail.com>
parents:
23409
diff
changeset
|
70 for line in lines: |
747401086a38
get-with-headers: support parsing and pretty printing JSON
Gregory Szorc <gregory.szorc@gmail.com>
parents:
23409
diff
changeset
|
71 print line.rstrip() |
747401086a38
get-with-headers: support parsing and pretty printing JSON
Gregory Szorc <gregory.szorc@gmail.com>
parents:
23409
diff
changeset
|
72 else: |
747401086a38
get-with-headers: support parsing and pretty printing JSON
Gregory Szorc <gregory.szorc@gmail.com>
parents:
23409
diff
changeset
|
73 sys.stdout.write(data) |
12182
1121af239761
tests: extend get-with-headers to support cache testing
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
10905
diff
changeset
|
74 |
18400
f1118507174b
get-with-headers: add a --headeronly switch
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents:
18393
diff
changeset
|
75 if twice and response.getheader('ETag', None): |
f1118507174b
get-with-headers: add a --headeronly switch
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents:
18393
diff
changeset
|
76 tag = response.getheader('ETag') |
12182
1121af239761
tests: extend get-with-headers to support cache testing
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
10905
diff
changeset
|
77 |
1121af239761
tests: extend get-with-headers to support cache testing
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
10905
diff
changeset
|
78 return response.status |
1121af239761
tests: extend get-with-headers to support cache testing
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
10905
diff
changeset
|
79 |
1121af239761
tests: extend get-with-headers to support cache testing
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
10905
diff
changeset
|
80 status = request(sys.argv[1], sys.argv[2], sys.argv[3:]) |
1121af239761
tests: extend get-with-headers to support cache testing
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
10905
diff
changeset
|
81 if twice: |
1121af239761
tests: extend get-with-headers to support cache testing
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
10905
diff
changeset
|
82 status = request(sys.argv[1], sys.argv[2], sys.argv[3:]) |
1121af239761
tests: extend get-with-headers to support cache testing
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
10905
diff
changeset
|
83 |
1121af239761
tests: extend get-with-headers to support cache testing
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
10905
diff
changeset
|
84 if 200 <= status <= 305: |
5561
22713dce19f6
hgweb: return meaningful HTTP status codes instead of nonsense
Bryan O'Sullivan <bos@serpentine.com>
parents:
2532
diff
changeset
|
85 sys.exit(0) |
22713dce19f6
hgweb: return meaningful HTTP status codes instead of nonsense
Bryan O'Sullivan <bos@serpentine.com>
parents:
2532
diff
changeset
|
86 sys.exit(1) |