annotate tests/dumbhttp.py @ 40329:ed55a0077490

wireprotov2: implement command for retrieving raw store files Implementing shallow clone of the changelog is hard. We want the 4.8 release to have a fast implementation of partial clone in wireprotov2. In order to achieve fast, we can't use deltas for transferring changelog and manifestlog data. Per discussions at the 4.8 sprint, this commit implements a somwwhat hacky and likely-to-be-changed-drastically-or-dropped command in wireprotov2 that facilitates access to raw store files, namely the changelog and manifestlog. Using this command, clients can perform a "stream clone" of sorts for just the changelog and manifestlog. This will allow clients to fetch the changelog and manifest revlogs, stream them to disk (which should be fast), then follow up filesdata requests for files revision data for a particular changeset. Differential Revision: https://phab.mercurial-scm.org/D5134
author Gregory Szorc <gregory.szorc@gmail.com>
date Tue, 16 Oct 2018 21:31:21 +0200
parents e46c3b6a47b5
children 2372284d9457
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
22959
10116463b0b1 tests: pull common http server setup out of individual tests
Mike Hommey <mh@glandium.org>
parents:
diff changeset
1 #!/usr/bin/env python
10116463b0b1 tests: pull common http server setup out of individual tests
Mike Hommey <mh@glandium.org>
parents:
diff changeset
2
27282
0bb8c405a7c7 tests/dumbhttp: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 23136
diff changeset
3 from __future__ import absolute_import
0bb8c405a7c7 tests/dumbhttp: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 23136
diff changeset
4
22959
10116463b0b1 tests: pull common http server setup out of individual tests
Mike Hommey <mh@glandium.org>
parents:
diff changeset
5 """
10116463b0b1 tests: pull common http server setup out of individual tests
Mike Hommey <mh@glandium.org>
parents:
diff changeset
6 Small and dumb HTTP server for use in tests.
10116463b0b1 tests: pull common http server setup out of individual tests
Mike Hommey <mh@glandium.org>
parents:
diff changeset
7 """
10116463b0b1 tests: pull common http server setup out of individual tests
Mike Hommey <mh@glandium.org>
parents:
diff changeset
8
27282
0bb8c405a7c7 tests/dumbhttp: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 23136
diff changeset
9 import optparse
31004
d05fefbb5ab3 dumbhttp: use IPv6 if HGIPV6 is set to 1
Jun Wu <quark@fb.com>
parents: 30506
diff changeset
10 import os
27282
0bb8c405a7c7 tests/dumbhttp: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 23136
diff changeset
11 import signal
31004
d05fefbb5ab3 dumbhttp: use IPv6 if HGIPV6 is set to 1
Jun Wu <quark@fb.com>
parents: 30506
diff changeset
12 import socket
27282
0bb8c405a7c7 tests/dumbhttp: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 23136
diff changeset
13 import sys
22959
10116463b0b1 tests: pull common http server setup out of individual tests
Mike Hommey <mh@glandium.org>
parents:
diff changeset
14
27282
0bb8c405a7c7 tests/dumbhttp: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 23136
diff changeset
15 from mercurial import (
38593
e46c3b6a47b5 py3: fix dumbhttp.py to convert --daemon-postexec arguments back to bytes
Yuya Nishihara <yuya@tcha.org>
parents: 37672
diff changeset
16 encoding,
37672
8bacc09814ba py3: make values bytes before passing into server.runservice()
Pulkit Goyal <7895pulkit@gmail.com>
parents: 34944
diff changeset
17 pycompat,
30506
d9d8d78e6bc9 server: move cmdutil.service() to new module (API)
Yuya Nishihara <yuya@tcha.org>
parents: 29566
diff changeset
18 server,
29566
075146e85bb6 py3: conditionalize BaseHTTPServer, SimpleHTTPServer and CGIHTTPServer import
Pulkit Goyal <7895pulkit@gmail.com>
parents: 28771
diff changeset
19 util,
27282
0bb8c405a7c7 tests/dumbhttp: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 23136
diff changeset
20 )
0bb8c405a7c7 tests/dumbhttp: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 23136
diff changeset
21
29566
075146e85bb6 py3: conditionalize BaseHTTPServer, SimpleHTTPServer and CGIHTTPServer import
Pulkit Goyal <7895pulkit@gmail.com>
parents: 28771
diff changeset
22 httpserver = util.httpserver
27282
0bb8c405a7c7 tests/dumbhttp: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 23136
diff changeset
23 OptionParser = optparse.OptionParser
22959
10116463b0b1 tests: pull common http server setup out of individual tests
Mike Hommey <mh@glandium.org>
parents:
diff changeset
24
31004
d05fefbb5ab3 dumbhttp: use IPv6 if HGIPV6 is set to 1
Jun Wu <quark@fb.com>
parents: 30506
diff changeset
25 if os.environ.get('HGIPV6', '0') == '1':
d05fefbb5ab3 dumbhttp: use IPv6 if HGIPV6 is set to 1
Jun Wu <quark@fb.com>
parents: 30506
diff changeset
26 class simplehttpserver(httpserver.httpserver):
d05fefbb5ab3 dumbhttp: use IPv6 if HGIPV6 is set to 1
Jun Wu <quark@fb.com>
parents: 30506
diff changeset
27 address_family = socket.AF_INET6
d05fefbb5ab3 dumbhttp: use IPv6 if HGIPV6 is set to 1
Jun Wu <quark@fb.com>
parents: 30506
diff changeset
28 else:
d05fefbb5ab3 dumbhttp: use IPv6 if HGIPV6 is set to 1
Jun Wu <quark@fb.com>
parents: 30506
diff changeset
29 simplehttpserver = httpserver.httpserver
d05fefbb5ab3 dumbhttp: use IPv6 if HGIPV6 is set to 1
Jun Wu <quark@fb.com>
parents: 30506
diff changeset
30
34944
bf2db35a6fe7 test-static-http: flush access log per request
Yuya Nishihara <yuya@tcha.org>
parents: 34925
diff changeset
31 class _httprequesthandler(httpserver.simplehttprequesthandler):
bf2db35a6fe7 test-static-http: flush access log per request
Yuya Nishihara <yuya@tcha.org>
parents: 34925
diff changeset
32 def log_message(self, format, *args):
bf2db35a6fe7 test-static-http: flush access log per request
Yuya Nishihara <yuya@tcha.org>
parents: 34925
diff changeset
33 httpserver.simplehttprequesthandler.log_message(self, format, *args)
bf2db35a6fe7 test-static-http: flush access log per request
Yuya Nishihara <yuya@tcha.org>
parents: 34925
diff changeset
34 sys.stderr.flush()
bf2db35a6fe7 test-static-http: flush access log per request
Yuya Nishihara <yuya@tcha.org>
parents: 34925
diff changeset
35
23136
6eab50a34fed tests: have dumbhttp.py use cmdutil.service() to wait for child to listen()
Yuya Nishihara <yuya@tcha.org>
parents: 22959
diff changeset
36 class simplehttpservice(object):
6eab50a34fed tests: have dumbhttp.py use cmdutil.service() to wait for child to listen()
Yuya Nishihara <yuya@tcha.org>
parents: 22959
diff changeset
37 def __init__(self, host, port):
6eab50a34fed tests: have dumbhttp.py use cmdutil.service() to wait for child to listen()
Yuya Nishihara <yuya@tcha.org>
parents: 22959
diff changeset
38 self.address = (host, port)
6eab50a34fed tests: have dumbhttp.py use cmdutil.service() to wait for child to listen()
Yuya Nishihara <yuya@tcha.org>
parents: 22959
diff changeset
39 def init(self):
34944
bf2db35a6fe7 test-static-http: flush access log per request
Yuya Nishihara <yuya@tcha.org>
parents: 34925
diff changeset
40 self.httpd = simplehttpserver(self.address, _httprequesthandler)
23136
6eab50a34fed tests: have dumbhttp.py use cmdutil.service() to wait for child to listen()
Yuya Nishihara <yuya@tcha.org>
parents: 22959
diff changeset
41 def run(self):
6eab50a34fed tests: have dumbhttp.py use cmdutil.service() to wait for child to listen()
Yuya Nishihara <yuya@tcha.org>
parents: 22959
diff changeset
42 self.httpd.serve_forever()
22959
10116463b0b1 tests: pull common http server setup out of individual tests
Mike Hommey <mh@glandium.org>
parents:
diff changeset
43
10116463b0b1 tests: pull common http server setup out of individual tests
Mike Hommey <mh@glandium.org>
parents:
diff changeset
44 if __name__ == '__main__':
10116463b0b1 tests: pull common http server setup out of individual tests
Mike Hommey <mh@glandium.org>
parents:
diff changeset
45 parser = OptionParser()
10116463b0b1 tests: pull common http server setup out of individual tests
Mike Hommey <mh@glandium.org>
parents:
diff changeset
46 parser.add_option('-p', '--port', dest='port', type='int', default=8000,
10116463b0b1 tests: pull common http server setup out of individual tests
Mike Hommey <mh@glandium.org>
parents:
diff changeset
47 help='TCP port to listen on', metavar='PORT')
10116463b0b1 tests: pull common http server setup out of individual tests
Mike Hommey <mh@glandium.org>
parents:
diff changeset
48 parser.add_option('-H', '--host', dest='host', default='localhost',
10116463b0b1 tests: pull common http server setup out of individual tests
Mike Hommey <mh@glandium.org>
parents:
diff changeset
49 help='hostname or IP to listen on', metavar='HOST')
34925
8b95e420e248 test-static-http: show all files accessed over HTTP
Yuya Nishihara <yuya@tcha.org>
parents: 31004
diff changeset
50 parser.add_option('--logfile', help='file name of access/error log')
22959
10116463b0b1 tests: pull common http server setup out of individual tests
Mike Hommey <mh@glandium.org>
parents:
diff changeset
51 parser.add_option('--pid', dest='pid',
10116463b0b1 tests: pull common http server setup out of individual tests
Mike Hommey <mh@glandium.org>
parents:
diff changeset
52 help='file name where the PID of the server is stored')
10116463b0b1 tests: pull common http server setup out of individual tests
Mike Hommey <mh@glandium.org>
parents:
diff changeset
53 parser.add_option('-f', '--foreground', dest='foreground',
10116463b0b1 tests: pull common http server setup out of individual tests
Mike Hommey <mh@glandium.org>
parents:
diff changeset
54 action='store_true',
10116463b0b1 tests: pull common http server setup out of individual tests
Mike Hommey <mh@glandium.org>
parents:
diff changeset
55 help='do not start the HTTP server in the background')
28451
c90cfe76e024 serve: accept multiple values for --daemon-postexec
Jun Wu <quark@fb.com>
parents: 28194
diff changeset
56 parser.add_option('--daemon-postexec', action='append')
22959
10116463b0b1 tests: pull common http server setup out of individual tests
Mike Hommey <mh@glandium.org>
parents:
diff changeset
57
10116463b0b1 tests: pull common http server setup out of individual tests
Mike Hommey <mh@glandium.org>
parents:
diff changeset
58 (options, args) = parser.parse_args()
10116463b0b1 tests: pull common http server setup out of individual tests
Mike Hommey <mh@glandium.org>
parents:
diff changeset
59
10116463b0b1 tests: pull common http server setup out of individual tests
Mike Hommey <mh@glandium.org>
parents:
diff changeset
60 signal.signal(signal.SIGTERM, lambda x, y: sys.exit(0))
10116463b0b1 tests: pull common http server setup out of individual tests
Mike Hommey <mh@glandium.org>
parents:
diff changeset
61
34925
8b95e420e248 test-static-http: show all files accessed over HTTP
Yuya Nishihara <yuya@tcha.org>
parents: 31004
diff changeset
62 if options.foreground and options.logfile:
8b95e420e248 test-static-http: show all files accessed over HTTP
Yuya Nishihara <yuya@tcha.org>
parents: 31004
diff changeset
63 parser.error("options --logfile and --foreground are mutually "
8b95e420e248 test-static-http: show all files accessed over HTTP
Yuya Nishihara <yuya@tcha.org>
parents: 31004
diff changeset
64 "exclusive")
22959
10116463b0b1 tests: pull common http server setup out of individual tests
Mike Hommey <mh@glandium.org>
parents:
diff changeset
65 if options.foreground and options.pid:
10116463b0b1 tests: pull common http server setup out of individual tests
Mike Hommey <mh@glandium.org>
parents:
diff changeset
66 parser.error("options --pid and --foreground are mutually exclusive")
10116463b0b1 tests: pull common http server setup out of individual tests
Mike Hommey <mh@glandium.org>
parents:
diff changeset
67
37672
8bacc09814ba py3: make values bytes before passing into server.runservice()
Pulkit Goyal <7895pulkit@gmail.com>
parents: 34944
diff changeset
68 opts = {b'pid_file': options.pid,
8bacc09814ba py3: make values bytes before passing into server.runservice()
Pulkit Goyal <7895pulkit@gmail.com>
parents: 34944
diff changeset
69 b'daemon': not options.foreground,
38593
e46c3b6a47b5 py3: fix dumbhttp.py to convert --daemon-postexec arguments back to bytes
Yuya Nishihara <yuya@tcha.org>
parents: 37672
diff changeset
70 b'daemon_postexec': pycompat.rapply(encoding.strtolocal,
e46c3b6a47b5 py3: fix dumbhttp.py to convert --daemon-postexec arguments back to bytes
Yuya Nishihara <yuya@tcha.org>
parents: 37672
diff changeset
71 options.daemon_postexec)}
23136
6eab50a34fed tests: have dumbhttp.py use cmdutil.service() to wait for child to listen()
Yuya Nishihara <yuya@tcha.org>
parents: 22959
diff changeset
72 service = simplehttpservice(options.host, options.port)
37672
8bacc09814ba py3: make values bytes before passing into server.runservice()
Pulkit Goyal <7895pulkit@gmail.com>
parents: 34944
diff changeset
73 runargs = [sys.executable, __file__] + sys.argv[1:]
8bacc09814ba py3: make values bytes before passing into server.runservice()
Pulkit Goyal <7895pulkit@gmail.com>
parents: 34944
diff changeset
74 runargs = [pycompat.fsencode(a) for a in runargs]
30506
d9d8d78e6bc9 server: move cmdutil.service() to new module (API)
Yuya Nishihara <yuya@tcha.org>
parents: 29566
diff changeset
75 server.runservice(opts, initfn=service.init, runfn=service.run,
34925
8b95e420e248 test-static-http: show all files accessed over HTTP
Yuya Nishihara <yuya@tcha.org>
parents: 31004
diff changeset
76 logfile=options.logfile,
37672
8bacc09814ba py3: make values bytes before passing into server.runservice()
Pulkit Goyal <7895pulkit@gmail.com>
parents: 34944
diff changeset
77 runargs=runargs)