Mercurial > hg
annotate mercurial/hgweb/protocol.py @ 26755:bb0b955d050d
streamclone: support for producing and consuming stream clone bundles
Up to this point, stream clones only existed as a dynamically generated
data format produced and consumed during streaming clones. In order to
support this efficient cloning format with the clone bundles feature, we
need a more formal, on disk representation of the streaming clone data.
This patch introduces a new "bundle" type for streaming clones. Unlike
existing bundles, it does not contain changegroup data. It does,
however, share the same concepts like the 4 byte header which identifies
the type of data that follows and the 2 byte abbreviation for
compression types (of which only "UN" is currently supported).
The new bundle format is essentially the existing stream clone version 1
data format with some headers at the beginning.
Content negotiation at stream clone request time checked for repository
format/requirements compatibility before initiating a stream clone. We
can't do active content negotiation when using clone bundles. So, we put
this set of requirements inside the payload so consumers have a built-in
mechanism for checking compatibility before reading and applying lots of
data. Of course, we will also advertise this requirements set in clone
bundles. But that's for another patch.
We currently don't have a mechanism to produce and consume this new
bundle format. This will be implemented in upcoming patches.
It's worth noting that if a legacy client attempts to `hg unbundle` a
stream clone bundle (with the "HGS1" header), it will abort with:
"unknown bundle version S1," which seems appropriate.
author | Gregory Szorc <gregory.szorc@gmail.com> |
---|---|
date | Sat, 17 Oct 2015 11:14:52 -0700 |
parents | 8d477543882b |
children | 37fcfe52c68c |
rev | line source |
---|---|
5598
d534ba1c4eb4
separate the wire protocol commands from the user interface commands
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
diff
changeset
|
1 # |
d534ba1c4eb4
separate the wire protocol commands from the user interface commands
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
diff
changeset
|
2 # Copyright 21 May 2005 - (c) 2005 Jake Edge <jake@edge2.net> |
d534ba1c4eb4
separate the wire protocol commands from the user interface commands
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
diff
changeset
|
3 # Copyright 2005-2007 Matt Mackall <mpm@selenic.com> |
d534ba1c4eb4
separate the wire protocol commands from the user interface commands
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
diff
changeset
|
4 # |
8225
46293a0c7e9f
updated license to be explicit about GPL version 2
Martin Geisler <mg@lazybytes.net>
parents:
8109
diff
changeset
|
5 # This software may be used and distributed according to the terms of the |
10263 | 6 # GNU General Public License version 2 or any later version. |
5598
d534ba1c4eb4
separate the wire protocol commands from the user interface commands
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
diff
changeset
|
7 |
14614
afccc64eea73
ui: use I/O descriptors internally
Idan Kamara <idankk86@gmail.com>
parents:
14494
diff
changeset
|
8 import cgi, cStringIO, zlib, urllib |
11595
368cd5325348
protocol: move hgweb protocol support back into protocol.py
Matt Mackall <mpm@selenic.com>
parents:
11594
diff
changeset
|
9 from mercurial import util, wireproto |
368cd5325348
protocol: move hgweb protocol support back into protocol.py
Matt Mackall <mpm@selenic.com>
parents:
11594
diff
changeset
|
10 from common import HTTP_OK |
5963
5be210afe1b8
hgweb: explicitly check if requested command exists
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
5915
diff
changeset
|
11 |
5993
948a41e77902
hgweb: explicit response status
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
5963
diff
changeset
|
12 HGTYPE = 'application/mercurial-0.1' |
15017
f4522df38c65
wireproto: add out-of-band error class to allow remote repo to report errors
Andrew Pritchard <andrewp@fogcreek.com>
parents:
14614
diff
changeset
|
13 HGERRTYPE = 'application/hg-error' |
11595
368cd5325348
protocol: move hgweb protocol support back into protocol.py
Matt Mackall <mpm@selenic.com>
parents:
11594
diff
changeset
|
14 |
20903
8d477543882b
wireproto: introduce an abstractserverproto class
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
18352
diff
changeset
|
15 class webproto(wireproto.abstractserverproto): |
14614
afccc64eea73
ui: use I/O descriptors internally
Idan Kamara <idankk86@gmail.com>
parents:
14494
diff
changeset
|
16 def __init__(self, req, ui): |
11595
368cd5325348
protocol: move hgweb protocol support back into protocol.py
Matt Mackall <mpm@selenic.com>
parents:
11594
diff
changeset
|
17 self.req = req |
368cd5325348
protocol: move hgweb protocol support back into protocol.py
Matt Mackall <mpm@selenic.com>
parents:
11594
diff
changeset
|
18 self.response = '' |
14614
afccc64eea73
ui: use I/O descriptors internally
Idan Kamara <idankk86@gmail.com>
parents:
14494
diff
changeset
|
19 self.ui = ui |
11595
368cd5325348
protocol: move hgweb protocol support back into protocol.py
Matt Mackall <mpm@selenic.com>
parents:
11594
diff
changeset
|
20 def getargs(self, args): |
14093
ce99d887585f
httprepo: long arguments support (issue2126)
Steven Brown <StevenGBrown@gmail.com>
parents:
13721
diff
changeset
|
21 knownargs = self._args() |
11595
368cd5325348
protocol: move hgweb protocol support back into protocol.py
Matt Mackall <mpm@selenic.com>
parents:
11594
diff
changeset
|
22 data = {} |
368cd5325348
protocol: move hgweb protocol support back into protocol.py
Matt Mackall <mpm@selenic.com>
parents:
11594
diff
changeset
|
23 keys = args.split() |
368cd5325348
protocol: move hgweb protocol support back into protocol.py
Matt Mackall <mpm@selenic.com>
parents:
11594
diff
changeset
|
24 for k in keys: |
368cd5325348
protocol: move hgweb protocol support back into protocol.py
Matt Mackall <mpm@selenic.com>
parents:
11594
diff
changeset
|
25 if k == '*': |
368cd5325348
protocol: move hgweb protocol support back into protocol.py
Matt Mackall <mpm@selenic.com>
parents:
11594
diff
changeset
|
26 star = {} |
14093
ce99d887585f
httprepo: long arguments support (issue2126)
Steven Brown <StevenGBrown@gmail.com>
parents:
13721
diff
changeset
|
27 for key in knownargs.keys(): |
13721
3458c15ab2f0
wireproto: fix handling of '*' args for HTTP and SSH
Peter Arrenbrecht <peter.arrenbrecht@gmail.com>
parents:
12704
diff
changeset
|
28 if key != 'cmd' and key not in keys: |
14093
ce99d887585f
httprepo: long arguments support (issue2126)
Steven Brown <StevenGBrown@gmail.com>
parents:
13721
diff
changeset
|
29 star[key] = knownargs[key][0] |
11595
368cd5325348
protocol: move hgweb protocol support back into protocol.py
Matt Mackall <mpm@selenic.com>
parents:
11594
diff
changeset
|
30 data['*'] = star |
368cd5325348
protocol: move hgweb protocol support back into protocol.py
Matt Mackall <mpm@selenic.com>
parents:
11594
diff
changeset
|
31 else: |
14093
ce99d887585f
httprepo: long arguments support (issue2126)
Steven Brown <StevenGBrown@gmail.com>
parents:
13721
diff
changeset
|
32 data[k] = knownargs[k][0] |
11595
368cd5325348
protocol: move hgweb protocol support back into protocol.py
Matt Mackall <mpm@selenic.com>
parents:
11594
diff
changeset
|
33 return [data[k] for k in keys] |
14093
ce99d887585f
httprepo: long arguments support (issue2126)
Steven Brown <StevenGBrown@gmail.com>
parents:
13721
diff
changeset
|
34 def _args(self): |
ce99d887585f
httprepo: long arguments support (issue2126)
Steven Brown <StevenGBrown@gmail.com>
parents:
13721
diff
changeset
|
35 args = self.req.form.copy() |
ce99d887585f
httprepo: long arguments support (issue2126)
Steven Brown <StevenGBrown@gmail.com>
parents:
13721
diff
changeset
|
36 chunks = [] |
14094
d10c6835497e
http: minor tweaks to long arg handling
Matt Mackall <mpm@selenic.com>
parents:
14093
diff
changeset
|
37 i = 1 |
14494
1ffeeb91c55d
check-code: flag 0/1 used as constant Boolean expression
Martin Geisler <mg@lazybytes.net>
parents:
14094
diff
changeset
|
38 while True: |
14094
d10c6835497e
http: minor tweaks to long arg handling
Matt Mackall <mpm@selenic.com>
parents:
14093
diff
changeset
|
39 h = self.req.env.get('HTTP_X_HGARG_' + str(i)) |
14093
ce99d887585f
httprepo: long arguments support (issue2126)
Steven Brown <StevenGBrown@gmail.com>
parents:
13721
diff
changeset
|
40 if h is None: |
ce99d887585f
httprepo: long arguments support (issue2126)
Steven Brown <StevenGBrown@gmail.com>
parents:
13721
diff
changeset
|
41 break |
ce99d887585f
httprepo: long arguments support (issue2126)
Steven Brown <StevenGBrown@gmail.com>
parents:
13721
diff
changeset
|
42 chunks += [h] |
14094
d10c6835497e
http: minor tweaks to long arg handling
Matt Mackall <mpm@selenic.com>
parents:
14093
diff
changeset
|
43 i += 1 |
14093
ce99d887585f
httprepo: long arguments support (issue2126)
Steven Brown <StevenGBrown@gmail.com>
parents:
13721
diff
changeset
|
44 args.update(cgi.parse_qs(''.join(chunks), keep_blank_values=True)) |
ce99d887585f
httprepo: long arguments support (issue2126)
Steven Brown <StevenGBrown@gmail.com>
parents:
13721
diff
changeset
|
45 return args |
11621
e46a8b2331a6
protocol: shuffle server methods to group send methods
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
11618
diff
changeset
|
46 def getfile(self, fp): |
e46a8b2331a6
protocol: shuffle server methods to group send methods
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
11618
diff
changeset
|
47 length = int(self.req.env['CONTENT_LENGTH']) |
e46a8b2331a6
protocol: shuffle server methods to group send methods
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
11618
diff
changeset
|
48 for s in util.filechunkiter(self.req, limit=length): |
e46a8b2331a6
protocol: shuffle server methods to group send methods
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
11618
diff
changeset
|
49 fp.write(s) |
e46a8b2331a6
protocol: shuffle server methods to group send methods
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
11618
diff
changeset
|
50 def redirect(self): |
14614
afccc64eea73
ui: use I/O descriptors internally
Idan Kamara <idankk86@gmail.com>
parents:
14494
diff
changeset
|
51 self.oldio = self.ui.fout, self.ui.ferr |
afccc64eea73
ui: use I/O descriptors internally
Idan Kamara <idankk86@gmail.com>
parents:
14494
diff
changeset
|
52 self.ui.ferr = self.ui.fout = cStringIO.StringIO() |
afccc64eea73
ui: use I/O descriptors internally
Idan Kamara <idankk86@gmail.com>
parents:
14494
diff
changeset
|
53 def restore(self): |
afccc64eea73
ui: use I/O descriptors internally
Idan Kamara <idankk86@gmail.com>
parents:
14494
diff
changeset
|
54 val = self.ui.fout.getvalue() |
afccc64eea73
ui: use I/O descriptors internally
Idan Kamara <idankk86@gmail.com>
parents:
14494
diff
changeset
|
55 self.ui.ferr, self.ui.fout = self.oldio |
afccc64eea73
ui: use I/O descriptors internally
Idan Kamara <idankk86@gmail.com>
parents:
14494
diff
changeset
|
56 return val |
11623
31d0a6d50ee2
protocol: extract compression from streaming mechanics
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
11622
diff
changeset
|
57 def groupchunks(self, cg): |
11595
368cd5325348
protocol: move hgweb protocol support back into protocol.py
Matt Mackall <mpm@selenic.com>
parents:
11594
diff
changeset
|
58 z = zlib.compressobj() |
14494
1ffeeb91c55d
check-code: flag 0/1 used as constant Boolean expression
Martin Geisler <mg@lazybytes.net>
parents:
14094
diff
changeset
|
59 while True: |
11595
368cd5325348
protocol: move hgweb protocol support back into protocol.py
Matt Mackall <mpm@selenic.com>
parents:
11594
diff
changeset
|
60 chunk = cg.read(4096) |
368cd5325348
protocol: move hgweb protocol support back into protocol.py
Matt Mackall <mpm@selenic.com>
parents:
11594
diff
changeset
|
61 if not chunk: |
368cd5325348
protocol: move hgweb protocol support back into protocol.py
Matt Mackall <mpm@selenic.com>
parents:
11594
diff
changeset
|
62 break |
11623
31d0a6d50ee2
protocol: extract compression from streaming mechanics
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
11622
diff
changeset
|
63 yield z.compress(chunk) |
31d0a6d50ee2
protocol: extract compression from streaming mechanics
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
11622
diff
changeset
|
64 yield z.flush() |
11595
368cd5325348
protocol: move hgweb protocol support back into protocol.py
Matt Mackall <mpm@selenic.com>
parents:
11594
diff
changeset
|
65 def _client(self): |
368cd5325348
protocol: move hgweb protocol support back into protocol.py
Matt Mackall <mpm@selenic.com>
parents:
11594
diff
changeset
|
66 return 'remote:%s:%s:%s' % ( |
368cd5325348
protocol: move hgweb protocol support back into protocol.py
Matt Mackall <mpm@selenic.com>
parents:
11594
diff
changeset
|
67 self.req.env.get('wsgi.url_scheme') or 'http', |
368cd5325348
protocol: move hgweb protocol support back into protocol.py
Matt Mackall <mpm@selenic.com>
parents:
11594
diff
changeset
|
68 urllib.quote(self.req.env.get('REMOTE_HOST', '')), |
368cd5325348
protocol: move hgweb protocol support back into protocol.py
Matt Mackall <mpm@selenic.com>
parents:
11594
diff
changeset
|
69 urllib.quote(self.req.env.get('REMOTE_USER', ''))) |
368cd5325348
protocol: move hgweb protocol support back into protocol.py
Matt Mackall <mpm@selenic.com>
parents:
11594
diff
changeset
|
70 |
368cd5325348
protocol: move hgweb protocol support back into protocol.py
Matt Mackall <mpm@selenic.com>
parents:
11594
diff
changeset
|
71 def iscmd(cmd): |
368cd5325348
protocol: move hgweb protocol support back into protocol.py
Matt Mackall <mpm@selenic.com>
parents:
11594
diff
changeset
|
72 return cmd in wireproto.commands |
368cd5325348
protocol: move hgweb protocol support back into protocol.py
Matt Mackall <mpm@selenic.com>
parents:
11594
diff
changeset
|
73 |
368cd5325348
protocol: move hgweb protocol support back into protocol.py
Matt Mackall <mpm@selenic.com>
parents:
11594
diff
changeset
|
74 def call(repo, req, cmd): |
14614
afccc64eea73
ui: use I/O descriptors internally
Idan Kamara <idankk86@gmail.com>
parents:
14494
diff
changeset
|
75 p = webproto(req, repo.ui) |
11625
cdeb861335d5
protocol: wrap non-string protocol responses in classes
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
11623
diff
changeset
|
76 rsp = wireproto.dispatch(repo, p, cmd) |
11626
2f8adc60e013
protocol: use generators instead of req.write() for hgweb stream responses
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
11625
diff
changeset
|
77 if isinstance(rsp, str): |
18352
e33b9b92a200
hgweb: pass the actual response body to request.response, not just the length
Mads Kiilerich <mads@kiilerich.com>
parents:
18346
diff
changeset
|
78 req.respond(HTTP_OK, HGTYPE, body=rsp) |
e33b9b92a200
hgweb: pass the actual response body to request.response, not just the length
Mads Kiilerich <mads@kiilerich.com>
parents:
18346
diff
changeset
|
79 return [] |
11626
2f8adc60e013
protocol: use generators instead of req.write() for hgweb stream responses
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
11625
diff
changeset
|
80 elif isinstance(rsp, wireproto.streamres): |
2f8adc60e013
protocol: use generators instead of req.write() for hgweb stream responses
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
11625
diff
changeset
|
81 req.respond(HTTP_OK, HGTYPE) |
2f8adc60e013
protocol: use generators instead of req.write() for hgweb stream responses
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
11625
diff
changeset
|
82 return rsp.gen |
2f8adc60e013
protocol: use generators instead of req.write() for hgweb stream responses
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
11625
diff
changeset
|
83 elif isinstance(rsp, wireproto.pushres): |
14614
afccc64eea73
ui: use I/O descriptors internally
Idan Kamara <idankk86@gmail.com>
parents:
14494
diff
changeset
|
84 val = p.restore() |
18346
6c2563b2c1c6
hgweb: use Content-Length for pushres
Mads Kiilerich <mads@kiilerich.com>
parents:
15017
diff
changeset
|
85 rsp = '%d\n%s' % (rsp.res, val) |
18352
e33b9b92a200
hgweb: pass the actual response body to request.response, not just the length
Mads Kiilerich <mads@kiilerich.com>
parents:
18346
diff
changeset
|
86 req.respond(HTTP_OK, HGTYPE, body=rsp) |
e33b9b92a200
hgweb: pass the actual response body to request.response, not just the length
Mads Kiilerich <mads@kiilerich.com>
parents:
18346
diff
changeset
|
87 return [] |
12703
40bb5853fc4b
wireproto: introduce pusherr() to deal with "unsynced changes" error
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents:
11626
diff
changeset
|
88 elif isinstance(rsp, wireproto.pusherr): |
12704
ca6e2adc3e4d
wireproto/http: drain the incoming bundle in case of errors
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents:
12703
diff
changeset
|
89 # drain the incoming bundle |
ca6e2adc3e4d
wireproto/http: drain the incoming bundle in case of errors
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents:
12703
diff
changeset
|
90 req.drain() |
14614
afccc64eea73
ui: use I/O descriptors internally
Idan Kamara <idankk86@gmail.com>
parents:
14494
diff
changeset
|
91 p.restore() |
12703
40bb5853fc4b
wireproto: introduce pusherr() to deal with "unsynced changes" error
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents:
11626
diff
changeset
|
92 rsp = '0\n%s\n' % rsp.res |
18352
e33b9b92a200
hgweb: pass the actual response body to request.response, not just the length
Mads Kiilerich <mads@kiilerich.com>
parents:
18346
diff
changeset
|
93 req.respond(HTTP_OK, HGTYPE, body=rsp) |
e33b9b92a200
hgweb: pass the actual response body to request.response, not just the length
Mads Kiilerich <mads@kiilerich.com>
parents:
18346
diff
changeset
|
94 return [] |
15017
f4522df38c65
wireproto: add out-of-band error class to allow remote repo to report errors
Andrew Pritchard <andrewp@fogcreek.com>
parents:
14614
diff
changeset
|
95 elif isinstance(rsp, wireproto.ooberror): |
f4522df38c65
wireproto: add out-of-band error class to allow remote repo to report errors
Andrew Pritchard <andrewp@fogcreek.com>
parents:
14614
diff
changeset
|
96 rsp = rsp.message |
18352
e33b9b92a200
hgweb: pass the actual response body to request.response, not just the length
Mads Kiilerich <mads@kiilerich.com>
parents:
18346
diff
changeset
|
97 req.respond(HTTP_OK, HGERRTYPE, body=rsp) |
e33b9b92a200
hgweb: pass the actual response body to request.response, not just the length
Mads Kiilerich <mads@kiilerich.com>
parents:
18346
diff
changeset
|
98 return [] |