annotate tests/test-wireproto.py @ 37048:fc5e261915b9

wireproto: require POST for all HTTPv2 requests Wire protocol version 1 transfers argument data via request headers by default. This has historically caused problems because servers institute limits on the length of individual HTTP headers as well as the total size of all request headers. Mercurial servers can advertise the maximum length of an individual header. But there's no guarantee any intermediate HTTP agents will accept headers up to that length. In the existing wire protocol, server operators typically also key off the HTTP request method to implement authentication. For example, GET requests translate to read-only requests and can be allowed. But read-write commands must use POST and require authentication. This has typically worked because the only wire protocol commands that use POST modify the repo (e.g. the "unbundle" command). There is an experimental feature to enable clients to transmit argument data via POST request bodies. This is technically a better and more robust solution. But we can't enable it by default because of servers assuming POST means write access. In version 2 of the wire protocol, the permissions of a request are encoded in the URL. And with it being a new protocol in a new URL space, we're not constrained by backwards compatibility requirements. This commit adopts the technically superior mechanism of using HTTP request bodies to send argument data by requiring POST for all commands. Strictly speaking, it may be possible to send request bodies on GET requests. But my experience is that not all HTTP stacks support this. POST pretty much always works. Using POST for read-only operations does sacrifice some RESTful design purity. But this API cares about practicality, not about being in Roy T. Fielding's REST ivory tower. There's a chance we may relax this restriction in the future. But for now, I want to see how far we can get with a POST only API. Differential Revision: https://phab.mercurial-scm.org/D2837
author Gregory Szorc <gregory.szorc@gmail.com>
date Tue, 13 Mar 2018 11:57:43 -0700
parents 65615c29e74f
children 45b39c69fae0
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
28675
fcafd84bc9c5 py3: make test-wireproto use print_function
Pulkit Goyal <7895pulkit@gmail.com>
parents: 28438
diff changeset
1 from __future__ import absolute_import, print_function
27301
5defcb7d6e5f tests: use absolulte_import in test-wireproto.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25912
diff changeset
2
28860
50d11dd8ac02 py3: use multi-line import in test-wireproto.py
timeless <timeless@mozdev.org>
parents: 28675
diff changeset
3 from mercurial import (
36074
2f7290555c96 wireproto: introduce type for raw byte responses (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33806
diff changeset
4 error,
36555
c28de190e20a py3: port tests/test-wireproto.py to Python 3
Pulkit Goyal <7895pulkit@gmail.com>
parents: 36074
diff changeset
5 pycompat,
36944
65615c29e74f tests: fix test-wireproto.py to work around serverrepo() not having a ui
Augie Fackler <augie@google.com>
parents: 36801
diff changeset
6 ui as uimod,
28861
86db5cb55d46 pycompat: switch to util.stringio for py3 compat
timeless <timeless@mozdev.org>
parents: 28860
diff changeset
7 util,
28860
50d11dd8ac02 py3: use multi-line import in test-wireproto.py
timeless <timeless@mozdev.org>
parents: 28675
diff changeset
8 wireproto,
36074
2f7290555c96 wireproto: introduce type for raw byte responses (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33806
diff changeset
9 wireprototypes,
28860
50d11dd8ac02 py3: use multi-line import in test-wireproto.py
timeless <timeless@mozdev.org>
parents: 28675
diff changeset
10 )
28861
86db5cb55d46 pycompat: switch to util.stringio for py3 compat
timeless <timeless@mozdev.org>
parents: 28860
diff changeset
11 stringio = util.stringio
14622
bd88561afb4b wireproto: add batching support to wirerepository
Peter Arrenbrecht <peter.arrenbrecht@gmail.com>
parents:
diff changeset
12
14764
a7d5816087a9 classes: fix class style problems found by b071cd58af50
Thomas Arendsen Hein <thomas@intevation.de>
parents: 14622
diff changeset
13 class proto(object):
14622
bd88561afb4b wireproto: add batching support to wirerepository
Peter Arrenbrecht <peter.arrenbrecht@gmail.com>
parents:
diff changeset
14 def __init__(self, args):
bd88561afb4b wireproto: add batching support to wirerepository
Peter Arrenbrecht <peter.arrenbrecht@gmail.com>
parents:
diff changeset
15 self.args = args
bd88561afb4b wireproto: add batching support to wirerepository
Peter Arrenbrecht <peter.arrenbrecht@gmail.com>
parents:
diff changeset
16 def getargs(self, spec):
bd88561afb4b wireproto: add batching support to wirerepository
Peter Arrenbrecht <peter.arrenbrecht@gmail.com>
parents:
diff changeset
17 args = self.args
36555
c28de190e20a py3: port tests/test-wireproto.py to Python 3
Pulkit Goyal <7895pulkit@gmail.com>
parents: 36074
diff changeset
18 args.setdefault(b'*', {})
14622
bd88561afb4b wireproto: add batching support to wirerepository
Peter Arrenbrecht <peter.arrenbrecht@gmail.com>
parents:
diff changeset
19 names = spec.split()
bd88561afb4b wireproto: add batching support to wirerepository
Peter Arrenbrecht <peter.arrenbrecht@gmail.com>
parents:
diff changeset
20 return [args[n] for n in names]
bd88561afb4b wireproto: add batching support to wirerepository
Peter Arrenbrecht <peter.arrenbrecht@gmail.com>
parents:
diff changeset
21
36801
66de4555cefd wireproto: formalize permissions checking as part of protocol interface
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36555
diff changeset
22 def checkperm(self, perm):
66de4555cefd wireproto: formalize permissions checking as part of protocol interface
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36555
diff changeset
23 pass
66de4555cefd wireproto: formalize permissions checking as part of protocol interface
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36555
diff changeset
24
17192
1ac628cd7113 peer: introduce real peer classes
Peter Arrenbrecht <peter.arrenbrecht@gmail.com>
parents: 14765
diff changeset
25 class clientpeer(wireproto.wirepeer):
36944
65615c29e74f tests: fix test-wireproto.py to work around serverrepo() not having a ui
Augie Fackler <augie@google.com>
parents: 36801
diff changeset
26 def __init__(self, serverrepo, ui):
14622
bd88561afb4b wireproto: add batching support to wirerepository
Peter Arrenbrecht <peter.arrenbrecht@gmail.com>
parents:
diff changeset
27 self.serverrepo = serverrepo
36944
65615c29e74f tests: fix test-wireproto.py to work around serverrepo() not having a ui
Augie Fackler <augie@google.com>
parents: 36801
diff changeset
28 self._ui = ui
25912
cbbdd085c991 batching: migrate basic noop batching into peer.peer
Augie Fackler <augie@google.com>
parents: 25708
diff changeset
29
33806
dedab036215d wireproto: use new peer interface
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33767
diff changeset
30 @property
dedab036215d wireproto: use new peer interface
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33767
diff changeset
31 def ui(self):
36944
65615c29e74f tests: fix test-wireproto.py to work around serverrepo() not having a ui
Augie Fackler <augie@google.com>
parents: 36801
diff changeset
32 return self._ui
33806
dedab036215d wireproto: use new peer interface
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33767
diff changeset
33
dedab036215d wireproto: use new peer interface
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33767
diff changeset
34 def url(self):
36555
c28de190e20a py3: port tests/test-wireproto.py to Python 3
Pulkit Goyal <7895pulkit@gmail.com>
parents: 36074
diff changeset
35 return b'test'
33806
dedab036215d wireproto: use new peer interface
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33767
diff changeset
36
dedab036215d wireproto: use new peer interface
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33767
diff changeset
37 def local(self):
dedab036215d wireproto: use new peer interface
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33767
diff changeset
38 return None
dedab036215d wireproto: use new peer interface
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33767
diff changeset
39
dedab036215d wireproto: use new peer interface
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33767
diff changeset
40 def peer(self):
dedab036215d wireproto: use new peer interface
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33767
diff changeset
41 return self
dedab036215d wireproto: use new peer interface
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33767
diff changeset
42
dedab036215d wireproto: use new peer interface
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33767
diff changeset
43 def canpush(self):
dedab036215d wireproto: use new peer interface
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33767
diff changeset
44 return True
dedab036215d wireproto: use new peer interface
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33767
diff changeset
45
dedab036215d wireproto: use new peer interface
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33767
diff changeset
46 def close(self):
dedab036215d wireproto: use new peer interface
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33767
diff changeset
47 pass
dedab036215d wireproto: use new peer interface
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33767
diff changeset
48
dedab036215d wireproto: use new peer interface
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33767
diff changeset
49 def capabilities(self):
36555
c28de190e20a py3: port tests/test-wireproto.py to Python 3
Pulkit Goyal <7895pulkit@gmail.com>
parents: 36074
diff changeset
50 return [b'batch']
25912
cbbdd085c991 batching: migrate basic noop batching into peer.peer
Augie Fackler <augie@google.com>
parents: 25708
diff changeset
51
14622
bd88561afb4b wireproto: add batching support to wirerepository
Peter Arrenbrecht <peter.arrenbrecht@gmail.com>
parents:
diff changeset
52 def _call(self, cmd, **args):
36555
c28de190e20a py3: port tests/test-wireproto.py to Python 3
Pulkit Goyal <7895pulkit@gmail.com>
parents: 36074
diff changeset
53 args = pycompat.byteskwargs(args)
36074
2f7290555c96 wireproto: introduce type for raw byte responses (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33806
diff changeset
54 res = wireproto.dispatch(self.serverrepo, proto(args), cmd)
2f7290555c96 wireproto: introduce type for raw byte responses (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33806
diff changeset
55 if isinstance(res, wireprototypes.bytesresponse):
2f7290555c96 wireproto: introduce type for raw byte responses (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33806
diff changeset
56 return res.data
2f7290555c96 wireproto: introduce type for raw byte responses (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33806
diff changeset
57 elif isinstance(res, bytes):
2f7290555c96 wireproto: introduce type for raw byte responses (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33806
diff changeset
58 return res
2f7290555c96 wireproto: introduce type for raw byte responses (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33806
diff changeset
59 else:
2f7290555c96 wireproto: introduce type for raw byte responses (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33806
diff changeset
60 raise error.Abort('dummy client does not support response type')
14622
bd88561afb4b wireproto: add batching support to wirerepository
Peter Arrenbrecht <peter.arrenbrecht@gmail.com>
parents:
diff changeset
61
28438
48fd02dac1d4 wireproto: make iterbatcher behave streamily over http(s)
Augie Fackler <augie@google.com>
parents: 27301
diff changeset
62 def _callstream(self, cmd, **args):
28861
86db5cb55d46 pycompat: switch to util.stringio for py3 compat
timeless <timeless@mozdev.org>
parents: 28860
diff changeset
63 return stringio(self._call(cmd, **args))
28438
48fd02dac1d4 wireproto: make iterbatcher behave streamily over http(s)
Augie Fackler <augie@google.com>
parents: 27301
diff changeset
64
14622
bd88561afb4b wireproto: add batching support to wirerepository
Peter Arrenbrecht <peter.arrenbrecht@gmail.com>
parents:
diff changeset
65 @wireproto.batchable
bd88561afb4b wireproto: add batching support to wirerepository
Peter Arrenbrecht <peter.arrenbrecht@gmail.com>
parents:
diff changeset
66 def greet(self, name):
bd88561afb4b wireproto: add batching support to wirerepository
Peter Arrenbrecht <peter.arrenbrecht@gmail.com>
parents:
diff changeset
67 f = wireproto.future()
36555
c28de190e20a py3: port tests/test-wireproto.py to Python 3
Pulkit Goyal <7895pulkit@gmail.com>
parents: 36074
diff changeset
68 yield {b'name': mangle(name)}, f
14622
bd88561afb4b wireproto: add batching support to wirerepository
Peter Arrenbrecht <peter.arrenbrecht@gmail.com>
parents:
diff changeset
69 yield unmangle(f.value)
bd88561afb4b wireproto: add batching support to wirerepository
Peter Arrenbrecht <peter.arrenbrecht@gmail.com>
parents:
diff changeset
70
14764
a7d5816087a9 classes: fix class style problems found by b071cd58af50
Thomas Arendsen Hein <thomas@intevation.de>
parents: 14622
diff changeset
71 class serverrepo(object):
14622
bd88561afb4b wireproto: add batching support to wirerepository
Peter Arrenbrecht <peter.arrenbrecht@gmail.com>
parents:
diff changeset
72 def greet(self, name):
36555
c28de190e20a py3: port tests/test-wireproto.py to Python 3
Pulkit Goyal <7895pulkit@gmail.com>
parents: 36074
diff changeset
73 return b"Hello, " + name
14622
bd88561afb4b wireproto: add batching support to wirerepository
Peter Arrenbrecht <peter.arrenbrecht@gmail.com>
parents:
diff changeset
74
18278
753acee7d6dd clfilter: make localpeer use a repo with "unserved" filter
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 17192
diff changeset
75 def filtered(self, name):
753acee7d6dd clfilter: make localpeer use a repo with "unserved" filter
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 17192
diff changeset
76 return self
753acee7d6dd clfilter: make localpeer use a repo with "unserved" filter
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 17192
diff changeset
77
14622
bd88561afb4b wireproto: add batching support to wirerepository
Peter Arrenbrecht <peter.arrenbrecht@gmail.com>
parents:
diff changeset
78 def mangle(s):
36555
c28de190e20a py3: port tests/test-wireproto.py to Python 3
Pulkit Goyal <7895pulkit@gmail.com>
parents: 36074
diff changeset
79 return b''.join(pycompat.bytechr(ord(c) + 1) for c in pycompat.bytestr(s))
14622
bd88561afb4b wireproto: add batching support to wirerepository
Peter Arrenbrecht <peter.arrenbrecht@gmail.com>
parents:
diff changeset
80 def unmangle(s):
36555
c28de190e20a py3: port tests/test-wireproto.py to Python 3
Pulkit Goyal <7895pulkit@gmail.com>
parents: 36074
diff changeset
81 return b''.join(pycompat.bytechr(ord(c) - 1) for c in pycompat.bytestr(s))
14622
bd88561afb4b wireproto: add batching support to wirerepository
Peter Arrenbrecht <peter.arrenbrecht@gmail.com>
parents:
diff changeset
82
bd88561afb4b wireproto: add batching support to wirerepository
Peter Arrenbrecht <peter.arrenbrecht@gmail.com>
parents:
diff changeset
83 def greet(repo, proto, name):
bd88561afb4b wireproto: add batching support to wirerepository
Peter Arrenbrecht <peter.arrenbrecht@gmail.com>
parents:
diff changeset
84 return mangle(repo.greet(unmangle(name)))
bd88561afb4b wireproto: add batching support to wirerepository
Peter Arrenbrecht <peter.arrenbrecht@gmail.com>
parents:
diff changeset
85
36555
c28de190e20a py3: port tests/test-wireproto.py to Python 3
Pulkit Goyal <7895pulkit@gmail.com>
parents: 36074
diff changeset
86 wireproto.commands[b'greet'] = (greet, b'name',)
14622
bd88561afb4b wireproto: add batching support to wirerepository
Peter Arrenbrecht <peter.arrenbrecht@gmail.com>
parents:
diff changeset
87
bd88561afb4b wireproto: add batching support to wirerepository
Peter Arrenbrecht <peter.arrenbrecht@gmail.com>
parents:
diff changeset
88 srv = serverrepo()
36944
65615c29e74f tests: fix test-wireproto.py to work around serverrepo() not having a ui
Augie Fackler <augie@google.com>
parents: 36801
diff changeset
89 clt = clientpeer(srv, uimod.ui())
14622
bd88561afb4b wireproto: add batching support to wirerepository
Peter Arrenbrecht <peter.arrenbrecht@gmail.com>
parents:
diff changeset
90
36555
c28de190e20a py3: port tests/test-wireproto.py to Python 3
Pulkit Goyal <7895pulkit@gmail.com>
parents: 36074
diff changeset
91 print(clt.greet(b"Foobar"))
33767
b47fe9733d76 peer: remove non iterating batcher (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 28861
diff changeset
92 b = clt.iterbatch()
36555
c28de190e20a py3: port tests/test-wireproto.py to Python 3
Pulkit Goyal <7895pulkit@gmail.com>
parents: 36074
diff changeset
93 list(map(b.greet, (b'Fo, =;:<o', b'Bar')))
14622
bd88561afb4b wireproto: add batching support to wirerepository
Peter Arrenbrecht <peter.arrenbrecht@gmail.com>
parents:
diff changeset
94 b.submit()
33767
b47fe9733d76 peer: remove non iterating batcher (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 28861
diff changeset
95 print([r for r in b.results()])