Mercurial: Changelog

wireproto: define content negotiation for HTTPv2 HTTP messages communicate their media types and what media types they can understand via the Content-Type and Accept header, respectively. While I don't want the wire protocol to lean too heavily on HTTP because I'm aiming for the wire protocol to be as transport agnostic as possible, it is nice to play by the spec if possible. This commit defines our media negotiation mechanism for version 2 of the HTTP protocol. Essentially, we mandate the use of a new media type and how clients and servers should react to various headers or lack thereof. The name of the media type is a placeholder. We purposefully don't yet define the format of the new media type because that's a lot of work. I feel pretty strongly that we should use Content-Type. I feel less strongly about Accept. I think it is reasonable for servers to return the media type that was submitted to them. So we may strike this header before the protocol is finished... Differential Revision: https://phab.mercurial-scm.org/D2850

hgweb: also set Content-Type header Our HTTP/WSGI server may convert the Content-Type HTTP request header to the CONTENT_TYPE WSGI environment key and not set HTTP_CONTENT_TYPE. Other WSGI server implementations do this, so I think the behavior is acceptable. So assuming this HTTP request header could get "lost" by the WSGI server, let's restore it on the request object like we do for Content-Length. FWIW, the WSGI server may also *invent* a Content-Type value. The default behavior of Python's RFC 822 message class returns a default media type if Content-Type isn't defined. This is kind of annoying. But RFC 7231 section 3.1.1.5 does say the recipient may assume a media type of application/octet-stream. Python's defaults are for text/plain (given we're using an RFC 822 parser). But whatever. Differential Revision: https://phab.mercurial-scm.org/D2849

wireproto: require POST for all HTTPv2 requests Wire protocol version 1 transfers argument data via request headers by default. This has historically caused problems because servers institute limits on the length of individual HTTP headers as well as the total size of all request headers. Mercurial servers can advertise the maximum length of an individual header. But there's no guarantee any intermediate HTTP agents will accept headers up to that length. In the existing wire protocol, server operators typically also key off the HTTP request method to implement authentication. For example, GET requests translate to read-only requests and can be allowed. But read-write commands must use POST and require authentication. This has typically worked because the only wire protocol commands that use POST modify the repo (e.g. the "unbundle" command). There is an experimental feature to enable clients to transmit argument data via POST request bodies. This is technically a better and more robust solution. But we can't enable it by default because of servers assuming POST means write access. In version 2 of the wire protocol, the permissions of a request are encoded in the URL. And with it being a new protocol in a new URL space, we're not constrained by backwards compatibility requirements. This commit adopts the technically superior mechanism of using HTTP request bodies to send argument data by requiring POST for all commands. Strictly speaking, it may be possible to send request bodies on GET requests. But my experience is that not all HTTP stacks support this. POST pretty much always works. Using POST for read-only operations does sacrifice some RESTful design purity. But this API cares about practicality, not about being in Roy T. Fielding's REST ivory tower. There's a chance we may relax this restriction in the future. But for now, I want to see how far we can get with a POST only API. Differential Revision: https://phab.mercurial-scm.org/D2837

wireproto: define permissions-based routing of HTTPv2 wire protocol Now that we have a scaffolding for serving version 2 of the HTTP protocol, let's start implementing it. A good place to start is URL routing and basic request processing semantics. We can focus on content types, capabilities detect, etc later. Version 2 of the HTTP wire protocol encodes the needed permissions of the request in the URL path. The reasons for this are documented in the added documentation. In short, a) it makes it really easy and fail proof for server administrators to implement path-based authentication and b) it will enable clients to realize very early in a server exchange that authentication will be required to complete the operation. This latter point avoids all kinds of complexity and problems, like dealing with Expect: 100-continue and clients finding out later during `hg push` that they need to provide authentication. This will avoid the current badness where clients send a full bundle, get an HTTP 403, provide authentication, then retransmit the bundle. In order to implement command checking, we needed to implement a protocol handler for the new wire protocol. Our handler is just small enough to run the code we've implemented. Tests for the defined functionality have been added. I very much want to refactor the permissions checking code and define a better response format. But this can be done later. Nothing is covered by backwards compatibility at this point. Differential Revision: https://phab.mercurial-scm.org/D2836

wireproto: support /api/* URL space for exposing APIs I will soon be introducing a new version of the HTTP wire protocol. One of the things I want to change with it is the URL routing. I want to rely on URL paths to define endpoints rather than the "cmd" query string argument. That should be pretty straightforward. I was thinking about what URL space to reserve for the new protocol. We /could/ put everything at a top-level path. e.g. /wireproto/* or /http-v2-wireproto/*. However, these constrain us a bit because they assume there will only be 1 API: version 2 of the HTTP wire protocol. I think there is room to grow multiple APIs. For example, there may someday be a proper JSON API to query or even manipulate the repository. And I don't think we should have to create a new top-level URL space for each API nor should we attempt to shoehorn each future API into the same shared URL space: that would just be too chaotic. This commits reserves the /api/* URL space for all our future API needs. Essentially, all requests to /api/* get routed to a new WSGI handler. By default, it 404's the entire URL space unless the "api server" feature is enabled. When enabled, requests to "/api" list available APIs. URLs of the form /api/<name>/* are reserved for a particular named API. Behavior within each API is left up to that API. So, we can grow new APIs easily without worrying about URL space conflicts. APIs can be registered by adding entries to a global dict. This allows extensions to provide their own APIs should they choose to do so. This is probably a premature feature. But IMO the code is easier to read if we're not dealing with API-specific behavior like config option querying inline. To prove it works, we implement a very basic API for version 2 of the HTTP wire protocol. It does nothing of value except facilitate testing of the /api/* URL space. We currently emit plain text responses for all /api/* endpoints. There's definitely room to look at Accept and other request headers to vary the response format. But we have to start somewhere. Differential Revision: https://phab.mercurial-scm.org/D2834

url: support suppressing Accept header Sending this header automatically could interfere with future testing and client behavior. Let's add a knob to disable the behavior. We don't have a control for User-Agent because urllib will send it if we don't set something. I don't feel like hacking into the bowels of urllib to figure out how to suppress that. UA shouldn't be used for anything meaningful. So it shouldn't pose any problems beyond non-determinism (since the header has the Mercurial version in it). Differential Revision: https://phab.mercurial-scm.org/D2843

util: don't log low-level I/O calls for HTTP peer `hg debugwireproto` is useful for testing HTTP interactions. Possibly more useful than `get-with-headers.py`. But one thing that makes it annoying for mid-level tests is that it logs all API calls, such as readline(). This makes output - especially headers - overly verbose. This commit teaches our file and socket observers to not log API calls on functions dealing with data. We change the behavior of `hg debugwireproto` to enable this mode by default. --debug can be added to restore the previous behavior. As the test changes demonstrate, this makes tests much easier to read. As a bonus, it also removes some required (glob) over lengths in system call results. One thing that's lacking is knowing which side sent data. But we can fix this in a follow-up once it becomes a problem. Differential Revision: https://phab.mercurial-scm.org/D2842