lfs: infer the blob store URL from an explicit pull source I don't see any easier way to do this because the update part of `hg pull -u` happens outside exchange.pull(), and commands.postincoming() doesn't take a path. So (ab)use the mechanism used by subrepos to redirect where subrepos are pulled from when an explicit path is given. As a bonus, this should allow lfs blobs to be pulled into a subrepo when it is checked out. An explicit push path can be handled within exchange.push(). That can be done next, outside of this dirty hack.

lfs: special case the null:// usercache instead of treating it as a url The previous code worked on Windows, but not on Unix, and a pending patch's test failed. The url being used was something like "/tmp/.../client1/null://", courtesy of ui.configpath(). Looking at the doc comment, this seems like it's maybe not the right function to call (why should a relative cache path be expanded relative to the repo root or config file?), but largefiles has been using it since 8b8dd13295db (Oct 2011). It was introduced in 1b591f9b7fd2 (Jan 2011) without comment or callers. A grep over the whole history shows that only largefiles used it until lfs and infinitepush came along recently. It looks like if the `if not os.path.isabs(v) or "://" not in v` in configpath() is changed to an 'and', both Linux and Windows are happy. I'm guessing that "://" is to pick off URLs, so that seems reasonable. But I'm not sure why it isn't explicitly "file://", and I thought that "file://foo" is relative anyway. (At least, there are doctests for file:///tmp in util.url.) There is no mention of this setting in the help, but it is referenced on the wiki page for largefiles. (There's no mention that this is intended to be a URL, and the example uses an absolute path.) I don't want this blocking the rest of the lfs server discovery stuff. It was also wrong to allow a file:// URL here, but not in largefiles.

tests: add tests showing pulling from infinitepush works over wire The current tests in test-infinitepush-ci.t showed that `hg pull -r <rev>` does not work. Digging in code, I found that we have logic for pulling from bundlestore without having client side logic. This patch adds test demonstrating that pulling from bundlestore works when working over wire. Pulling from bundlestore when the peer is a localpeer still does not works. Differential Revision: https://phab.mercurial-scm.org/D3072

fix: use a portable python script instead of sed in test Differential Revision: https://phab.mercurial-scm.org/D2988

py3: use pycompat.bytestr() where repr in involved Differential Revision: https://phab.mercurial-scm.org/D3244

httppeer: support protocol upgrade With the new handshake defined and in place on the server, we can now implement it on the client. The HTTP handshake mechanism has been taught to add headers advertising its support for the new capabilities response. Response handling has been adjusted to allow CBOR responses through. And makepeer() has been taught to instantiate a mutually supported peer. The HTTPv2 peer class doesn't implement the full peer interface. So HTTPv2 is not yet usable as a peer. Like the server side, we support registering handlers for different API services. This allows extensions to easily implement API services and peers. A practical use case for this is to provide a previous implementation of the experimental version 2 wire protocol to a future version of Mercurial. We know there will be BC breaks after 4.6 ships. But someone could take the peer and server code from 4.6, drop it in an extension, and allow its use indefinitely. Differential Revision: https://phab.mercurial-scm.org/D3243

wireproto: define and implement HTTP handshake to upgrade protocol When clients connect to repositories over HTTP, they issue a request to the well-known URL "?cmd=capabilities" to fetch the repository capabilities. This is the handshake portion of the HTTP protocol. This commit defines a mechanism to use that HTTP request to return information about modern server features. If a client sends an X-HgUpgrade-* header containing a list of client-supported API names, the server responds with a response containing information about available services. This includes the normal capabilities string. So if the server doesn't support any newer services, the client can easily fall back. By advertising supported services from clients, server operators can see and log what client support exists in the wild. This will also help with debugging. The response contains the base path to API services. We know there are potential issues with the <repo>/api/ URL space conflicting with hgwebdir and subrepos. By making the API URL dynamic from the perspective of the client, the URL for APIs is not subject to backwards compatibility concerns - at least as long as a ?cmd=capabilities request is made. We've also defined the ``cbor`` client capability for the X-HgProto-* header. This MUST be sent in order to get the modern response from "?cmd=capabilities". During implementation, I initially always sent an application/mercurial-cbor response. However, the handshake mechanism will be more future compatible if the client is in charge of which formats to request. We already perform content negotiation from X-HgProto-*, so keying off this for the capabilities response feels appropriate. In addition, I initially used application/cbor. However, it is conceivable that a non-Mercurial server could serve application/cbor. To rule out this possibility, I've invented a new media type that is Mercurial specific and can't be confused for generic CBOR. Differential Revision: https://phab.mercurial-scm.org/D3242

httppeer: only advertise partial-pull if capabilities are known We don't need to be advertising client protocol parameters as part of the capabilities request during the handshake because nothing in version 1 of the wire protocol will use this data. i.e. the advertisement is wasteful. Differential Revision: https://phab.mercurial-scm.org/D3241

httppeer: always add x-hg* headers to Vary header Before, we manually updated the Vary header value for each header contributing to it. All X-Hg* headers are reserved for the Mercurial protocol and could have caching implications. So it makes sense to always add these headers to Vary. A test revealed that X-HgArgs-Post wasn't being added to Vary. This is only sent on POST requests. POST requests generally aren't cacheable. However, it is possible if the server sends the appropriate headers. Mercurial shouldn't be sending those headers. But let's not take any chances. Differential Revision: https://phab.mercurial-scm.org/D3240

httppeer: don't accept very old media types (BC) Versions of Mercurial older than 1.0 emitted the text/plain and application/hg-changegroup media types in response to wire protocol commands. Way back in 8760d0c83b9b in 2005, the code validating these media types was added, presumably for backwards compatibility. 0b245edec124 a short time before that commit changed things from text/plain and application/hg-changegroup to application/mercurial-0.1 and application/hg-0.1. 8760d0c83b9b seemed to indicate ("for now") that the BC compatibility was temporary. But that code has lived until this day. It has been more than 10 years and nobody should be running pre 1.0 servers. Pretty much the only risk to this is if there's a server somewhere advertising the old media types or server software is interfering and not letting Mercurial send the proper Content-Type header. I think the chances are rare. The wire protocol docs were created (by me) from reading existing code. So the deletions don't constitute a spec change as much as reflecting the reality of how things have been for years. .. bc:: The HTTP client no longer accepts text/plain and application/hg-changegroup Content-Type values as a valid Mercurial command response. These should only be encountered on pre 1.0 Mercurial servers. Differential Revision: https://phab.mercurial-scm.org/D3239

httppeer: allow opener to be passed to makepeer() This allows us to use makepeer() in `hg debugwireproto`. Differential Revision: https://phab.mercurial-scm.org/D3238

httppeer: perform capabilities request in makepeer() Previously, we constructed an httppeer then always ran _fetchcaps() to issue the capabilities command. We want to issue the capabilities command before constructing a peer instance so we can construct an appropriate peer instance depending on the capabilities result. With the code for making and sending requests moved out of httppeer, it is now possible to send command requests without an httppeer. This commit creates a new function for making the capabilities request and calls it as part of makepeer(). This code should be functionality equivalent to what existed before. Differential Revision: https://phab.mercurial-scm.org/D3237

httppeer: extract common response handling into own function This allows the common redirect detection, content type validation, and decompression wrapping to be usable outside of httppeer instances. Differential Revision: https://phab.mercurial-scm.org/D3236

httppeer: move error handling and response wrapping into sendrequest This is common for all HTTP requests. It should be part of sendrequest(). Differential Revision: https://phab.mercurial-scm.org/D3235