logexchange: use command executor for wire protocol commands Differential Revision: https://phab.mercurial-scm.org/D3290

streamclone: use command executor for wire protocol commands Differential Revision: https://phab.mercurial-scm.org/D3289

discovery: use command executor interface We're trying to port all wire protocol code to use the new interface so we can implement wire protocol version 2 clients. Differential Revision: https://phab.mercurial-scm.org/D3288

discovery: don't redundantly call branchmap We were calling the remote command twice without mutation the remote in between. Derp. Differential Revision: https://phab.mercurial-scm.org/D3287

wireproto: convert legacy commands to command executor Calls to the legacy commands "changegroup" and "changegroupsubset" have been ported to the new command executor interface. Because we always pass arguments by name and not position, some inconsistent names throughout the code base have been unified. As part of this change, we no longer had any remaining callers of the legacy command methods {between, branches, changegroup, changegroupsubset}. So, these interfaces/methods have been dropped from peer interfaces. We still have an interface declaring these methods. But that interface is implemented on the concrete peer types and isn't part of the generic peer interface. (The implementations of the command executor continue to call these methods.) The ultimate goal is to remove the per-command methods from the generic peer interface: the only interface-conforming way to call a command will be with the new executor API. At some point, we may want to move the methods outside of the peer classes and change the executor implementations to not call methods directly on a peer instance. Differential Revision: https://phab.mercurial-scm.org/D3273

treediscovery: switch to command executor interface We now have a new interface for requesting that commands run. Switch to it. Differential Revision: https://phab.mercurial-scm.org/D3272

wireproto: remove iterbatch() from peer interface (API) Good riddance. Some tests have been ported to the new API. This probably should have been done in earlier commits. But duplicating the test coverage would have been difficult. It was easier this way. .. api:: The wire protocol peer's ``iterbatch()`` for bulk executing commands has been remove.d Use ``peer.commandexecutor()`` instead. Differential Revision: https://phab.mercurial-scm.org/D3271

largefiles: use command executor for batch operation This is the only other user of iterbatch() in core. Tests changed because the new command executor is smart enough to not send a "batch" command over the wire if only 1 command was requested. There is still coverage for the "batch" command in this test though. Differential Revision: https://phab.mercurial-scm.org/D3270

wireproto: implement batching on peer executor interface This is a bit more complicated than non-batch requests because we need to buffer sends until the last request arrives *and* we need to support resolving futures as data arrives from the remote. In a classical concurrent.futures executor model, the future "starts" as soon as it is submitted. However, we have nothing to start until the last command is submitted. If we did nothing, calling result() would deadlock, since the future hasn't "started." So in the case where we queue the command, we return a special future type whose result() will trigger sendcommands(). This eliminates the deadlock potential. It also serves as a check against callers who may be calling result() prematurely, as it will prevent any subsequent callcommands() from working. This behavior is slightly annoying and a bit restrictive. But it's the world that half duplex connections forces on us. In order to support streaming responses, we were previously using a generator. But with a futures-based API, we're using futures and not generators. So in order to get streaming, we need a background thread to read data from the server. The approach taken in this patch is to leverage the ThreadPoolExecutor from concurrent.futures for managing a background thread. We create an executor and future that resolves when all response data is processed (or an error occurs). When exiting the context manager, we wait on that background reading before returning. I was hoping we could manually spin up a threading.Thread and this would be simple. But I ran into a few deadlocks when implementing. After looking at the source code to concurrent.futures, I figured it would just be easier to use a ThreadPoolExecutor than implement all the code needed to manually manage a thread. To prove this works, a use of the batch API in discovery has been updated. Differential Revision: https://phab.mercurial-scm.org/D3269

wireproto: implement command executor interface for version 1 peers Now that we've defined our new interface for issuing commands, let's implement it. We add the interface to the base peer interface. This means all peer types must implement it. The only peer types that we have are the local peer in localrepo and a shared wire peer for version 1 of the wire protocol. The local peer implementation is pretty straightforward. We don't do anything fancy and just return a resolved future with the result of a method call. This is similar to what localiterbatcher does. The wire protocol version 1 implementation is a bit more complicated and is a more robust implementation. The wire executor queues commands by default. And because the new executor interface always allows multiple commands but not all version 1 commands are @batchable, it has to check that the requested commands are batchable if multiple commands are being requested. The wire executor currently only supports executing a single command. This is for simplicity reasons. Support for multiple commands will be added in a separate commit. To prove the new interface works, a call to the "known" command during discovery has been updated to use the new API. It's worth noting that both implementations require a method having the command name to exist on the peer. There is at least one caller in core that don't have a method calls peer._call() directly. We may need to shore up the requirements later... Differential Revision: https://phab.mercurial-scm.org/D3268

repository: define new interface for running commands Today, the peer interface exposes methods for each command that can be executed. In addition, there is an iterbatch() API that allows commands to be issued in batches and provides an iterator over the results. This is a glorified wrapper around the "batch" wire command. Wire protocol version 2 supports nicer things (such as batching any command and out-of-order replies). It will require a more flexible API for executing commands. This commit introduces a new peer interface for making command requests. In the new world, you can't simply call a method on the peer to execute a command: you need to obtain an object to be used for executing commands. That object can be used to issue a single command or it can batch multiple requests. In the case of full duplex peers, the command may even be sent out over the wire immediately. There are no per-command methods. Instead, there is a generic method to call a command. The implementation can then perform domain specific processing for specific commands. This includes passing data via a specially named argument. Arguments are also passed as a dictionary instead of using **kwargs. While **kwargs is nicer to use, we've historically gotten into trouble using it because there will inevitably be a conflict between the name of an argument to a wire protocol command and an argument we want to pass into a function. Instead of a command returning a value, it returns a future which will resolve to a value. This opens the door for out-of-order response handling and concurrent response handling in the version 2 protocol. Differential Revision: https://phab.mercurial-scm.org/D3267

pycompat: export a handle on concurrent.futures On Python 3, we use the built-in version in the standard library. Else we use our vendored backport. Differential Revision: https://phab.mercurial-scm.org/D3266

setup: add packages for concurrent.futures We conceivably don't need to distribute this package on Python 3 since we will use the version in the standard library. However, we want installs to be usable of multiple versions of Python. So it is best to always have it. Differential Revision: https://phab.mercurial-scm.org/D3265

futures: switch to absolute and relative imports This makes the package conform with our importing policy, silencing a number of warnings. It also makes the package usable when it isn't named "concurrent.futures." Differential Revision: https://phab.mercurial-scm.org/D3264

tests: silence pyflakes for thirdparty/concurrent It is complaining about unused imports. Differential Revision: https://phab.mercurial-scm.org/D3263

futures: get rid of extend_path This is used so mutliple directories can provide a package. We don't need it when vendoring. Differential Revision: https://phab.mercurial-scm.org/D3262

thirdparty: vendor futures 3.2.0 Python 3 has a concurrent.futures package in the standard library for representing futures. The "futures" package on PyPI is a backport of this package to work with Python 2. The wire protocol code today has its own future concept for handling of "batch" requests. The frame-based protocol will also want to use futures. I've heavily used the "futures" package on Python 2 in other projects and it is pretty nice. It even has a built-in thread and process pool for running functions in parallel. I've used this heavily for concurrent I/O and other GIL-less activities. The existing futures API in the wire protocol code is not as nice as concurrent.futures. Since concurrent.futures is in the Python standard library and will presumably be the long-term future for futures in our code base, let's vendor the backport so we can use proper futures today. # no-check-commit because of style violations Differential Revision: https://phab.mercurial-scm.org/D3261

py3: make sure decode() first argument is str Uses pycompat.sysstr() to make sure we uses bytes on Python 2 and unicodes on Python 3. Differential Revision: https://phab.mercurial-scm.org/D3279

patch: make extract() a context manager (API) Previously, this function was creating a temporary file and relying on callers to unlink it. Yuck. We convert the function to a context manager and tie the lifetime of the temporary file to that of the context manager. This changed indentation not only from the context manager, but also from the elination of try blocks. It was just easier to split the heart of extract() into its own function. The single consumer of this function has been refactored to use it as a context manager. Code for cleaning up the file in tryimportone() has also been removed. .. api:: ``patch.extract()`` is now a context manager. Callers no longer have to worry about deleting the temporary file it creates, as the file is tied to the lifetime of the context manager. Differential Revision: https://phab.mercurial-scm.org/D3306

cmdutil: pass in parsed patch to tryimportone() (API) Previously, we parsed the patch in tryimportone(). This assumes the input is in a patch format that needs to be parsed. We want to support feeding in data from other formats. So let's let the caller handle the parsing. One wonky thing about patch parsing is that patch.extract() creates a temp file to hold the diffs and it is up to tryimportone() to unlink that temp file. I'll improve this in a subsequent commit. Differential Revision: https://phab.mercurial-scm.org/D3305

stringutil: support more types with pprint() bytearray wasn't working. Integers and floats were not being formatted. I /think/ %f is portable across both Python 2 and 3, as it should default to 6 decimal points on each. Differential Revision: https://phab.mercurial-scm.org/D3302

fix: port most of the way to python 3 Only most of the way because we now have to decide: if we want to keep the current .format() interface for the config in hgrc, we have to use unicodes to do formatting in Python 3, rather than bytes. I'm basically fine with that, so a follow-up patch will do so. Differential Revision: https://phab.mercurial-scm.org/D3300

lfs: teach the blob server to handle --prefix

hgweb: fallback to checking wsgireq.env for REPO_NAME for 3rd party hosting Starting with d7fd203e36cc, SCM Manager began to 404 any repository access. What's happening is that it is generating a python script that creates an hgweb application (not hgwebdir), and launches hgweb via wsgicgi. It must be setting REPO_NAME in the process environment before launching this script, which gets picked up and put into wsgireq.env when wsgicgi launches the hgweb application. >From there, other variables (notably 'apppath' and 'dispatchpath') are constructed differently. d7fd203e36cc^ (working): apppath: /hg/eng/devsetup dispatchpath: pathinfo: /eng/devsetup reponame: eng/devsetup d7fd203e36cc: apppath: /hg dispatchpath: eng/devsetup pathinfo: /eng/devsetup reponame: None REPO_NAME: eng/devsetup Rather than having an existing installation break when Mercurial is upgraded, just resume checking the environment. I have no idea how many other hosting solutions would break without restoring this.

peer: scatter module to the wind (API) peer.py hardly contained any code. The code it did contain was generic to the version 1 peer interface or specific to the local repository peer. So code has been moved to wireprotov1peer and localrepo, as appropriate. Differential Revision: https://phab.mercurial-scm.org/D3260

wireproto: move version 1 peer functionality to standalone module (API) wireproto.py contains code for both the client and the server. There *should* be a somewhat strong separation between the two. This commit extracts the client-side code from wireproto.py into a new module - wireprotov1peer. Differential Revision: https://phab.mercurial-scm.org/D3259

wireproto: move gboptsmap to wireprototypes and rename (API) This is also shared between client and server and will need to exist in a shared module when that code is split into different modules. Differential Revision: https://phab.mercurial-scm.org/D3258

wireproto: move value encoding functions to wireprototypes (API) These functions should live in the same place. I plan to separate client from server code in upcoming commits. wireprototypes is where we are putting shared code like this. Differential Revision: https://phab.mercurial-scm.org/D3257

httppeer: basic implementation of capabilities interface This is a bit crude. The capabilities mechanism for version 2 of the wire protocol is a bit different from version 1. And code in core is relying on strings passed to capable() matching strings advertised by the "capabilities" wire protocol command. I may refactor the internal checking mechanism to be a bit more abstract or based on interfaces. Time will tell... Differential Revision: https://phab.mercurial-scm.org/D3256

repository: split capabilities methods into separate interface So we can implement them without having to implement support for every command. Differential Revision: https://phab.mercurial-scm.org/D3255

httppeer: implement ipeerconnection This is low hanging fruit. We might as well start somewhere. Differential Revision: https://phab.mercurial-scm.org/D3254

py3: whitelist another six passing tests Differential Revision: https://phab.mercurial-scm.org/D3286

py3: whitelist another nine passing tests Differential Revision: https://phab.mercurial-scm.org/D3253

hgweb: use our forked wsgiheaders module instead of stdlib one Now we use bytes for headers, rather than native strings. Differential Revision: https://phab.mercurial-scm.org/D2854

wsgiheaders: import a bytes-ified fork of wsgiref.headers from cpython@46f5072 This will let us restore Python 3 compatibility for tests that do http things. Differential Revision: https://phab.mercurial-scm.org/D3245

export: enable formatter support (API) This change is basically the same as "hg cat". A formatter object is created by caller. .. api:: ``cmdutil.export()`` takes a formatter as an argument.

export: extract function to write patch to file object (API) This is common use case of cmdutil.export(), and we wouldn't want to handle formatter thingy everywhere. .. api:: The ``fp`` argument is removed from ``cmdutil.export()``. Use ``cmdutil.exportfile()`` instead.

export: port _exportsingle() to formatter Pass 'fm' instead of 'write', and use fm.plain(), fm.write(), etc. instead. The callers will be updated later.

export: serialize revisions to be exported per destination file Prepares for porting to the formatter API, where we can't simply append to existing files because JSON can't be streamed for example. The modemap hack is removed since cmdutil.export() was the only user. I also made the destination filename printed only once.

export: split cmdutil.export() to single-file and maybe-multiple-files cases Porting "hg export" to formatter is a bit hard because cmdutil.export() may append to files if the fntemplate is specified. This patch splits the hard part from the trivial case.

export: remove unused argument 'rev' from _exportsingle()

export: do not start pager if output will be written to file A copy of 3b569745af6c.

formatter: carry opts to file-based formatters by basefm This makes it slightly easier to port "hg export" to formatter.

formatter: remove unused private attributes from baseformatter No idea what they were for.

fix: add --all flag to fix non-public non-obsolete revisions Differential Revision: https://phab.mercurial-scm.org/D3213

fsmonitor: layer on another hack in bser.c for os.stat() compat (issue5811) It's unclear to me how these `bserobj_tuple` objects are used, other than as stat objects. This should fix fsmonitor in the wake of ffa3026d4196 and similar changes. I regret the hack here, but the code already has plenty of hg-specific hacks. :( It feels like we should be able to use int(result.st_mtime) globally, but that doesn't work. See issue4836 for a bug that was hard to track down relating to rounding behavior causing very subtle dirstate problems. Differential Revision: https://phab.mercurial-scm.org/D2939

tests: use `f --newer` instead of `stat -c` in test-fix.t Also increase sleep to two seconds so this test will likely pass on FAT32. Differential Revision: https://phab.mercurial-scm.org/D3252

tests: glob away fqdn wherever we print it These localhost instances are actually from a getfqdn call, which means on some of my test systems it comes out as localhost.localdomain or 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa. I'm tired of this, so let's glob it away. Differential Revision: https://phab.mercurial-scm.org/D3251

hgweb: use native strings when interfacing with stdlib headers We're still parsing the stdlib-provided headers here, so we need to tread carefully and use native strings. Yuck. Differential Revision: https://phab.mercurial-scm.org/D3250

wireprotoserver: headers are bytes for us internally, use bytes This re-fixes test-pull-http.t on Python 3. Probably many others as well. Differential Revision: https://phab.mercurial-scm.org/D3249

hgweb: put response headers back into str for Python 3 This fixes a lot of hanging tests on Python 3, because "Content-Length" was getting sent as "b'Content-Length'" (yes, really) and then clients would expect a close-is-end body instead of counting off a certain number of bytes. Differential Revision: https://phab.mercurial-scm.org/D3248

tests: load showstack in test-pull-http.t so network hangs are easier to find This also gives us some minimal "it loads" coverage on showstack, which I rather like. showstack doesn't work on Windows per mbarbison, so it's disabled there. I added this in service of debugging a hang introduced on Python 3 by revision a88d68dc3ee8. I'm still not sure what the problem there is, but this at least gives us a little bit of a chance to figure out what's going on. Differential Revision: https://phab.mercurial-scm.org/D3247

keepalive: rewrite readinto() to not use read() It turns out http.client on Python 3 sometimes uses readinto() in the implementation of read(). Unfortunately, Python 2 doesn't have a readinto() in httplib's client, so we have to support both codepaths. Subclassing is bad, folks. Differential Revision: https://phab.mercurial-scm.org/D3246

py3: use bytes() instead of str() on util.url() We internally deal with bytes and anything as string breaks things. Differential Revision: https://phab.mercurial-scm.org/D3285

py3: use stringutil.forcebytestr() to convert error messages to bytes Differential Revision: https://phab.mercurial-scm.org/D3284

py3: suppress the return value of write() in tests/test-subrepo-missing.t write() on Python 3 returns a value whereas does not return anything on Python 2. So we need to supress the value. Differential Revision: https://phab.mercurial-scm.org/D3283

py3: add b'' prefixes in tests/test-linerange.py This makes the test pass on Python 3. # skip-blame because just b'' prefixes Differential Revision: https://phab.mercurial-scm.org/D3282

py3: add b'' prefix to make the regex bytes # skip-blame because just b'' prefix Differential Revision: https://phab.mercurial-scm.org/D3281

py3: use b'%d' to convert int to bytes instead of str() Differential Revision: https://phab.mercurial-scm.org/D3280

py3: use print as a function in test-convert-git.t Differential Revision: https://phab.mercurial-scm.org/D3278

py3: use '%d' for integers instead of '%s' Differential Revision: https://phab.mercurial-scm.org/D3277

py3: prevent transformer from adding b'' by adding r'' prefix These are cases where we need to use str, therefore we add r'' prefix. # skip-blame because just r'' prefixes Differential Revision: https://phab.mercurial-scm.org/D3276

py3: use pycompat.{strkwargs|byteskwargs} in infinitepush Differential Revision: https://phab.mercurial-scm.org/D3275

py3: make sure we open file in bytes mode Differential Revision: https://phab.mercurial-scm.org/D3274

diffhelpers: be more tolerant for stripped empty lines of CRLF ending Exchange. It appears to trim lines containing only whitespace as well as converting LF to CRLF.

diffhelpers: make return value of testhunk() more Pythonic It's no longer C.

patch: error out if reached to EOF while reading hunk This was where out-of-bounds read occurred in old C extension.

diffhelpers: remove unused return value from fixnewline() and addlines()

diffhelpers: move out of pure package

diffhelpers: naming and whitespace cleanup

diffhelpers: remove C implementation in favor of pure Python version

patch: stop using cext.diffhelpers The C implementation has a couple of memory bugs, and lacks error handling which could lead to SEGV. I could fix them one by one (and I mostly finished that), but the performance gain provided by cext.diffhelper is quite low. Besides, diffhelpers.addlines() calls back Python, linereader.readline(), from the innermost loop. $ hg export -R mozilla-central 0:100 > patch $ ls -lh patch -rw-r--r-- 184M patch $ hg init repo && hg -R repo import patch --time --bypass (cext) time: real 34.970 secs (user 32.720+0.000 sys 2.230+0.000) (pure) time: real 35.950 secs (user 33.600+0.000 sys 2.330+0.000) So, let's simply use the pure Python implementation.

diffhelpers: port docstrings from cext to pure I'll remove the C implementation.

py3: get rid of character access from pure.diffhelpers 's' is a result of readline(), so 'c == "\n"' means 's == "\n"'.

lfs: handle paths that don't end with '/' when inferring the blob store While here, I also checked the lfs.url config directly instead of testing the scheme, as requested by Yuya.

lfs: infer the blob store URL from an explicit push dest or default-push Unlike pull, the blobs are uploaded within the exchange.push() window, so simply wrap it and swap in a properly configured remote store. The '_subtoppath' field shouldn't be available during this window, but give the passed path priority for clarity. At one point I hit an AttributeError in one of the convert tests when trying to save the original remote blobstore when the swap was run unconditionally. I wrapped it in a util.safehasattr(), but then today I wasn't able to reproduce it. But now the whole thing is tucked under the requirement guard because without the requirement, there are no blobs in the repo, even if the extension is loaded.

lfs: infer the blob store URL from an explicit pull source I don't see any easier way to do this because the update part of `hg pull -u` happens outside exchange.pull(), and commands.postincoming() doesn't take a path. So (ab)use the mechanism used by subrepos to redirect where subrepos are pulled from when an explicit path is given. As a bonus, this should allow lfs blobs to be pulled into a subrepo when it is checked out. An explicit push path can be handled within exchange.push(). That can be done next, outside of this dirty hack.

lfs: special case the null:// usercache instead of treating it as a url The previous code worked on Windows, but not on Unix, and a pending patch's test failed. The url being used was something like "/tmp/.../client1/null://", courtesy of ui.configpath(). Looking at the doc comment, this seems like it's maybe not the right function to call (why should a relative cache path be expanded relative to the repo root or config file?), but largefiles has been using it since 8b8dd13295db (Oct 2011). It was introduced in 1b591f9b7fd2 (Jan 2011) without comment or callers. A grep over the whole history shows that only largefiles used it until lfs and infinitepush came along recently. It looks like if the `if not os.path.isabs(v) or "://" not in v` in configpath() is changed to an 'and', both Linux and Windows are happy. I'm guessing that "://" is to pick off URLs, so that seems reasonable. But I'm not sure why it isn't explicitly "file://", and I thought that "file://foo" is relative anyway. (At least, there are doctests for file:///tmp in util.url.) There is no mention of this setting in the help, but it is referenced on the wiki page for largefiles. (There's no mention that this is intended to be a URL, and the example uses an absolute path.) I don't want this blocking the rest of the lfs server discovery stuff. It was also wrong to allow a file:// URL here, but not in largefiles.

tests: add tests showing pulling from infinitepush works over wire The current tests in test-infinitepush-ci.t showed that `hg pull -r <rev>` does not work. Digging in code, I found that we have logic for pulling from bundlestore without having client side logic. This patch adds test demonstrating that pulling from bundlestore works when working over wire. Pulling from bundlestore when the peer is a localpeer still does not works. Differential Revision: https://phab.mercurial-scm.org/D3072

fix: use a portable python script instead of sed in test Differential Revision: https://phab.mercurial-scm.org/D2988

py3: use pycompat.bytestr() where repr in involved Differential Revision: https://phab.mercurial-scm.org/D3244

httppeer: support protocol upgrade With the new handshake defined and in place on the server, we can now implement it on the client. The HTTP handshake mechanism has been taught to add headers advertising its support for the new capabilities response. Response handling has been adjusted to allow CBOR responses through. And makepeer() has been taught to instantiate a mutually supported peer. The HTTPv2 peer class doesn't implement the full peer interface. So HTTPv2 is not yet usable as a peer. Like the server side, we support registering handlers for different API services. This allows extensions to easily implement API services and peers. A practical use case for this is to provide a previous implementation of the experimental version 2 wire protocol to a future version of Mercurial. We know there will be BC breaks after 4.6 ships. But someone could take the peer and server code from 4.6, drop it in an extension, and allow its use indefinitely. Differential Revision: https://phab.mercurial-scm.org/D3243

wireproto: define and implement HTTP handshake to upgrade protocol When clients connect to repositories over HTTP, they issue a request to the well-known URL "?cmd=capabilities" to fetch the repository capabilities. This is the handshake portion of the HTTP protocol. This commit defines a mechanism to use that HTTP request to return information about modern server features. If a client sends an X-HgUpgrade-* header containing a list of client-supported API names, the server responds with a response containing information about available services. This includes the normal capabilities string. So if the server doesn't support any newer services, the client can easily fall back. By advertising supported services from clients, server operators can see and log what client support exists in the wild. This will also help with debugging. The response contains the base path to API services. We know there are potential issues with the <repo>/api/ URL space conflicting with hgwebdir and subrepos. By making the API URL dynamic from the perspective of the client, the URL for APIs is not subject to backwards compatibility concerns - at least as long as a ?cmd=capabilities request is made. We've also defined the ``cbor`` client capability for the X-HgProto-* header. This MUST be sent in order to get the modern response from "?cmd=capabilities". During implementation, I initially always sent an application/mercurial-cbor response. However, the handshake mechanism will be more future compatible if the client is in charge of which formats to request. We already perform content negotiation from X-HgProto-*, so keying off this for the capabilities response feels appropriate. In addition, I initially used application/cbor. However, it is conceivable that a non-Mercurial server could serve application/cbor. To rule out this possibility, I've invented a new media type that is Mercurial specific and can't be confused for generic CBOR. Differential Revision: https://phab.mercurial-scm.org/D3242

httppeer: only advertise partial-pull if capabilities are known We don't need to be advertising client protocol parameters as part of the capabilities request during the handshake because nothing in version 1 of the wire protocol will use this data. i.e. the advertisement is wasteful. Differential Revision: https://phab.mercurial-scm.org/D3241

httppeer: always add x-hg* headers to Vary header Before, we manually updated the Vary header value for each header contributing to it. All X-Hg* headers are reserved for the Mercurial protocol and could have caching implications. So it makes sense to always add these headers to Vary. A test revealed that X-HgArgs-Post wasn't being added to Vary. This is only sent on POST requests. POST requests generally aren't cacheable. However, it is possible if the server sends the appropriate headers. Mercurial shouldn't be sending those headers. But let's not take any chances. Differential Revision: https://phab.mercurial-scm.org/D3240

httppeer: don't accept very old media types (BC) Versions of Mercurial older than 1.0 emitted the text/plain and application/hg-changegroup media types in response to wire protocol commands. Way back in 8760d0c83b9b in 2005, the code validating these media types was added, presumably for backwards compatibility. 0b245edec124 a short time before that commit changed things from text/plain and application/hg-changegroup to application/mercurial-0.1 and application/hg-0.1. 8760d0c83b9b seemed to indicate ("for now") that the BC compatibility was temporary. But that code has lived until this day. It has been more than 10 years and nobody should be running pre 1.0 servers. Pretty much the only risk to this is if there's a server somewhere advertising the old media types or server software is interfering and not letting Mercurial send the proper Content-Type header. I think the chances are rare. The wire protocol docs were created (by me) from reading existing code. So the deletions don't constitute a spec change as much as reflecting the reality of how things have been for years. .. bc:: The HTTP client no longer accepts text/plain and application/hg-changegroup Content-Type values as a valid Mercurial command response. These should only be encountered on pre 1.0 Mercurial servers. Differential Revision: https://phab.mercurial-scm.org/D3239

httppeer: allow opener to be passed to makepeer() This allows us to use makepeer() in `hg debugwireproto`. Differential Revision: https://phab.mercurial-scm.org/D3238

httppeer: perform capabilities request in makepeer() Previously, we constructed an httppeer then always ran _fetchcaps() to issue the capabilities command. We want to issue the capabilities command before constructing a peer instance so we can construct an appropriate peer instance depending on the capabilities result. With the code for making and sending requests moved out of httppeer, it is now possible to send command requests without an httppeer. This commit creates a new function for making the capabilities request and calls it as part of makepeer(). This code should be functionality equivalent to what existed before. Differential Revision: https://phab.mercurial-scm.org/D3237

httppeer: extract common response handling into own function This allows the common redirect detection, content type validation, and decompression wrapping to be usable outside of httppeer instances. Differential Revision: https://phab.mercurial-scm.org/D3236

httppeer: move error handling and response wrapping into sendrequest This is common for all HTTP requests. It should be part of sendrequest(). Differential Revision: https://phab.mercurial-scm.org/D3235

httppeer: extract code for creating a request into own function Some of this feels awkward, such as having to pass in a function to evaluate a capability. And this code is generally pretty difficult to read. I didn't want to perform too much refactoring as part of the code move since it would make review more difficult. Differential Revision: https://phab.mercurial-scm.org/D3234

httppeer: extract code for performing an HTTP request This is generic and doesn't need to live as a method of httppeer. Differential Revision: https://phab.mercurial-scm.org/D3233

httppeer: move requestbuilder defaults into makepeer() argument Upcoming commits will move the initial ?cmd=capabilities handshake request out of httppeer so the handshake can be performed before a peer instance is constructed. In order to do this, we'll need to refactor code for making HTTP requests. The type used to construct HTTP requests is configurable. If we'll be making HTTP requests outside of httppeer, we should be able to use a custom request builder. So move the definition of that type into makepeer(). Extensions can monkeypatch the function and override the argument value. Differential Revision: https://phab.mercurial-scm.org/D3232

wireproto: move version 2 command handlers to wireprotov2server This is relatively straightforward. As part of this, we introduced a local @wireprotocommand that wraps the main one and defines a v2 only policy by default. Because the hacky HTTPv2 peer isn't using capabilities response yet, we had to move some code around to force import of wireprotov2server so commands are registered. This is super hacky. But this code will go away once the HTTPv2 peer is using the capabilities response to derive permissions. Differential Revision: https://phab.mercurial-scm.org/D3231

wireproto: extract HTTP version 2 code to own module wireprotoserver has generic and version 1 server code. The wireproto module also has both version 1 and version 2 command implementations. Upcoming work I want to do will make it difficult for this code to live in the current locations. Plus, it kind of makes sense for the version 2 code to live in an isolated module. This commit copies the HTTPv2 bits from wireprotoserver into a new module. We do it as a file copy to preserve history. A future commit will be copying wire protocol commands into this module as well. But there is little history of that code, so it makes sense to take history for wireprotoserver. Differential Revision: https://phab.mercurial-scm.org/D3230

wireproto: client reactor support for receiving frames We can now feed received frames into the client reactor and it will validate their sanity, dispatch them appropriately. The hacky HTTP peer has been updated to use the new code. No existing tests changed, somewhat proving the code works as expected. Rudimentary unit tests for the new functionality have been implemented. Differential Revision: https://phab.mercurial-scm.org/D3224

wireproto: introduce a reactor for client-side state We have a nice state machine of sorts for reacting to server-side events. Now it is time to implement the client equivalent. We introduce a "clientreactor." It allows callers to request that commands be issued. It has multiple modes of operation to reflect what the underlying transport supports. e.g. for SSH, we can perform wire sends immediately but for HTTP we need to buffer sends until all command requests are received. In addition, SSH allows sending multiple requests as long as the connection is open. But HTTP/1.1 only allows sending request data once. For SSH, we'll have one reactor per connection. For HTTP, we'll have one reactor per HTTP request. But because code that calls wire protocol commands should not be aware of how the underlying transport works, this will all be abstracted away by the peer interface. Our crude HTTP peer has been updated to use the reactor instead of formulating frames directly. No behavior should have changed here and tests seem to confirm that. Basic unit tests for the reactor behavior have been added. Differential Revision: https://phab.mercurial-scm.org/D3223

tests: extract wire protocol framing tests to own file I was lazy when I put these in test-wireproto-serverreactor.py. Let's do it properly. Differential Revision: https://phab.mercurial-scm.org/D3222

wireproto: disallow commands handlers for multiple transport versions I think it will be more trouble than it is worth to code version 1 and version 2 command handlers to the same interface. It will feel awkward to shoehorn functionality into e.g. the version 1 protocol handler interface. This would likely constrain the ability for version 2 to evolve. Previous commits introduced a clean separation between command handlers for version 1 and version 2 transports. This commit reinforces that separation by dropping support for having a single command handler service both version 1 and version 2 transports. Differential Revision: https://phab.mercurial-scm.org/D3208

wireproto: make @wireprotocommand version 1 only by default For backwards compatibility reasons. We want extension provided commands to opt in to version 2 rather than get inherited automatically. This will facilitate a clean break between the protocols. As part of this, we duplicate some commands used in tests so there are different command handlers per transport. Differential Revision: https://phab.mercurial-scm.org/D3207

wireproto: only expose "getbundle" and "unbundle" to v1 transports These are the most complicated wire protocol commands. I don't want to deal with porting them just yet. Let's disable both of them on version 2 transports so we drive the final wedge between command handlers and start to evolve version 2 command handlers more. Differential Revision: https://phab.mercurial-scm.org/D3206

wireproto: port lookup to wire protocol v2 This is pretty straightforward. We don't yet handle errors because we don't have an error handling mechanism in place yet. I'm also tempted to fold this into `known`. We'll come back to this later. Differential Revision: https://phab.mercurial-scm.org/D3205

wireproto: port pushkey command to wire protocol version 2 It doesn't do output redirection yet. And I'd love to generally overhaul the pushkey protocol for wire protocol version 2. But this will be a bit of effort. Let's do it as a follow-up. Differential Revision: https://phab.mercurial-scm.org/D3204

wireproto: only expose "clonebundles" to version 1 transports This may make a comeback in wire protocol version 2. The feature definitely needs to be carried forward. But at this juncture, I'm flirting with the idea of implementing this via a "redirect" mechanism at the command response level itself rather than something that requires one-off client support for querying and handling. i.e. I want to make it so servers can say "fetch this first and then come back" and clients handle that automatically. This would not only support clone bundles, but would also support piece-meal "pull bundles." Whatever happens, we can deal with it down the road. Differential Revision: https://phab.mercurial-scm.org/D3203

wireproto: define and expose types of wire command arguments Exposing the set of argument names is cool. But with wire protocol version 2, we're using CBOR to transport arguments and this allows us to have typing for arguments. Typed arguments are much nicer because they will cut down on transfer overhead and processing overhead for decoding values. This commit teaches @wireprotocommand to accept a dictionary for arguments. The arguments registered for version 2 transports are canonically stored as dictionaries rather than a space-delimited string. It is an error to defined arguments with a dictionary for commands using version 1 transports. This reinforces my intent to fully decouple command handlers for version 2 transports. Differential Revision: https://phab.mercurial-scm.org/D3202

wireproto: only expose "stream_out" to version 1 transports I have plans to implement stream clone using a better mechanism than this existing command. Let's not carry it forward to wire protocol version 2. Differential Revision: https://phab.mercurial-scm.org/D3201

wireproto: implement capabilities for wire protocol v2 The capabilities mechanism for wire protocol version 2 represents a clean break from version 1. Instead of effectively exchanging a set of capabilities, we're exchanging a rich data structure. This data structure currently contains information about every available command, including its accepted arguments. It also contains information about supported compression formats. Exposing information about supported commands will allow clients to automatically generate bindings to the server. Clients will be able to do things like detect when they are attempting to run a command that isn't known to the server. Exposing the required permissions to run a command can be used by clients to determine if they have privileges to call a command before actually calling it. We could potentially even have clients send credentials preemptively without waiting for the server to deny the command request. Lots of potential here. The data returned by this command will likely evolve heavily. So we shouldn't bikeshed the implementation just yet. Differential Revision: https://phab.mercurial-scm.org/D3200

context: add deprecation warnings for deprecated types of changeids It's close to code freeze, and dropping support for repo['123'] and repo ['my-bookmark'] and repo['deadbeef'] is pretty dispruptive, so this just adds deprecation warnings so extensions can easily find the places they need to fix. Differential Revision: https://phab.mercurial-scm.org/D3197

revsymbol: stop delegating to repo.__getitem__ for unhandled symbols (API) The only remaining cases where we were delegating unhandled symbols to repo.__getitem__ should now be when the symbol could not be found. In that case we just delegated to repo.__getitem__ for the error message. Let's just copy the error message instead. If there were any cases where we got e.g. a binary nodeid or an integer revnum into revsymbol() (e.g. via repo.lookup()), we'd now start raising an exception instead. That is why this is marked (API). This affects one test case, but the new behavior seems better to me. I can't tell if the old behavior was desired or if the test was just there to document how it happened to work. Differential Revision: https://phab.mercurial-scm.org/D3196

context: handle partial nodeids in revsymbol() Similar reasoning as previous patches. Differential Revision: https://phab.mercurial-scm.org/D3195

context: handle namespaces in revsymbol() Similar reasoning as previous patches. Differential Revision: https://phab.mercurial-scm.org/D3194

scmutil: handle full hex nodeids in revsymbol() This is a bit unfortunate, but it enables moving other pieces out of changectx's constructor without affecting the order in which we look up things (e.g. hex nodeid before bookmark). We convert nodeid to revnum before calling repo.__getitem__, even though that will result in converting back to nodeid later. This is so we can handle the LookupError and attempt to interpret the string as something else (e.g. a bookmark). We also need to start handling WdirUnsupported now, since the full hex nodeid "ffff..." represents the working directory. The exception is raised by the revlog layer. Differential Revision: https://phab.mercurial-scm.org/D3193

context: handle stringified ints in revsymbol() This patch copies the handling of stringified ints from changectx's constructor. It then calls repo.__getitem__ with the int. Since that method only interprets integers as revnums the first thing it does, this will not be redoing any of the work already done. We leave the old code in place so we can later deprecate it instead of breaking extensions. Differential Revision: https://phab.mercurial-scm.org/D3146

py3: make sure we write bytes to file # skip-blame because just b'' prefix Differential Revision: https://phab.mercurial-scm.org/D3220

py3: workaround comparing NoneType and integers Comparing None with integers was fine in Python 2 but returns error in Python 3 which is nice. This patch replaces None with -1 where sorting is done and some related logic. Differential Revision: https://phab.mercurial-scm.org/D3219

py3: use sys.stdout instead of print in test-mq-qpush-fail.t Differential Revision: https://phab.mercurial-scm.org/D3218

py3: use '//' for integer division in tests/test-mq-qimport.t Differential Revision: https://phab.mercurial-scm.org/D3217

py3: use pycompat.byterepr() instead of repr() Differential Revision: https://phab.mercurial-scm.org/D3216

py3: use bytes instead of str in isinstance() We deal with bytes internally and things should be bytes Differential Revision: https://phab.mercurial-scm.org/D3215

py3: use .startswith() instead of bytes[0] Differential Revision: https://phab.mercurial-scm.org/D3214