py3: use urllib.parse.unquote_plus instead of urllib.unquote_plus The later is not present in Python 3. Differential Revision: https://phab.mercurial-scm.org/D3322

py3: add b'' prefixes to tests/test-status-inprocess.py # skip-blame because just b'' prefixes Differential Revision: https://phab.mercurial-scm.org/D3321

lock: don't use 'file' as a variable name Differential Revision: https://phab.mercurial-scm.org/D3320

py3: use b"%d" instead of str() to convert integers to bytes Differential Revision: https://phab.mercurial-scm.org/D3319

wireproto: expose repository formats via capabilities Servers need to expose their set of repository storage requirements in order to facilitate streaming clones (clients need to know if they are capable of reading the raw storage files that the server exposes). Differential Revision: https://phab.mercurial-scm.org/D3335

wireprotoframing: record when new stream is encountered Without this, we choke after receiving the 2nd frame in a stream. Not sure how we made it this far without finding this bug. Differential Revision: https://phab.mercurial-scm.org/D3334

wireprotoframing: use value passed into function Oops. Differential Revision: https://phab.mercurial-scm.org/D3333

httppeer: handle error response from client reactor With this in place, we're now seeing useful errors when running tests with the new wire protocol enabled! Differential Revision: https://phab.mercurial-scm.org/D3332

wireproto: add media type to version 2 capabilities response This is useful to advertise because servers reject unsupported media types. A client may wish to speak multiple media types and choose the one the server supports. I doubt we'll ever use multiple media types or negotiation in core. But during the course of developing this protocol, I may end up making extensions that backport and forward port protocol support as needed to support Mercurial deploys in the wild. e.g. I may deploy support for an older protocol on a server so old clients can continue using it. It's worth pursuing changing the SSH protocol's upgrade mechanism to support multiple media types as well... Differential Revision: https://phab.mercurial-scm.org/D3299

debugcommands: use command executor for invoking commands Now that all peers support the command executor interface, we can use it in place of peer._call() (which isn't part of the public API). Tests changed because we are now returning the decoded response instead of the raw bytes. The raw bytes over the wire are already logged. So we're not losing any test coverage. In fact, we're adding test coverage because we're testing decoding of those command responses as well. Differential Revision: https://phab.mercurial-scm.org/D3298

httppeer: implement command executor for version 2 peer Now that we have a new API for issuing commands which is compatible with wire protocol version 2, we can start using it with wire protocol version 2. This commit replaces our hacky implementation of _call() with something a bit more robust based on the new command executor interface. We now have proper support for issuing multiple commands per HTTP request. Each HTTP request maintains its own client reactor. The implementation is similar to the one in the legacy wire protocol. We use a ThreadPoolExecutor for spinning up a thread to read the HTTP response in the background. This allows responses to resolve in any order. While not implemented on the server yet, a client could use concurrent.futures.as_completed() with a collection of futures and handle responses as they arrive from the server. The return value from issued commands is still a simple list of raw or decoded CBOR data. This is still super hacky. We will want a rich data type for representing command responses. But at least this commit gets us one step closer to a proper peer implementation. Differential Revision: https://phab.mercurial-scm.org/D3297

repository: remove ipeercommands from ipeerbase AFAICT all callers in core have moved to the commandexecutor interface for invoking wire protocol commands. Or at least they aren't using the named methods on ipeercommands to invoke them. This means we can drop ipeercommands from the ipeerbase interface. As far as interface based programming goes, it is now illegal to call an ipeercommands method for issuing wire protocol commands. However, the methods are still there, so they will still work. At some point we will want to break that API... Differential Revision: https://phab.mercurial-scm.org/D3318

wireproto: properly call clonebundles command We should not be using _call() to make wire protocol calls because it isn't part of the peer API. But clonebundles wasn't part of the supported commands in the peer API! So this commit defines that command in the commands interface, implements it, and teaches the one caller in core to call it using the command executor interface. Differential Revision: https://phab.mercurial-scm.org/D3317

exchange: use command executor for getbundle The code consuming the bundle has been moved to inside the context manager, as that is supposed to be part of the API. (Although it doesn't matter for version 1 peers.) Differential Revision: https://phab.mercurial-scm.org/D3316

exchange: use command executor for pushkey Differential Revision: https://phab.mercurial-scm.org/D3315

wireproto: use command executor for unbundle This also required unifying the name of the argument because the new API always passes arguments by keyword. I decided to change implementations to "bundle" instead of the interface to "cg" because "bundle" is more appropriate in a modern world. Differential Revision: https://phab.mercurial-scm.org/D3314

debugcommands: perform handshake when obtaining httpv2 peer If we obtain an httpv2peer directly, the instance doesn't have an API descriptor and therefore doesn't know about the remote's commands, feature support, etc. This doesn't matter now. But when we implement the peer so it consults the API descriptor as part of sending commands, it will. So we change the logic for obtaining an http version 2 peer to go through makepeer() so the peer will perform the handshake and pass the API descriptor to the httpv2peer instance. Tests changed because we now perform a ?cmd=capabilities when obtaining version 2 peers. The Content-Length header is globbed because compression info will lack zstandard for pure builds. Differential Revision: https://phab.mercurial-scm.org/D3296

wireproto: rename HTTPV2 so it less like HTTP/2 Per review suggestion on D3230 from Augie. Differential Revision: https://phab.mercurial-scm.org/D3295

bundlerepo: use command executor for wire protocol commands Differential Revision: https://phab.mercurial-scm.org/D3294

bundlerepo: rename "other" to "peer" Differential Revision: https://phab.mercurial-scm.org/D3293

bookmarks: use command executor for wire protocol commands And change the name of a variable to reflect that is is a peer. Differential Revision: https://phab.mercurial-scm.org/D3292

hg: use command executor for wire protocol commands As part of this, I realized that some uses of lookup in a loop could be converted to use a batch request. But I didn't change behavior and left in a todo to track potentially changing this. Differential Revision: https://phab.mercurial-scm.org/D3291

logexchange: use command executor for wire protocol commands Differential Revision: https://phab.mercurial-scm.org/D3290

streamclone: use command executor for wire protocol commands Differential Revision: https://phab.mercurial-scm.org/D3289

discovery: use command executor interface We're trying to port all wire protocol code to use the new interface so we can implement wire protocol version 2 clients. Differential Revision: https://phab.mercurial-scm.org/D3288

discovery: don't redundantly call branchmap We were calling the remote command twice without mutation the remote in between. Derp. Differential Revision: https://phab.mercurial-scm.org/D3287

wireproto: convert legacy commands to command executor Calls to the legacy commands "changegroup" and "changegroupsubset" have been ported to the new command executor interface. Because we always pass arguments by name and not position, some inconsistent names throughout the code base have been unified. As part of this change, we no longer had any remaining callers of the legacy command methods {between, branches, changegroup, changegroupsubset}. So, these interfaces/methods have been dropped from peer interfaces. We still have an interface declaring these methods. But that interface is implemented on the concrete peer types and isn't part of the generic peer interface. (The implementations of the command executor continue to call these methods.) The ultimate goal is to remove the per-command methods from the generic peer interface: the only interface-conforming way to call a command will be with the new executor API. At some point, we may want to move the methods outside of the peer classes and change the executor implementations to not call methods directly on a peer instance. Differential Revision: https://phab.mercurial-scm.org/D3273

treediscovery: switch to command executor interface We now have a new interface for requesting that commands run. Switch to it. Differential Revision: https://phab.mercurial-scm.org/D3272

wireproto: remove iterbatch() from peer interface (API) Good riddance. Some tests have been ported to the new API. This probably should have been done in earlier commits. But duplicating the test coverage would have been difficult. It was easier this way. .. api:: The wire protocol peer's ``iterbatch()`` for bulk executing commands has been remove.d Use ``peer.commandexecutor()`` instead. Differential Revision: https://phab.mercurial-scm.org/D3271

largefiles: use command executor for batch operation This is the only other user of iterbatch() in core. Tests changed because the new command executor is smart enough to not send a "batch" command over the wire if only 1 command was requested. There is still coverage for the "batch" command in this test though. Differential Revision: https://phab.mercurial-scm.org/D3270

wireproto: implement batching on peer executor interface This is a bit more complicated than non-batch requests because we need to buffer sends until the last request arrives *and* we need to support resolving futures as data arrives from the remote. In a classical concurrent.futures executor model, the future "starts" as soon as it is submitted. However, we have nothing to start until the last command is submitted. If we did nothing, calling result() would deadlock, since the future hasn't "started." So in the case where we queue the command, we return a special future type whose result() will trigger sendcommands(). This eliminates the deadlock potential. It also serves as a check against callers who may be calling result() prematurely, as it will prevent any subsequent callcommands() from working. This behavior is slightly annoying and a bit restrictive. But it's the world that half duplex connections forces on us. In order to support streaming responses, we were previously using a generator. But with a futures-based API, we're using futures and not generators. So in order to get streaming, we need a background thread to read data from the server. The approach taken in this patch is to leverage the ThreadPoolExecutor from concurrent.futures for managing a background thread. We create an executor and future that resolves when all response data is processed (or an error occurs). When exiting the context manager, we wait on that background reading before returning. I was hoping we could manually spin up a threading.Thread and this would be simple. But I ran into a few deadlocks when implementing. After looking at the source code to concurrent.futures, I figured it would just be easier to use a ThreadPoolExecutor than implement all the code needed to manually manage a thread. To prove this works, a use of the batch API in discovery has been updated. Differential Revision: https://phab.mercurial-scm.org/D3269

wireproto: implement command executor interface for version 1 peers Now that we've defined our new interface for issuing commands, let's implement it. We add the interface to the base peer interface. This means all peer types must implement it. The only peer types that we have are the local peer in localrepo and a shared wire peer for version 1 of the wire protocol. The local peer implementation is pretty straightforward. We don't do anything fancy and just return a resolved future with the result of a method call. This is similar to what localiterbatcher does. The wire protocol version 1 implementation is a bit more complicated and is a more robust implementation. The wire executor queues commands by default. And because the new executor interface always allows multiple commands but not all version 1 commands are @batchable, it has to check that the requested commands are batchable if multiple commands are being requested. The wire executor currently only supports executing a single command. This is for simplicity reasons. Support for multiple commands will be added in a separate commit. To prove the new interface works, a call to the "known" command during discovery has been updated to use the new API. It's worth noting that both implementations require a method having the command name to exist on the peer. There is at least one caller in core that don't have a method calls peer._call() directly. We may need to shore up the requirements later... Differential Revision: https://phab.mercurial-scm.org/D3268