largefiles: use command executor for batch operation This is the only other user of iterbatch() in core. Tests changed because the new command executor is smart enough to not send a "batch" command over the wire if only 1 command was requested. There is still coverage for the "batch" command in this test though. Differential Revision: https://phab.mercurial-scm.org/D3270

wireproto: implement batching on peer executor interface This is a bit more complicated than non-batch requests because we need to buffer sends until the last request arrives *and* we need to support resolving futures as data arrives from the remote. In a classical concurrent.futures executor model, the future "starts" as soon as it is submitted. However, we have nothing to start until the last command is submitted. If we did nothing, calling result() would deadlock, since the future hasn't "started." So in the case where we queue the command, we return a special future type whose result() will trigger sendcommands(). This eliminates the deadlock potential. It also serves as a check against callers who may be calling result() prematurely, as it will prevent any subsequent callcommands() from working. This behavior is slightly annoying and a bit restrictive. But it's the world that half duplex connections forces on us. In order to support streaming responses, we were previously using a generator. But with a futures-based API, we're using futures and not generators. So in order to get streaming, we need a background thread to read data from the server. The approach taken in this patch is to leverage the ThreadPoolExecutor from concurrent.futures for managing a background thread. We create an executor and future that resolves when all response data is processed (or an error occurs). When exiting the context manager, we wait on that background reading before returning. I was hoping we could manually spin up a threading.Thread and this would be simple. But I ran into a few deadlocks when implementing. After looking at the source code to concurrent.futures, I figured it would just be easier to use a ThreadPoolExecutor than implement all the code needed to manually manage a thread. To prove this works, a use of the batch API in discovery has been updated. Differential Revision: https://phab.mercurial-scm.org/D3269

wireproto: implement command executor interface for version 1 peers Now that we've defined our new interface for issuing commands, let's implement it. We add the interface to the base peer interface. This means all peer types must implement it. The only peer types that we have are the local peer in localrepo and a shared wire peer for version 1 of the wire protocol. The local peer implementation is pretty straightforward. We don't do anything fancy and just return a resolved future with the result of a method call. This is similar to what localiterbatcher does. The wire protocol version 1 implementation is a bit more complicated and is a more robust implementation. The wire executor queues commands by default. And because the new executor interface always allows multiple commands but not all version 1 commands are @batchable, it has to check that the requested commands are batchable if multiple commands are being requested. The wire executor currently only supports executing a single command. This is for simplicity reasons. Support for multiple commands will be added in a separate commit. To prove the new interface works, a call to the "known" command during discovery has been updated to use the new API. It's worth noting that both implementations require a method having the command name to exist on the peer. There is at least one caller in core that don't have a method calls peer._call() directly. We may need to shore up the requirements later... Differential Revision: https://phab.mercurial-scm.org/D3268

repository: define new interface for running commands Today, the peer interface exposes methods for each command that can be executed. In addition, there is an iterbatch() API that allows commands to be issued in batches and provides an iterator over the results. This is a glorified wrapper around the "batch" wire command. Wire protocol version 2 supports nicer things (such as batching any command and out-of-order replies). It will require a more flexible API for executing commands. This commit introduces a new peer interface for making command requests. In the new world, you can't simply call a method on the peer to execute a command: you need to obtain an object to be used for executing commands. That object can be used to issue a single command or it can batch multiple requests. In the case of full duplex peers, the command may even be sent out over the wire immediately. There are no per-command methods. Instead, there is a generic method to call a command. The implementation can then perform domain specific processing for specific commands. This includes passing data via a specially named argument. Arguments are also passed as a dictionary instead of using **kwargs. While **kwargs is nicer to use, we've historically gotten into trouble using it because there will inevitably be a conflict between the name of an argument to a wire protocol command and an argument we want to pass into a function. Instead of a command returning a value, it returns a future which will resolve to a value. This opens the door for out-of-order response handling and concurrent response handling in the version 2 protocol. Differential Revision: https://phab.mercurial-scm.org/D3267

pycompat: export a handle on concurrent.futures On Python 3, we use the built-in version in the standard library. Else we use our vendored backport. Differential Revision: https://phab.mercurial-scm.org/D3266

setup: add packages for concurrent.futures We conceivably don't need to distribute this package on Python 3 since we will use the version in the standard library. However, we want installs to be usable of multiple versions of Python. So it is best to always have it. Differential Revision: https://phab.mercurial-scm.org/D3265

futures: switch to absolute and relative imports This makes the package conform with our importing policy, silencing a number of warnings. It also makes the package usable when it isn't named "concurrent.futures." Differential Revision: https://phab.mercurial-scm.org/D3264

tests: silence pyflakes for thirdparty/concurrent It is complaining about unused imports. Differential Revision: https://phab.mercurial-scm.org/D3263

futures: get rid of extend_path This is used so mutliple directories can provide a package. We don't need it when vendoring. Differential Revision: https://phab.mercurial-scm.org/D3262

thirdparty: vendor futures 3.2.0 Python 3 has a concurrent.futures package in the standard library for representing futures. The "futures" package on PyPI is a backport of this package to work with Python 2. The wire protocol code today has its own future concept for handling of "batch" requests. The frame-based protocol will also want to use futures. I've heavily used the "futures" package on Python 2 in other projects and it is pretty nice. It even has a built-in thread and process pool for running functions in parallel. I've used this heavily for concurrent I/O and other GIL-less activities. The existing futures API in the wire protocol code is not as nice as concurrent.futures. Since concurrent.futures is in the Python standard library and will presumably be the long-term future for futures in our code base, let's vendor the backport so we can use proper futures today. # no-check-commit because of style violations Differential Revision: https://phab.mercurial-scm.org/D3261

py3: make sure decode() first argument is str Uses pycompat.sysstr() to make sure we uses bytes on Python 2 and unicodes on Python 3. Differential Revision: https://phab.mercurial-scm.org/D3279

patch: make extract() a context manager (API) Previously, this function was creating a temporary file and relying on callers to unlink it. Yuck. We convert the function to a context manager and tie the lifetime of the temporary file to that of the context manager. This changed indentation not only from the context manager, but also from the elination of try blocks. It was just easier to split the heart of extract() into its own function. The single consumer of this function has been refactored to use it as a context manager. Code for cleaning up the file in tryimportone() has also been removed. .. api:: ``patch.extract()`` is now a context manager. Callers no longer have to worry about deleting the temporary file it creates, as the file is tied to the lifetime of the context manager. Differential Revision: https://phab.mercurial-scm.org/D3306

cmdutil: pass in parsed patch to tryimportone() (API) Previously, we parsed the patch in tryimportone(). This assumes the input is in a patch format that needs to be parsed. We want to support feeding in data from other formats. So let's let the caller handle the parsing. One wonky thing about patch parsing is that patch.extract() creates a temp file to hold the diffs and it is up to tryimportone() to unlink that temp file. I'll improve this in a subsequent commit. Differential Revision: https://phab.mercurial-scm.org/D3305

stringutil: support more types with pprint() bytearray wasn't working. Integers and floats were not being formatted. I /think/ %f is portable across both Python 2 and 3, as it should default to 6 decimal points on each. Differential Revision: https://phab.mercurial-scm.org/D3302

fix: port most of the way to python 3 Only most of the way because we now have to decide: if we want to keep the current .format() interface for the config in hgrc, we have to use unicodes to do formatting in Python 3, rather than bytes. I'm basically fine with that, so a follow-up patch will do so. Differential Revision: https://phab.mercurial-scm.org/D3300

lfs: teach the blob server to handle --prefix

hgweb: fallback to checking wsgireq.env for REPO_NAME for 3rd party hosting Starting with d7fd203e36cc, SCM Manager began to 404 any repository access. What's happening is that it is generating a python script that creates an hgweb application (not hgwebdir), and launches hgweb via wsgicgi. It must be setting REPO_NAME in the process environment before launching this script, which gets picked up and put into wsgireq.env when wsgicgi launches the hgweb application. >From there, other variables (notably 'apppath' and 'dispatchpath') are constructed differently. d7fd203e36cc^ (working): apppath: /hg/eng/devsetup dispatchpath: pathinfo: /eng/devsetup reponame: eng/devsetup d7fd203e36cc: apppath: /hg dispatchpath: eng/devsetup pathinfo: /eng/devsetup reponame: None REPO_NAME: eng/devsetup Rather than having an existing installation break when Mercurial is upgraded, just resume checking the environment. I have no idea how many other hosting solutions would break without restoring this.

peer: scatter module to the wind (API) peer.py hardly contained any code. The code it did contain was generic to the version 1 peer interface or specific to the local repository peer. So code has been moved to wireprotov1peer and localrepo, as appropriate. Differential Revision: https://phab.mercurial-scm.org/D3260

wireproto: move version 1 peer functionality to standalone module (API) wireproto.py contains code for both the client and the server. There *should* be a somewhat strong separation between the two. This commit extracts the client-side code from wireproto.py into a new module - wireprotov1peer. Differential Revision: https://phab.mercurial-scm.org/D3259

wireproto: move gboptsmap to wireprototypes and rename (API) This is also shared between client and server and will need to exist in a shared module when that code is split into different modules. Differential Revision: https://phab.mercurial-scm.org/D3258

wireproto: move value encoding functions to wireprototypes (API) These functions should live in the same place. I plan to separate client from server code in upcoming commits. wireprototypes is where we are putting shared code like this. Differential Revision: https://phab.mercurial-scm.org/D3257

httppeer: basic implementation of capabilities interface This is a bit crude. The capabilities mechanism for version 2 of the wire protocol is a bit different from version 1. And code in core is relying on strings passed to capable() matching strings advertised by the "capabilities" wire protocol command. I may refactor the internal checking mechanism to be a bit more abstract or based on interfaces. Time will tell... Differential Revision: https://phab.mercurial-scm.org/D3256

repository: split capabilities methods into separate interface So we can implement them without having to implement support for every command. Differential Revision: https://phab.mercurial-scm.org/D3255

httppeer: implement ipeerconnection This is low hanging fruit. We might as well start somewhere. Differential Revision: https://phab.mercurial-scm.org/D3254