wireprotoserver: document and improve the httplib workaround This workaround dates all the way back to a42d27bc809d in 2008. The code is esoteric enough to warrant an inline explanation. So I've added one. At the time the code was written, the only wire protocol command that accepted an HTTP request body was "unbundle." In the years since, we've grown the ability to accept command arguments via HTTP POST requests. So, the code has been changed to apply the httplib workaround to all HTTP POST requests. While staring at this code, I realized that the HTTP response body in case of error is always the same. And, it appears to mimic the behavior of a failed call to the "unbundle" command. Since we can hit this code path on theoretically any protocol request (since self.check_perm accepts custom auth checking functions which may raise), I'm having a hard time believing that clients react well to an "unbundle" response payload on any wire protocol command. I wouldn't be surprised if our test coverage for this feature only covers HTTP POST calls to "unbundle." In other words, the experimental support for sending arguments via HTTP POST request bodies may result in badness on the client. Something to investigate another time perhaps... Differential Revision: https://phab.mercurial-scm.org/D2064

wireprotoserver: move error response handling out of hgweb The exception handler for ErrorResponse has more to do with the wire protocol than the generic HTTP server. Move the code so it lives alongside other wire protocol code. Differential Revision: https://phab.mercurial-scm.org/D2021

hgweb: move call to protocol handler outside of try..except The protocol handler doesn't raise ErrorResponse. So it doesn't need to be in this `try..except ErrorResponse` block. Differential Revision: https://phab.mercurial-scm.org/D2020

wireprotoserver: move protocol parsing and dispatch out of hgweb Previously, hgweb_mod had code for detecting if the request was for the wire protocol. It would then (eventually) call wireprotoserver.callhttp() to dispatch the request handling. Detection of wire protocol requests is not trivial. There's currently a big gotcha in the handling of the "cmd" request parameter, for example. Furthermore, in the near future we will have a second HTTP protocol handler. Its mechanism for calling commands will be a bit different. And we don't want the low-level logic for detecting protocol commands to live in hgweb. We establish a new function in wireprotoserver for detecting an HTTP protocol request and for giving the caller an easy-to-use mechanism for dispatching requests to it. Some wire protocol specific functionality still lives in hgweb. This will be addressed in subsequent commits. Differential Revision: https://phab.mercurial-scm.org/D2019

largefiles: register wire protocol commands with modern APIs The wireproto.wireprotocommand decorator is the preferred mechanism for registering wire protocol commands. In addition, wireproto.commands is no longer a 2-tuple and use of that 2-tuple API should be considered deprecated. This commit ports largefiles to use wireproto.wireprotocommand() and ports to the "commandentry" API. As part of this, the definition of the "lheads" wire protocol command is moved to the proper stanza. We stop short of actually using wireprotocommand as a decorator in order to minimize churn. We should ideally move wire protocol commands to the registrar mechanism. But that's for another changeset. Differential Revision: https://phab.mercurial-scm.org/D2018

wireproto: function for testing if wire protocol command is available Currently, we perform simple membership testing for whether a wire command is available. In the future, not all wire protocol commands will be available on all transports. For example, a legacy transport may not support newer commands. In preparation of this, teach the protocol handlers to call into a function to determine if a wire protocol command is available. That function currently does membership testing like before, so behavior should be identical. In the case of the HTTP server, behavior is a bit wonkier. "cmd" is used by both the wire protocol and hgweb. We do want the protocol handler to handle requests for all commands that look like wire protocol commands, even if they aren't available. Otherwise, the fallback to hgweb would only confuse automated clients and make it easier for hgweb to accidentally implement a "cmd" that is identical to wire protocol commands (since they aren't centrally registered). Differential Revision: https://phab.mercurial-scm.org/D1999

wireproto: define and use types for wire protocol commands Wire protocol commands have historically been declared as 2-tuples in wireproto.commands. There are some additional features I'd like to implement that require going beyond 2-tuples. But because the 2-tuple API (both destructuring assignment and direct assignment into the dict) is used throughout the code base and in 3rd party extensions, we can't do a trivial type change. This commit creates a new "commandentry" type to represent declared wire protocol commands. It implements __getitem__ and __iter__ so it can quack like a 2-tuple. The @wireprotocommand decorator now creates "commandentry" instances. We also create a "commanddict" type to represent the dictionary of declared wire protocol commands. It inherits from "dict" but provides a custom __setitem__ to coerce passed 2-tuples to "commandentry" instances. wireproto.commands is now an instance of this type. Various callers in core rely on the new functionality. And tests pass. So I'm reasonably confident things will "just work" in 3rd party extensions as well. Differential Revision: https://phab.mercurial-scm.org/D1998

wireproto: improve docstring for @wireprotocommand I'm about to add more arguments and want them to be documented. Plus, good documentation is nice to have. Differential Revision: https://phab.mercurial-scm.org/D1997

wireproto: remove unnecessary exception trapping The `try..except error.Abort` was added in 8474be4412ca back in 2012. The intent was to ensure a failing pushkey hook didn't crash the server. Since that changeset, repo.pushkey() and the hooks mechanism is now much more robust about trapping errors itself. As such, we no longer need this try..except block. So it has been removed. Differential Revision: https://phab.mercurial-scm.org/D1996

sshpeer: implement peer for version 2 of wire protocol Since the protocol is now negotiated before we construct a peer instance, we can return the negotiated protocol from the handshake function and instantiate an appropriate peer class for the protocol. Version 2 of the SSH protocol is currently identical to version 1 post handshake. So our sshv2peer class just inherits from sshv1peer for the time being. This will obviously change over time. Differential Revision: https://phab.mercurial-scm.org/D2063

sshpeer: rename sshpeer class to sshv1peer (API) With the introduction of version 2 of the SSH wire protocol, we will need a new peer class to speak that protocol because it will be too difficult to shoehorn a single class to speak two protocols. We rename sshpeer.sshpeer to sshpeer.sshv1peer to reflect the fact that there will be multiple versions of the peer depending on the negotiated protocol. .. api:: sshpeer.sshpeer renamed to sshpeer.sshv1peer. Differential Revision: https://phab.mercurial-scm.org/D2062

sshpeer: initial definition and implementation of new SSH protocol The existing SSH protocol has several design flaws. Future commits will elaborate on these flaws as new features are introduced to combat these flaws. For now, hopefully you can take me for my word that a ground up rewrite of the SSH protocol is needed. This commit lays the foundation for a new SSH protocol by defining a mechanism to upgrade the SSH transport channel away from the default (version 1) protocol to something modern (which we'll call "version 2" for now). This upgrade process is detailed in the internals documentation for the wire protocol. The gist of it is the client sends a request line preceding the "hello" command/line which basically says "I'm requesting an upgrade: here's what I support." If the server recognizes that line, it processes the upgrade request and the transport channel is switched to use the new version of the protocol. If not, it sends an empty response, which is how all Mercurial SSH servers from the beginning of time reacted to unknown commands. The upgrade request is effectively ignored and the client continues to use the existing version of the protocol as if nothing happened. The new version of the SSH protocol is completely identical to version 1 aside from the upgrade dance and the bytes that follow. The immediate bytes that follow the protocol switch are defined to be a length framed "capabilities: " line containing the remote's advertised capabilities. In reality, this looks very similar to what the "hello" response would look like. But it will evolve quickly. The methodology by which the protocol will evolve is important. I'm not going to introduce the new protocol all at once. That would likely lead to endless bike shedding and forward progress would stall. Instead, I intend to tricle out new features and diversions from the existing protocol in small, incremental changes. To support the gradual evolution of the protocol, the on-the-wire advertised protocol name contains an "exp" to denote "experimental" and a 4 digit field to capture the sub-version of the protocol. Whenever we make a BC change to the wire protocol, we can increment this version and lock out all older clients because it will appear as a completely different protocol version. This means we can incur as many breaking changes as we want. We don't have to commit to supporting any one feature or idea for a long period of time. We can even evolve the handshake mechanism, because that is defined as being an implementation detail of the negotiated protocol version! Hopefully this lowers the barrier to accepting changes to the protocol and for experimenting with "radical" ideas during its development. In core, sshpeer received most of the attention. We haven't even implemented the server bits for the new protocol in core yet. Instead, we add very primitive support to our test server, mainly just to exercise the added code paths in sshpeer. Differential Revision: https://phab.mercurial-scm.org/D2061 # no-check-commit because of required foo_bar naming

internals: refactor wire protocol documentation Upcoming work will introduce a new version of the HTTP and SSH transports. The differences will be significant enough to consider them new transports. So, we now attach a version number to each transport. In addition, having the handshake documented after the transport and in a single shared section made it harder to follow the flow of the connection. The handshake documentation is now moved to the protocol section it describes. We now have a generic section about the purpose of the handshake, which was rewritten significantly. Differential Revision: https://phab.mercurial-scm.org/D2060

revlog: rename 'self.checkinlinesize' into '_enforceinlinesize' The name change has two motivations: 1) The function has no external caller, so we move it to protected space. 2) the function does more than checking it also split the data if we have more data than 'inline' supports.

revlog: add a _datareadfp context manager for data access needs The helper handles: 1) is there a file handle already open that we shall just reuse, 2) is the revlog inlined or not. Using a context manager for all read access will help setting up file pointer caching in later changesets.

revlog: use context manager for data file lifetime in checksize This is clearer, safer and more modern.

revlog: use context manager for index file lifetime in checkinlinesize This is clearer, safer and more modern.

revlog: use context manager for data file lifetime in checkinlinesize This is clearer, safer and more modern.

revlog: use context manager for index file life time in __init__ This is clearer, safer and more modern.

revlog: move index file opening in a method Having file operation centralized into a single spot help to factor common logic out (eg: special flag handling according to the mode). It is also the first step to simplify file handling during batch operation (eg: revlog cloning). However, that part does not seems to be a hotspot yet.

revlog: move datafile opening in a method Having file operation centralized into a single spot help to factor common logic out. It is also the first step to simplify file handling during batch operation (eg: revlog cloning). However, that part does not seems to be a hotspot yet.

parseindex: implement context manager method on the wrapper This is needed for incoming cleanups.

parseindex: also forward keyword argument in a debug wrapper Otherwise, it gets in the way of a coming refactoring.

dumprevlog: handle being passed a mode parameter This makes it closer to an actual file opener. Otherwise, it gets in the way of a coming refactoring.