debugcommands: support sending HTTP requests with debugwireproto We implement an action that can issue an HTTP request. We can define headers via arguments and specify a file to use for the HTTP request body. The request uses the HTTP peer's opener, which is already configured for auth, etc. This is both good and bad. Good in that we get some nice behavior out of the box. Bad in that some HTTP request headers are added automatically. Differential Revision: https://phab.mercurial-scm.org/D2841

debugcommands: support connecting to HTTP peers Now that we have the plumbing for logging socket activity, let's hook it up to `hg debugwireproto` so we can collect low-level activity on sockets. The new code is a bit incomplete. But it is better than nothing: `hg debugwireproto` is still heavily evolving. The added test demonstrates some interesting behavior. For example, we're calling socket.makefile() and doing I/O on that. TIL. We're also sending an "Accept-Encoding: identity" request header. Differential Revision: https://phab.mercurial-scm.org/D2726

url: add HTTP handler that uses a proxied socket Now that we have a socket proxy that can log I/O, we need to teach mechanisms that open URLs how to use it. We invent a custom HTTP handler class that knows how to proxy sockets as soon as they are opened. We teach the high-level opener() to accept logging arguments so a logging HTTP handler can be constructed. We don't yet support intercepting HTTPS sockets because I don't want to go down that rabbit hole. For the record, the urllib API is crazy and it took way too long to figure out at what levels I needed to plug in to modify the socket. # no-check-commit because we must name http_open that way Differential Revision: https://phab.mercurial-scm.org/D2722

util: observable proxy objects for sockets We previously introduced proxy objects and observers for file objects to help implement low-level tests for the SSH wire protocol. In this commit, we do the same for sockets in order to help test the HTTP server. We only proxy/observe some socket methods. I didn't feel like implementing all the methods because there are so many of them and implementing them will provide no short term value. We can always implement them later. # no-check-commit because we implement foo_bar methods on stdlib types Differential Revision: https://phab.mercurial-scm.org/D2721

hgweb: allow defining Server response header for HTTP server By default, the string contains the Python version. Server operators may want to change this. Because we like deterministic tests, we change the test harness to always set this value to a known string. Various globs over the server header have now been removed. # no-check-commit because we add version_string() Differential Revision: https://phab.mercurial-scm.org/D2840

tests: use $HTTP_DATE$ for Date header Support for the $HTTP_DATE$ substitution was recently added. Let's adopt it more widely. We had to tweak the substitution to be case insensitive, since HTTP headers are case insensitive. I also found a minor test issue not globbing over the length of the Server response header. Differential Revision: https://phab.mercurial-scm.org/D2839

debugcommands: introduce actions to perform deterministic reads "readavailable" is useful as a debugging device to see what data is available on a pipe. But the mechanism isn't deterministic because what's available on a pipe is highly conditional on timing, system load, OS behavior, etc. This makes it not suitable for tests. We introduce "ereadline," "read," and "eread" for performing deterministic I/O operations (at least on blocking file descriptors). We stop short of converting existing consumers of "readavailable" in tests because we're working out race conditions and deadlocks on Windows. But the goal is to eventually move tests away from "readavailable" to these new APIs. Differential Revision: https://phab.mercurial-scm.org/D2720

httppeer: refactor how httppeer is created (API) Previously, we passed a bunch of arguments to httppeer.__init__, validated them, then possibly constructed a valid instance. A short while ago, we refactored sshpeer so all the validation and setup work occurs before the constructor. We introduced a makepeer() to hold most of this logic. This commit gives httppeer the same treatment. As a sign that the previous design was poor, __del__ no longer conditionally checks for the presence of an attribute that may not be defined (it is always defined in the new code). .. api:: httppeer.httppeer.__init__ now takes additional arguments. Instances should be obtained by calling httppeer.instance() or httppeer.makepeer() instead. Differential Revision: https://phab.mercurial-scm.org/D2725

bundle: condition the changegroup part when creating a new bundle We will generate stream bundle in the next changesets which doesn't need the changegroup part. Differential Revision: https://phab.mercurial-scm.org/D1951

stack: return a sorted smartrev by default Most stack customers will display a list of revisions, sort it by default. Differential Revision: https://phab.mercurial-scm.org/D2399