Gregory Szorc <gregory.szorc@gmail.com> [Wed, 14 Mar 2018 16:51:34 -0700] rev 37060
wireproto: add request IDs to frames
One of my primary goals with the new wire protocol is to make
operations faster and enable both client and server-side
operations to scale to multiple CPU cores.
One of the ways we make server interactions faster is by reducing
the number of round trips to that server.
With the existing wire protocol, the "batch" command facilitates
executing multiple commands from a single request payload. The way
it works is the requests for multiple commands are serialized. The
server executes those commands sequentially then serializes all
their results. As an optimization for reducing round trips, this
is very effective. The technical implementation, however, is pretty
bad and suffers from a number of deficiencies. For example, it
creates a new place where authorization to run a command must be
checked. (The lack of this checking in older Mercurial releases
was CVE-2018-1000132.)
The principles behind the "batch" command are sound. However, the
execution is not. Therefore, I want to ditch "batch" in the
new wire protocol and have protocol level support for issuing
multiple requests in a single round trip.
This commit introduces support in the frame-based wire protocol to
facilitate this. We do this by adding a "request ID" to each frame.
If a server sees frames associated with different "request IDs," it
handles them as separate requests. All of this happening possibly
as part of the same message from client to server (the same request
body in the case of HTTP).
We /could/ model the exchange the way pipelined HTTP requests do,
where the server processes requests in order they are issued and
received. But this artifically constrains scalability. A better
model is to allow multi-requests to be executed concurrently and
for responses to be sent and handled concurrently. So the
specification explicitly allows this. There is some work to be done
around specifying dependencies between multi-requests. We take
the easy road for now and punt on this problem, declaring that
if order is important, clients must not issue the request until
responses to dependent requests have been received.
This commit focuses on the boilerplate of implementing the request
ID. The server reactor still can't manage multiple, in-flight
request IDs. This will be addressed in a subsequent commit.
Because the wire semantics have changed, we bump the version of the
media type.
Differential Revision: https://phab.mercurial-scm.org/D2869
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 14 Mar 2018 14:01:16 -0700] rev 37059
wireproto: buffer output frames when in half duplex mode
Previously, when told that a response was ready, the server reactor
would instruct the caller to send frames immediately. This was OK
as an initial implementation. But it would not work for half-duplex
connections where the sender can't receive until all data has been
transmitted - such as httplib based clients.
In this commit, we teach the reactor that output frames should
be buffered until end of input is seen. This required a new
event to inform the reactor of end of input. The result from that
event will instruct the consumer to send all buffered frames.
The HTTP server is buffered by default.
This change effectively hides the complexity of buffering within
the reactor so that transports need not be concerned about it.
This helps keep the transports "dumb" and will make implementing
multiple requests-responses per atomic exchange (like an HTTP
request) much simpler.
Differential Revision: https://phab.mercurial-scm.org/D2860
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 14 Mar 2018 13:57:52 -0700] rev 37058
wireproto: define and implement responses in framing protocol
Previously, we only had client-side frame types defined. This commit
defines and implements basic support for server-side frame types.
We introduce two frame types - one for representing the raw bytes
result of a command and another for representing error results.
The types are quite primitive and behavior will expand over time.
But you have to start somewhere.
Our server reactor gains methods to react to an intent to send a
response. Again, following the "sans I/O" pattern, the reactor
doesn't actually send the data. Instead, it gives the caller a
generator to frames that it can send out over the wire.
Differential Revision: https://phab.mercurial-scm.org/D2858
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 14 Mar 2018 13:32:31 -0700] rev 37057
wireproto: implement basic command dispatching for HTTPv2
Now that we can ingest frames and decode them to requests to run
commands, we are able to actually run those commands. So this
commit starts to implement that.
There are numerous shortcomings. We can't operate on commands
with "*" arguments. We can only emit bytesresponse results. We
don't yet issue a response in the unified framing protocol.
But it's a start.
Differential Revision: https://phab.mercurial-scm.org/D2857
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 14 Mar 2018 08:18:15 -0700] rev 37056
wireproto: nominally don't expose "batch" to version 2 wire transports
The unified frame-based protocol will (eventually) support
multiple requests per client transmission. This means that the
[very hacky] "batch" command has no purpose existing in this protocol.
This commit marks the command as applying to v1 transports only.
But because SSHv2 == SSHv1 currently, we had to hack it back in
for the SSHv2 transport. Bleh.
Tests changed because the capabilities string changed. The order of
tokens in the string is not important.
Differential Revision: https://phab.mercurial-scm.org/D2856
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 14 Mar 2018 15:25:06 -0700] rev 37055
wireproto: implement basic frame reading and processing
We just implemented support for writing frames. Now let's implement
support for reading them.
The bulk of the new code is for a class that maintains the state of
a server. Essentially, you construct an instance, feed frames to it,
and it tells you what you should do next. The design is inspired by
the "sans I/O" movement and the reactor pattern. We don't want to
perform I/O or any major blocking event during frame ingestion because
this arbitrarily limits ways that server pieces can be implemented.
For example, it makes it much harder to swap in an alternate
implementation based on asyncio or do crazy things like have requests
dispatch to other processes.
We do still implement readframe() which does I/O. But it is decoupled
from the server reactor. And important parsing of frame headers is
a standalone function. So I/O is only needed to obtain frame data.
Because testing server-side ingest is useful and difficult on running
servers, we create a new "debugreflect" endpoint that will echo back
to the client what was received and how it was interpreted. This could
be useful for a server admin, someone implementing a client. But
immediately, it is useful for testing: we're able to demonstrate that
frames are parsed correctly and turned into requests to run commands
without having to implement command dispatch on the server!
In addition, we implement Python level unit tests for the reactor.
This is vastly more efficient than sending requests to the
"debugreflect" endpoint and vastly more powerful for advanced
testing.
Differential Revision: https://phab.mercurial-scm.org/D2852
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 19 Mar 2018 16:49:53 -0700] rev 37054
wireproto: define and implement protocol for issuing requests
The existing HTTP and SSH wire protocols suffer from a host of flaws
and shortcomings. I've been wanting to rewrite the protocol for a while
now. Supporting partial clone - which will require new wire protocol
commands and capabilities - and other advanced server functionality
will be much easier if we start from a clean slate and don't have
to be constrained by limitations of the existing wire protocol.
This commit starts to introduce a new data exchange format for
use over the wire protocol.
The new protocol is built on top of "frames," which are atomic
units of metadata + data. Frames will make it easier to implement
proxies and other mechanisms that want to inspect data without
having to maintain state. The existing frame metadata is very
minimal and it will evolve heavily. (We will eventually support
things like concurrent requests, out-of-order responses,
compression, side-channels for status updates, etc. Some of
these will require additions to the frame header.)
Another benefit of frames is that all reads are of a fixed size.
A reader works by consuming a frame header, extracting the payload
length, then reading that many bytes. No lookahead, buffering, or
memory reallocations are needed.
The new protocol attempts to be transport agnostic. I want all that's
required to use the new protocol to be a pair of unidirectional,
half-duplex pipes. (Yes, we will eventually make use of full-duplex
pipes, but that's for another commit.) Notably, when the SSH
transport switches to this new protocol, stderr will be unused.
This is by design: the lack of stderr on HTTP harms protocol
behavior there. By shoehorning everything into a pair of pipes,
we can have more consistent behavior across transports.
We currently only define the client side parts of the new protocol,
specifically the bits for requesting that a command run. This keeps
the new code and feature small and somewhat easy to review.
We add support to `hg debugwireproto` for writing frames into
HTTP request bodies. Our tests that issue commands to the new
HTTP endpoint have been updated to transmit frames. The server
bits haven't been touched to consume the frames yet. This will
occur in the next commit...
Astute readers may notice that the command name is transmitted in
both the HTTP request URL and the command request frame. This is
partially a kludge from me initially implementing the frame-based
protocol for SSH first. But it is also a feature: I intend to
eventually support issuing multiple commands per HTTP request. This
will allow us to replace the abomination that is the "batch" wire
protocol command with a protocol-level mechanism for performing
multi-dispatch. Because I want the frame-based protocol to be
as similar as possible across transports, I'd rather we (redundantly)
include the command name in the frame than differ behavior between
transports that have out-of-band routing information (like HTTP)
readily available.
Differential Revision: https://phab.mercurial-scm.org/D2851
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 13 Mar 2018 19:44:59 -0700] rev 37053
wireproto: define content negotiation for HTTPv2
HTTP messages communicate their media types and what media types
they can understand via the Content-Type and Accept header,
respectively.
While I don't want the wire protocol to lean too heavily on HTTP
because I'm aiming for the wire protocol to be as transport
agnostic as possible, it is nice to play by the spec if possible.
This commit defines our media negotiation mechanism for version
2 of the HTTP protocol. Essentially, we mandate the use of a
new media type and how clients and servers should react to
various headers or lack thereof.
The name of the media type is a placeholder. We purposefully don't
yet define the format of the new media type because that's a lot
of work.
I feel pretty strongly that we should use Content-Type. I feel
less strongly about Accept. I think it is reasonable for servers
to return the media type that was submitted to them. So we may
strike this header before the protocol is finished...
Differential Revision: https://phab.mercurial-scm.org/D2850
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 13 Mar 2018 14:15:10 -0700] rev 37052
hgweb: also set Content-Type header
Our HTTP/WSGI server may convert the Content-Type HTTP request
header to the CONTENT_TYPE WSGI environment key and not set
HTTP_CONTENT_TYPE. Other WSGI server implementations
do this, so I think the behavior is acceptable.
So assuming this HTTP request header could get "lost" by the WSGI
server, let's restore it on the request object like we do for
Content-Length.
FWIW, the WSGI server may also *invent* a Content-Type value. The
default behavior of Python's RFC 822 message class returns a default
media type if Content-Type isn't defined. This is kind of annoying.
But RFC 7231 section 3.1.1.5 does say the recipient may assume a media
type of application/octet-stream. Python's defaults are for
text/plain (given we're using an RFC 822 parser). But whatever.
Differential Revision: https://phab.mercurial-scm.org/D2849
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 13 Mar 2018 11:57:43 -0700] rev 37051
wireproto: require POST for all HTTPv2 requests
Wire protocol version 1 transfers argument data via request
headers by default. This has historically caused problems because
servers institute limits on the length of individual HTTP headers
as well as the total size of all request headers. Mercurial servers
can advertise the maximum length of an individual header. But
there's no guarantee any intermediate HTTP agents will accept
headers up to that length.
In the existing wire protocol, server operators typically also
key off the HTTP request method to implement authentication.
For example, GET requests translate to read-only requests and
can be allowed. But read-write commands must use POST and require
authentication. This has typically worked because the only wire
protocol commands that use POST modify the repo (e.g. the
"unbundle" command).
There is an experimental feature to enable clients to transmit
argument data via POST request bodies. This is technically a
better and more robust solution. But we can't enable it by default
because of servers assuming POST means write access.
In version 2 of the wire protocol, the permissions of a request
are encoded in the URL. And with it being a new protocol in a new
URL space, we're not constrained by backwards compatibility
requirements.
This commit adopts the technically superior mechanism of using
HTTP request bodies to send argument data by requiring POST for
all commands. Strictly speaking, it may be possible to send
request bodies on GET requests. But my experience is that not all
HTTP stacks support this. POST pretty much always works. Using POST
for read-only operations does sacrifice some RESTful design
purity. But this API cares about practicality, not about being
in Roy T. Fielding's REST ivory tower.
There's a chance we may relax this restriction in the future. But
for now, I want to see how far we can get with a POST only API.
Differential Revision: https://phab.mercurial-scm.org/D2837
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 19 Mar 2018 16:43:47 -0700] rev 37050
wireproto: define permissions-based routing of HTTPv2 wire protocol
Now that we have a scaffolding for serving version 2 of the HTTP
protocol, let's start implementing it.
A good place to start is URL routing and basic request processing
semantics. We can focus on content types, capabilities detect, etc
later.
Version 2 of the HTTP wire protocol encodes the needed permissions
of the request in the URL path. The reasons for this are documented
in the added documentation. In short, a) it makes it really easy and
fail proof for server administrators to implement path-based
authentication and b) it will enable clients to realize very early in
a server exchange that authentication will be required to complete
the operation. This latter point avoids all kinds of complexity and
problems, like dealing with Expect: 100-continue and clients finding
out later during `hg push` that they need to provide authentication.
This will avoid the current badness where clients send a full bundle,
get an HTTP 403, provide authentication, then retransmit the bundle.
In order to implement command checking, we needed to implement a
protocol handler for the new wire protocol. Our handler is just
small enough to run the code we've implemented.
Tests for the defined functionality have been added.
I very much want to refactor the permissions checking code and define
a better response format. But this can be done later. Nothing is
covered by backwards compatibility at this point.
Differential Revision: https://phab.mercurial-scm.org/D2836
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 13 Mar 2018 16:53:21 -0700] rev 37049
wireproto: support /api/* URL space for exposing APIs
I will soon be introducing a new version of the HTTP wire protocol.
One of the things I want to change with it is the URL routing.
I want to rely on URL paths to define endpoints rather than the
"cmd" query string argument. That should be pretty straightforward.
I was thinking about what URL space to reserve for the new protocol.
We /could/ put everything at a top-level path. e.g.
/wireproto/* or /http-v2-wireproto/*. However, these constrain us
a bit because they assume there will only be 1 API: version 2 of
the HTTP wire protocol. I think there is room to grow multiple
APIs. For example, there may someday be a proper JSON API to query
or even manipulate the repository. And I don't think we should have
to create a new top-level URL space for each API nor should we
attempt to shoehorn each future API into the same shared URL space:
that would just be too chaotic.
This commits reserves the /api/* URL space for all our future API
needs. Essentially, all requests to /api/* get routed to a new WSGI
handler. By default, it 404's the entire URL space unless the
"api server" feature is enabled. When enabled, requests to "/api"
list available APIs. URLs of the form /api/<name>/* are reserved for
a particular named API. Behavior within each API is left up to that
API. So, we can grow new APIs easily without worrying about URL
space conflicts.
APIs can be registered by adding entries to a global dict. This allows
extensions to provide their own APIs should they choose to do so.
This is probably a premature feature. But IMO the code is easier
to read if we're not dealing with API-specific behavior like config
option querying inline.
To prove it works, we implement a very basic API for version 2
of the HTTP wire protocol. It does nothing of value except
facilitate testing of the /api/* URL space.
We currently emit plain text responses for all /api/* endpoints.
There's definitely room to look at Accept and other request headers
to vary the response format. But we have to start somewhere.
Differential Revision: https://phab.mercurial-scm.org/D2834
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 13 Mar 2018 10:34:36 -0700] rev 37048
url: support suppressing Accept header
Sending this header automatically could interfere with future
testing and client behavior. Let's add a knob to disable the
behavior.
We don't have a control for User-Agent because urllib will send
it if we don't set something. I don't feel like hacking into the
bowels of urllib to figure out how to suppress that. UA shouldn't
be used for anything meaningful. So it shouldn't pose any problems
beyond non-determinism (since the header has the Mercurial version in
it).
Differential Revision: https://phab.mercurial-scm.org/D2843
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 13 Mar 2018 11:20:07 -0700] rev 37047
util: don't log low-level I/O calls for HTTP peer
`hg debugwireproto` is useful for testing HTTP interactions. Possibly
more useful than `get-with-headers.py`. But one thing that makes it
annoying for mid-level tests is that it logs all API calls, such
as readline(). This makes output - especially headers - overly
verbose.
This commit teaches our file and socket observers to not log API
calls on functions dealing with data.
We change the behavior of `hg debugwireproto` to enable this mode
by default. --debug can be added to restore the previous behavior.
As the test changes demonstrate, this makes tests much easier to
read. As a bonus, it also removes some required (glob) over lengths
in system call results.
One thing that's lacking is knowing which side sent data. But we can
fix this in a follow-up once it becomes a problem.
Differential Revision: https://phab.mercurial-scm.org/D2842
Martin von Zweigbergk <martinvonz@google.com> [Wed, 21 Mar 2018 09:56:41 -0700] rev 37046
rebase: rename conclude[memory]node() to commit[memory]node()
The functions do little more than commit at this point.
Differential Revision: https://phab.mercurial-scm.org/D2924
Martin von Zweigbergk <martinvonz@google.com> [Tue, 20 Mar 2018 22:11:10 -0700] rev 37045
rebase: pass in "user" instead of "ctx" to conclude[memory]node()
This was the only remaining part of the context object that was
needed.
Differential Revision: https://phab.mercurial-scm.org/D2923
Martin von Zweigbergk <martinvonz@google.com> [Tue, 20 Mar 2018 21:59:04 -0700] rev 37044
rebase: look up default date outside of conclude[memory]node()
Differential Revision: https://phab.mercurial-scm.org/D2922
Martin von Zweigbergk <martinvonz@google.com> [Tue, 20 Mar 2018 21:58:32 -0700] rev 37043
rebase: move config override out of conclude[memory]node()
Differential Revision: https://phab.mercurial-scm.org/D2921
Martin von Zweigbergk <martinvonz@google.com> [Tue, 20 Mar 2018 21:53:36 -0700] rev 37042
rebase: pass in entire "overrides" dict to conclude[memory]node()
As with previous patches, this was done the same way in both
functions, so let's make the caller do it instead.
Differential Revision: https://phab.mercurial-scm.org/D2920
Martin von Zweigbergk <martinvonz@google.com> [Tue, 20 Mar 2018 21:49:09 -0700] rev 37041
rebase: pass in "keepbranch" to conclude[memory]node()
Both functions calculated the same "keepbranch" value from the
"keepbranches" we passed in, so let's make the caller do it instead.
Differential Revision: https://phab.mercurial-scm.org/D2919
Martin von Zweigbergk <martinvonz@google.com> [Tue, 20 Mar 2018 21:45:48 -0700] rev 37040
rebase: inline _makextrafn() now that we have only one caller
Also avoid even creating a function since we just end up calling it
right away.
Differential Revision: https://phab.mercurial-scm.org/D2918
Martin von Zweigbergk <martinvonz@google.com> [Tue, 20 Mar 2018 21:41:53 -0700] rev 37039
rebase: pass in "extra" itself into conclude[memory]node()
We were passing in a function instead for no clear reason (probably
historical, but I haven't bothered looking).
Differential Revision: https://phab.mercurial-scm.org/D2917
Martin von Zweigbergk <martinvonz@google.com> [Wed, 21 Mar 2018 11:05:32 -0700] rev 37038
rebase: look up commit message to reuse outside of conclude[memory]node()
This was done the same way in both functions, so let's let the single
caller do it.
Differential Revision: https://phab.mercurial-scm.org/D2916
Martin von Zweigbergk <martinvonz@google.com> [Wed, 21 Mar 2018 11:04:13 -0700] rev 37037
rebase: pass in ctx, not rev, to conclude[memory]node()
They both need it and there's no locking that might make the results
different, so let's do it in one place. This also lets us move out
more common code in the following patches.
Differential Revision: https://phab.mercurial-scm.org/D2915
Martin von Zweigbergk <martinvonz@google.com> [Wed, 21 Mar 2018 11:03:31 -0700] rev 37036
rebase: extract common _concludenode()
_rebasenode() and _finishrebase() had a lot of code in common. This
commit extracts some of that. This means we will also have a single
caller of concludenode() and concludememorynode(), which gives us a
place to put code that is common between those two functions (which is
quite a bit).
Differential Revision: https://phab.mercurial-scm.org/D2914
Martin von Zweigbergk <martinvonz@google.com> [Wed, 21 Mar 2018 11:01:19 -0700] rev 37035
rebase: store rebase state after each commit
Before this patch, we stored the rebase state early in the processing
of a node, before we updated the rebase state to indicate that the
node was processed. This meant that we could redo the working copy
merge and run into conflicts. However, this only happened in the
--collapse case if the rebase was interrupted while editing the final
commit message; in the case earlier interruptions, we would instead
detect the in-process revision by finding two dirstate parents.
This patch moves the writing of the rebase state to after we have
completed the revision completely, and, importantly, after we have
updated the rebase state to mark it done. This means we'll realize
that all nodes have been rebased in the case mentioned above of
editing the final commit message of a --collapse. See change to test
case.
I also moved the writing outside of the large if/elif block in
_rebasenode(). This shouldn't matter much, but seems cleaner. One
observable effect is if rebase was interrupted just after ignoring an
obsolete node ("not rebasing ####, already in destination"), we used
to come up with the same decision after --continue too, but after this
patch we'll instead say "already rebased ###". This seems more
consistent, since that's what we would do with obsolete nodes that had
been marked done earlier in the process (not only just before the
interruption).
Differential Revision: https://phab.mercurial-scm.org/D2913
Martin von Zweigbergk <martinvonz@google.com> [Wed, 21 Mar 2018 10:46:00 -0700] rev 37034
rebase: register status file generator only once when using single transaction
rebase.storestatus() behaved differently depending on whether a
transaction is passed to it. If a transaction is passed, it registers
a "file generator" that runs when the transaction commits. If no
transaction was passed, it writes the rebase state immediately. This
imprecise timing of the writing makes it hard to reason about, so
let's make it more explicit which behavior we're getting by checking
if we have a transaction before calling it. For the single-transaction
case, move the call to storestatus(tr) early and do it only once since
it's only going to write the file (at most) once anyway.
Differential Revision: https://phab.mercurial-scm.org/D2912
Martin von Zweigbergk <martinvonz@google.com> [Sun, 18 Mar 2018 22:26:23 -0700] rev 37033
tests: add some more tests to test-rebase-collapse.t
Some of these are currently broken.
Differential Revision: https://phab.mercurial-scm.org/D2911
Martin von Zweigbergk <martinvonz@google.com> [Sun, 18 Mar 2018 15:25:23 -0700] rev 37032
tests: use drawdag in test-rebase-collapse.t
This makes the tests shorter (410 lines -> 240 lines), faster (17s ->
12s), and easier to read (the setup for each test case is just before
the test, and we can refer to commits by name).
Differential Revision: https://phab.mercurial-scm.org/D2910
Martin von Zweigbergk <martinvonz@google.com> [Tue, 20 Mar 2018 17:25:16 -0700] rev 37031
rebase: remove unused default argument values from conclude[memory]node()
It's only "commitmsg" that we don't always pass.
Differential Revision: https://phab.mercurial-scm.org/D2909
Martin von Zweigbergk <martinvonz@google.com> [Tue, 20 Mar 2018 17:30:08 -0700] rev 37030
rebase: get "inmemory" state directly from rebase runtime
As far as I can tell, rbsrt.inmemory is equivalent to
rbsrt.wctx.isinmemory(), so let's use the shorter form.
Differential Revision: https://phab.mercurial-scm.org/D2908
Pulkit Goyal <7895pulkit@gmail.com> [Mon, 25 Dec 2017 15:56:07 +0530] rev 37029
commands: use keyword arguments in update function
This will help us in having a dictionary with the values of all the arguments
and we can add more flags without adding an argument to the function.
Differential Revision: https://phab.mercurial-scm.org/D2896
Martin von Zweigbergk <martinvonz@google.com> [Tue, 20 Mar 2018 13:53:02 -0700] rev 37028
rebase: pass "inmemory" directly to _definedestmap()
We no longer reassign rbsrt.inmemory in _definedestmap(), so we don't
need to pass the whole rebase runtime instance anymore, thus
making it clear that it won't be updated.
Differential Revision: https://phab.mercurial-scm.org/D2905
Anton Shestakov <av6@dwimlabs.net> [Tue, 20 Mar 2018 14:19:28 +0800] rev 37027
hgweb: explain instabilities of unstable changesets (the rest of the themes)
Matt Harbison <matt_harbison@yahoo.com> [Tue, 20 Mar 2018 21:10:29 -0400] rev 37026
test-merge-tools: stabilize for Windows
See fe5c4b795999.
Matt Harbison <matt_harbison@yahoo.com> [Tue, 20 Mar 2018 20:33:37 -0400] rev 37025
test-http-protocol: drop an extraneous glob for Windows
Martin von Zweigbergk <martinvonz@google.com> [Mon, 19 Mar 2018 08:07:18 -0700] rev 37024
merge with stable
Yuya Nishihara <yuya@tcha.org> [Sun, 03 Apr 2016 14:16:47 +0900] rev 37023
hgweb: evaluate the "default" value as template
Strictly speaking, everything in the map file is a template. So let's not
take out an unparsed template string.
Yuya Nishihara <yuya@tcha.org> [Fri, 16 Mar 2018 21:39:32 +0900] rev 37022
templater: use named function to expand template against mapping dict (API)
And replace __call__(t, **mapping) in favor of generate(t, mapping). I prefer
a named function here since the templater isn't a simple function-like object.
.. api::
The templater is no longer callable. Use ``templater.generate(t, mapping)``
instead of ``templater(t, **pycompat.strkwargs(mapping))``.
Yuya Nishihara <yuya@tcha.org> [Mon, 19 Mar 2018 21:09:23 +0900] rev 37021
highlight: peek Content-Type header set by hgweb
There should be no need to re-render the mimetype template since it's set
before dispatching webcommands.
Anton Shestakov <av6@dwimlabs.net> [Tue, 20 Mar 2018 14:12:05 +0800] rev 37020
obsolete: add a note that explains creating aliases for marker flags
Gregory Szorc <gregory.szorc@gmail.com> [Sun, 04 Mar 2018 10:34:09 -0500] rev 37019
repair: rename _backup to backupbundle
A future commit will introduce a caller from outside the module.
The function should have already been public because histedit was
calling it.
Differential Revision: https://phab.mercurial-scm.org/D2666
Yuya Nishihara <yuya@tcha.org> [Mon, 19 Mar 2018 22:10:40 +0900] rev 37018
templatefuncs: do not stringify result of if*() expression
Returning a generator means that the result is a byte string. I can't find
any reason to make the "if" condition lazy since it is evaluated anyway
when {if()} has to be evaluated. So let's simply make if*() return an input
expression unmodified.
Yuya Nishihara <yuya@tcha.org> [Mon, 19 Mar 2018 21:55:02 +0900] rev 37017
templater: add brief doc about internal data types
Hopefully this will help reviewing upcoming my series. Start small.
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 13 Mar 2018 11:17:10 -0700] rev 37016
debugcommands: support sending HTTP requests with debugwireproto
We implement an action that can issue an HTTP request. We can define
headers via arguments and specify a file to use for the HTTP request
body.
The request uses the HTTP peer's opener, which is already configured
for auth, etc. This is both good and bad. Good in that we get some
nice behavior out of the box. Bad in that some HTTP request headers
are added automatically.
Differential Revision: https://phab.mercurial-scm.org/D2841
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 13 Mar 2018 10:02:03 -0700] rev 37015
debugcommands: support connecting to HTTP peers
Now that we have the plumbing for logging socket activity, let's hook
it up to `hg debugwireproto` so we can collect low-level activity on
sockets.
The new code is a bit incomplete. But it is better than nothing:
`hg debugwireproto` is still heavily evolving.
The added test demonstrates some interesting behavior. For example,
we're calling socket.makefile() and doing I/O on that. TIL. We're also
sending an "Accept-Encoding: identity" request header.
Differential Revision: https://phab.mercurial-scm.org/D2726
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 12 Mar 2018 15:43:36 -0700] rev 37014
url: add HTTP handler that uses a proxied socket
Now that we have a socket proxy that can log I/O, we need to teach
mechanisms that open URLs how to use it.
We invent a custom HTTP handler class that knows how to proxy
sockets as soon as they are opened. We teach the high-level
opener() to accept logging arguments so a logging HTTP handler
can be constructed.
We don't yet support intercepting HTTPS sockets because I
don't want to go down that rabbit hole.
For the record, the urllib API is crazy and it took way too long
to figure out at what levels I needed to plug in to modify
the socket.
# no-check-commit because we must name http_open that way
Differential Revision: https://phab.mercurial-scm.org/D2722
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 13 Mar 2018 17:42:00 -0700] rev 37013
util: observable proxy objects for sockets
We previously introduced proxy objects and observers for file objects
to help implement low-level tests for the SSH wire protocol.
In this commit, we do the same for sockets in order to help test the
HTTP server.
We only proxy/observe some socket methods. I didn't feel like
implementing all the methods because there are so many of them and
implementing them will provide no short term value. We can always
implement them later.
# no-check-commit because we implement foo_bar methods on stdlib types
Differential Revision: https://phab.mercurial-scm.org/D2721
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 13 Mar 2018 09:45:56 -0700] rev 37012
hgweb: allow defining Server response header for HTTP server
By default, the string contains the Python version. Server
operators may want to change this.
Because we like deterministic tests, we change the test harness
to always set this value to a known string. Various globs over
the server header have now been removed.
# no-check-commit because we add version_string()
Differential Revision: https://phab.mercurial-scm.org/D2840
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 13 Mar 2018 08:58:52 -0700] rev 37011
tests: use $HTTP_DATE$ for Date header
Support for the $HTTP_DATE$ substitution was recently added. Let's
adopt it more widely.
We had to tweak the substitution to be case insensitive, since
HTTP headers are case insensitive. I also found a minor test
issue not globbing over the length of the Server response header.
Differential Revision: https://phab.mercurial-scm.org/D2839
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 12 Mar 2018 15:49:02 -0700] rev 37010
debugcommands: introduce actions to perform deterministic reads
"readavailable" is useful as a debugging device to see what data is
available on a pipe. But the mechanism isn't deterministic because
what's available on a pipe is highly conditional on timing, system
load, OS behavior, etc. This makes it not suitable for tests.
We introduce "ereadline," "read," and "eread" for performing
deterministic I/O operations (at least on blocking file descriptors).
We stop short of converting existing consumers of "readavailable"
in tests because we're working out race conditions and deadlocks
on Windows. But the goal is to eventually move tests away from
"readavailable" to these new APIs.
Differential Revision: https://phab.mercurial-scm.org/D2720
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 07 Mar 2018 20:41:59 -0800] rev 37009
httppeer: refactor how httppeer is created (API)
Previously, we passed a bunch of arguments to httppeer.__init__,
validated them, then possibly constructed a valid instance.
A short while ago, we refactored sshpeer so all the validation and
setup work occurs before the constructor. We introduced a makepeer()
to hold most of this logic.
This commit gives httppeer the same treatment.
As a sign that the previous design was poor, __del__ no longer
conditionally checks for the presence of an attribute that may
not be defined (it is always defined in the new code).
.. api::
httppeer.httppeer.__init__ now takes additional arguments.
Instances should be obtained by calling httppeer.instance()
or httppeer.makepeer() instead.
Differential Revision: https://phab.mercurial-scm.org/D2725
Boris Feld <boris.feld@octobus.net> [Wed, 31 Jan 2018 09:41:47 +0100] rev 37008
bundle: condition the changegroup part when creating a new bundle
We will generate stream bundle in the next changesets which doesn't need the
changegroup part.
Differential Revision: https://phab.mercurial-scm.org/D1951
Boris Feld <boris.feld@octobus.net> [Fri, 19 Jan 2018 17:29:50 +0100] rev 37007
stack: return a sorted smartrev by default
Most stack customers will display a list of revisions, sort it by default.
Differential Revision: https://phab.mercurial-scm.org/D2399
Boris Feld <boris.feld@octobus.net> [Fri, 19 Jan 2018 17:09:24 +0100] rev 37006
histedit: use the new stack definition for histedit
Now that we have a common stack definition, use it in the hg histedit command.
Differential Revision: https://phab.mercurial-scm.org/D2398
Boris Feld <boris.feld@octobus.net> [Fri, 19 Jan 2018 16:52:56 +0100] rev 37005
show: use the new stack definition for show stack
Now that we have a common stack definition, use it in the hg show stack
command.
Differential Revision: https://phab.mercurial-scm.org/D2397
Boris Feld <boris.feld@octobus.net> [Fri, 19 Jan 2018 16:52:02 +0100] rev 37004
stack: import Evolve stack test file
Import Evolve stack test file for starting to have good coverage on stack
definition.
Differential Revision: https://phab.mercurial-scm.org/D2396
Boris Feld <boris.feld@octobus.net> [Fri, 19 Jan 2018 15:25:06 +0100] rev 37003
stack: add a new module for stack-related commands
Bootstrap the module with a function that returns the stack for the current
revision. The definition of the stack is ihnerited from histedit default
revset and used by hg show stack.
Differential Revision: https://phab.mercurial-scm.org/D2395
Kyle Lippincott <spectral@google.com> [Fri, 19 Jan 2018 19:07:58 -0800] rev 37002
filemerge: use a single temp dir instead of temp files
This can help to remove the clutter from UIs that display just the filenames;
instead of seeing foo~local.C9ru9r.txt and foo~base.2DMV22.txt (in the /tmp
directory on most platforms), we create a single new directory and use that,
producing filenames like /tmp/hgmerge.C9ru9r/foo~local.txt.
Differential Revision: https://phab.mercurial-scm.org/D2888
Kyle Lippincott <spectral@google.com> [Fri, 19 Jan 2018 19:14:09 -0800] rev 37001
filemerge: move temp file unlinks to _maketempfiles
Differential Revision: https://phab.mercurial-scm.org/D2887
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 16 Mar 2018 09:41:21 -0700] rev 37000
hgweb: refactor multirequest to be a dict of lists
... instead of a list of 2-tuples.
This makes key lookups faster. The only downside is we lose total
ordering of all entries. But we weren't relying on that before, so
it's no loss.
Differential Revision: https://phab.mercurial-scm.org/D2881
Pulkit Goyal <7895pulkit@gmail.com> [Sun, 04 Mar 2018 22:35:29 +0530] rev 36999
py3: use "%d" % int instead of str(int)
We need to use bytes internally.
Differential Revision: https://phab.mercurial-scm.org/D2895
Pulkit Goyal <7895pulkit@gmail.com> [Mon, 19 Mar 2018 00:06:10 +0530] rev 36998
py3: use pycompat.bytestr() in dirstate.py
This prevents extra b'' prefixes in output.
Differential Revision: https://phab.mercurial-scm.org/D2894
Pulkit Goyal <7895pulkit@gmail.com> [Mon, 19 Mar 2018 00:02:59 +0530] rev 36997
py3: make tests/test-log-linerange.t work on Python 3
Made sure we write bytes to file and we suppress the return value of open().
Differential Revision: https://phab.mercurial-scm.org/D2893
Pulkit Goyal <7895pulkit@gmail.com> [Mon, 19 Mar 2018 00:04:38 +0530] rev 36996
py3: use print as a function in tests/test-narrow-clone-non-narrow-server.t
Differential Revision: https://phab.mercurial-scm.org/D2892
Augie Fackler <augie@google.com> [Sun, 04 Mar 2018 16:23:03 -0500] rev 36995
tests: stop over-specifying tempfile name
Python 3 has more random characters in the default template, which is fine.
Differential Revision: https://phab.mercurial-scm.org/D2676
Matt Harbison <matt_harbison@yahoo.com> [Sat, 10 Mar 2018 23:58:01 -0500] rev 36994
wireproto: explicitly flush stdio to prevent stalls on Windows
This is the key to fixing the hangs on Windows in D2720[1]. I put flushes in a
bunch of other places that didn't help, but I suspect that's more a lack of test
coverage than anything else.
Chasing down stuff like this is pretty painful. I'm wondering if we can put a
proxy around sys.stderr (and sys.stdout?) on Windows (only when daemonized?)
that will flush on every write (or at least every write with a '\n').
[1] https://www.mercurial-scm.org/pipermail/mercurial-devel/2018-March/113352.html
Matt Harbison <matt_harbison@yahoo.com> [Sun, 18 Mar 2018 15:32:49 -0400] rev 36993
test-lfs: glob over some output changes
These changes started with 2090044a288d. They aren't important to LFS testing.
Yuya Nishihara <yuya@tcha.org> [Fri, 16 Mar 2018 21:24:12 +0900] rev 36992
templater: factor out helper that renders named template as string
This is quite common in non-web templating, and **kwargs expansion is annoying
because of the unicode-ness of Python3.
Yuya Nishihara <yuya@tcha.org> [Sat, 17 Mar 2018 11:23:04 +0900] rev 36991
templater: rename .render(mapping) to .renderdefault(mapping) (API)
I'm going to add templ.render(t, mapping) and templ.generate(t, mapping) in
place of stringify(templ(t, **mapping)) and templ(t, **mapping) respectively.
.. api::
The ``render(mapping)`` method of the templater has been renamed to
``renderdefault(mapping)``.
Yuya Nishihara <yuya@tcha.org> [Sat, 17 Mar 2018 15:22:14 +0900] rev 36990
templater: fix invalid reference of runsymbol in doctest
Oversight at da710c3df4dc. I was just lucky that this lambda function is
never evaluated.
Yuya Nishihara <yuya@tcha.org> [Sun, 18 Mar 2018 12:54:03 +0900] rev 36989
cmdutil: drop redundant import of merge module
Yuya Nishihara <yuya@tcha.org> [Sun, 11 Mar 2018 21:36:28 +0900] rev 36988
annotate: add support for template keywords and functions depending on ctx
Yuya Nishihara <yuya@tcha.org> [Sun, 11 Mar 2018 21:26:15 +0900] rev 36987
formatter: unblock storing fctx as a template resource
To keep templateformatter._renderitem() simple, a repo instance is looked
through ctx if available. This is probably good for future subrepo support
where ctx.repo() may be different from the global repo.
Yuya Nishihara <yuya@tcha.org> [Sun, 11 Mar 2018 21:12:02 +0900] rev 36986
templater: process mapping dict by resource callables
A resource item is a callable, so let's make it look up a resource object
by itself.
Yuya Nishihara <yuya@tcha.org> [Sun, 11 Mar 2018 21:05:29 +0900] rev 36985
templater: convert resources to a table of callables for future extension
I'm going to add a full templating support to the annotate command. As the
annotate is a filectx-oriented command, we'll need a way to look up a ctx
from a bounded fctx only when necessary.
This is the minimal change to support that. I'm thinking of defining a proper
interface to look up template resources to fix other issues, but that isn't
ready yet.
(Changes from V1: just updated tests and patch descriptions.)
Augie Fackler <augie@google.com> [Sat, 17 Mar 2018 21:46:17 -0400] rev 36984
merge with stable
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 16 Mar 2018 12:10:15 -0700] rev 36983
hgweb: convert an assert to a ProgrammingError
Because assert may get optimized away.
Differential Revision: https://phab.mercurial-scm.org/D2882
Kyle Lippincott <spectral@google.com> [Fri, 19 Jan 2018 18:52:50 -0800] rev 36982
filemerge: give some variables in _xmerge more descriptive names
Differential Revision: https://phab.mercurial-scm.org/D2886
Martin von Zweigbergk <martinvonz@google.com> [Thu, 15 Mar 2018 13:19:55 -0700] rev 36981
rebase: allow in-memory merge of the working copy parent
Before this patch and when the rebase involved the working copy parent
(and thus the working copy too), we would not do in-memory rebase even
if requested to. The in-code comment explains that the reason had
something to do with avoiding an extra update, but I don't know which
update that refers to. Perhaps an earlier version of the code used to
update to the destination before rebasing even if in-memory rebase was
requested? That seems to not be done at least since aa660c1203a9
(rebase: do not bail on uncomitted changes if rebasing in-memory,
2017-12-07).
To see if this still made it slower, I create a single tiny commit on
top of one branch of the mozilla-unified repo (commit a1098c82 to be
exact) and rebased it to another branch (commit d4e9a7be). Before this
patch that took 11.8s and after this patch it took 8.6s (I only did
two runs each, but the timings were very consistent).
Differential Revision: https://phab.mercurial-scm.org/D2876
Martin von Zweigbergk <martinvonz@google.com> [Sat, 17 Mar 2018 09:33:17 -0700] rev 36980
uncommit: simplify condition for keeping commit
We used to keep the commit if it would have become empty and --keep
was not passed. Since we just changed it so we also keep it if any
patterns were passed on the command line, the only remaining case
where we prune the commit is when no arguments were passed and --keep
was not passed either, we can simplify the "not files and not
allowempty" to just "not allowempty". Let's also rename "allowempty"
to "keepcommit" since it's no longer about allowing an empty commit.
Differential Revision: https://phab.mercurial-scm.org/D2891
Martin von Zweigbergk <martinvonz@google.com> [Sat, 17 Mar 2018 09:49:21 -0700] rev 36979
uncommit: document when the commit will be pruned
Differential Revision: https://phab.mercurial-scm.org/D2890
Martin von Zweigbergk <martinvonz@google.com> [Wed, 14 Mar 2018 11:16:49 -0700] rev 36978
uncommit: leave empty commit if all files are uncommitted
We had instructions for our internal users to add copy information
after commit like this:
hg uncommit <srcfile> <dstfile>
hg cp --after <srcfile> <dstfile>
hg amend
That usually works, but if the rename was the only change in that
commit, then the commit would get pruned. It's easy to fix the recipe:
just pass the --keep option. However, it seems too subtle, so I think
this is an indication that the commit should not be pruned if any
patterns were given.
Differential Revision: https://phab.mercurial-scm.org/D2862
Yuya Nishihara <yuya@tcha.org> [Sat, 17 Mar 2018 00:06:14 +0900] rev 36977
test-template-engine: do not evaluate unused keywords by custom engine
If the custom engine, "mytemplater", were installed as the default, it would
enter to an infinite recursion at stringify(v) because template keywords may
generate a nested mapping containing the same keywords.
Spotted by a future patch which will replace context.resource('templ')(...)
with context.process(...).
Yuya Nishihara <yuya@tcha.org> [Fri, 16 Mar 2018 23:59:13 +0900] rev 36976
py3: make test-template-engine.t bytes-safe
Yuya Nishihara <yuya@tcha.org> [Thu, 15 Mar 2018 20:10:54 +0900] rev 36975
test-template-engine: deduplicate methods of custom template engine
Boris Feld <boris.feld@octobus.net> [Thu, 18 Jan 2018 15:10:22 +0100] rev 36974
revbranchcache: advertise and use 'rbc' exchange capability
The feature is now advertised and use.
Updating the branchmap cache can be very expensive (up to minutes on large
repository) and fetching revision branch data is about 80% of that. Exchanging
the rev branch cache over the wire really help to recover from branch map
invalidation.
(There is a good chance other in flight chance would conflict on
test-http-bad-server.t and other. So here is a small note to help update that
test again: capabilities=19bytes, part-107bytes)
Boris Feld <boris.feld@octobus.net> [Wed, 21 Feb 2018 19:26:41 +0100] rev 36973
revbranchcache: disable the new part for narrow hg bundle
The lack of some revisions confuses the new cache part. To simplify things, we
disable it for now.
Boris Feld <boris.feld@octobus.net> [Thu, 18 Jan 2018 14:58:02 +0100] rev 36972
revbranchcache: add the necessary bit to send 'rbc' data over bundle2
Getbundle is now capable of sending rev-branch-cache information for the
changesets it bundle. The data sent are mostly nodes so it is quite compact.
The goal of the rev-branch-cache is to speed up branch map computation,
especially when the branchmap gets invalidated so we send data for all
exchanged changesets. In addition, computing the relevant heads to send in
case of partial pulling would be challenging.
The feature is still inactive since the capability is not advertised yet.
Boris Feld <boris.feld@octobus.net> [Wed, 21 Feb 2018 17:33:00 +0100] rev 36971
bundle: include advisory rev branch cache part in bundle2 bundle
`hg bundle` command producing bundle2 will now include an optional part
containing the revision-branch cache data.
The data sent are mostly nodes so it is quite compact. The goal of the
rev-branch-cache is to speed up branch map computation, especially when the
branchmap gets invalidated so we send data for all exchanged changesets. In
addition, computing the relevant heads to send in case of partial pulling would
be challenging.
As a reminder, the rev branch cache data significantly speed up branch
computation. Having it around provides a small speedup to pull/clone and much
higher tolerance to branch map cache invalidation that might happens from later
commands.
On the Mercurial repository, computing the visible branchmap from scratch move
from 2.00 seconds to 0.34s (a -83% speedup).
Using this new part, Unbundling the full Mercurial repository moves from 25.736
seconds to 24.030 seconds (around -7% speedup). The bundle size increase is
around 3% (from 22.43 MB to 23.13MB)
On an half a million revision repository with twenty thousand
branches, computing the branchmap moves from 75 seconds to 45 second (-40%) if
the caches is used.
A bundle containing 50 000 changesets in such repository get a 0.5% size
increase from such part for a -3% unbundling time speedup.
Boris Feld <boris.feld@octobus.net> [Wed, 21 Feb 2018 17:26:22 +0100] rev 36970
rev-branch-cache: add a function to generate a part
The function is able to produce a rbc part consumed by the function introduced
into previous changesets. More details on usage and impact in the next
changesets.
Boris Feld <boris.feld@octobus.net> [Wed, 21 Feb 2018 17:35:04 +0100] rev 36969
revbranchcache: add a bundle2 handler for a rbc part
We add the necessary bit to process a part containing rev-branch-cache
information. The local rev branch cache is then updated with the received
information.
Computing branch cache can become a significant part of the time spent pulling.
Boris Feld <boris.feld@octobus.net> [Thu, 18 Jan 2018 14:21:05 +0100] rev 36968
revbranchcache: add a public function to update the data
We want to exchange more cached data over the wire. To do so, we need a clean
way to update the cache on the receiving ends.
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 15 Mar 2018 11:19:16 -0700] rev 36967
httppeer: consolidate _requestbuilder assignments and document
I collapsed multiple assignments because they don't appear to
be necessary. We don't invoke the requestbuilder in anything
called during __init__. So there's no reason to not define it
earlier AFAICT.
This seemingly useless attribute is actually an extension point.
Document it as such.
(A previous version of this patch removed the attribute because
it appeared to just be an alias.)
Differential Revision: https://phab.mercurial-scm.org/D2723
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 05 Mar 2018 00:18:07 -0500] rev 36966
commands: don't check for merge.update() truthiness
AFAICT ``stats`` is always a tuple in these cases. We don't
need to check if the variable has a truthy value.
Differential Revision: https://phab.mercurial-scm.org/D2691
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 07 Mar 2018 19:57:50 -0800] rev 36965
httppeer: alias url as urlmod
"url" is a common variable name. We do this aliasing elsewhere to
avoid shadowing.
Differential Revision: https://phab.mercurial-scm.org/D2724