Gregory Szorc <gregory.szorc@gmail.com> [Tue, 13 Mar 2018 11:57:43 -0700] rev 37048
wireproto: require POST for all HTTPv2 requests
Wire protocol version 1 transfers argument data via request
headers by default. This has historically caused problems because
servers institute limits on the length of individual HTTP headers
as well as the total size of all request headers. Mercurial servers
can advertise the maximum length of an individual header. But
there's no guarantee any intermediate HTTP agents will accept
headers up to that length.
In the existing wire protocol, server operators typically also
key off the HTTP request method to implement authentication.
For example, GET requests translate to read-only requests and
can be allowed. But read-write commands must use POST and require
authentication. This has typically worked because the only wire
protocol commands that use POST modify the repo (e.g. the
"unbundle" command).
There is an experimental feature to enable clients to transmit
argument data via POST request bodies. This is technically a
better and more robust solution. But we can't enable it by default
because of servers assuming POST means write access.
In version 2 of the wire protocol, the permissions of a request
are encoded in the URL. And with it being a new protocol in a new
URL space, we're not constrained by backwards compatibility
requirements.
This commit adopts the technically superior mechanism of using
HTTP request bodies to send argument data by requiring POST for
all commands. Strictly speaking, it may be possible to send
request bodies on GET requests. But my experience is that not all
HTTP stacks support this. POST pretty much always works. Using POST
for read-only operations does sacrifice some RESTful design
purity. But this API cares about practicality, not about being
in Roy T. Fielding's REST ivory tower.
There's a chance we may relax this restriction in the future. But
for now, I want to see how far we can get with a POST only API.
Differential Revision: https://phab.mercurial-scm.org/D2837
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 19 Mar 2018 16:43:47 -0700] rev 37047
wireproto: define permissions-based routing of HTTPv2 wire protocol
Now that we have a scaffolding for serving version 2 of the HTTP
protocol, let's start implementing it.
A good place to start is URL routing and basic request processing
semantics. We can focus on content types, capabilities detect, etc
later.
Version 2 of the HTTP wire protocol encodes the needed permissions
of the request in the URL path. The reasons for this are documented
in the added documentation. In short, a) it makes it really easy and
fail proof for server administrators to implement path-based
authentication and b) it will enable clients to realize very early in
a server exchange that authentication will be required to complete
the operation. This latter point avoids all kinds of complexity and
problems, like dealing with Expect: 100-continue and clients finding
out later during `hg push` that they need to provide authentication.
This will avoid the current badness where clients send a full bundle,
get an HTTP 403, provide authentication, then retransmit the bundle.
In order to implement command checking, we needed to implement a
protocol handler for the new wire protocol. Our handler is just
small enough to run the code we've implemented.
Tests for the defined functionality have been added.
I very much want to refactor the permissions checking code and define
a better response format. But this can be done later. Nothing is
covered by backwards compatibility at this point.
Differential Revision: https://phab.mercurial-scm.org/D2836
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 13 Mar 2018 16:53:21 -0700] rev 37046
wireproto: support /api/* URL space for exposing APIs
I will soon be introducing a new version of the HTTP wire protocol.
One of the things I want to change with it is the URL routing.
I want to rely on URL paths to define endpoints rather than the
"cmd" query string argument. That should be pretty straightforward.
I was thinking about what URL space to reserve for the new protocol.
We /could/ put everything at a top-level path. e.g.
/wireproto/* or /http-v2-wireproto/*. However, these constrain us
a bit because they assume there will only be 1 API: version 2 of
the HTTP wire protocol. I think there is room to grow multiple
APIs. For example, there may someday be a proper JSON API to query
or even manipulate the repository. And I don't think we should have
to create a new top-level URL space for each API nor should we
attempt to shoehorn each future API into the same shared URL space:
that would just be too chaotic.
This commits reserves the /api/* URL space for all our future API
needs. Essentially, all requests to /api/* get routed to a new WSGI
handler. By default, it 404's the entire URL space unless the
"api server" feature is enabled. When enabled, requests to "/api"
list available APIs. URLs of the form /api/<name>/* are reserved for
a particular named API. Behavior within each API is left up to that
API. So, we can grow new APIs easily without worrying about URL
space conflicts.
APIs can be registered by adding entries to a global dict. This allows
extensions to provide their own APIs should they choose to do so.
This is probably a premature feature. But IMO the code is easier
to read if we're not dealing with API-specific behavior like config
option querying inline.
To prove it works, we implement a very basic API for version 2
of the HTTP wire protocol. It does nothing of value except
facilitate testing of the /api/* URL space.
We currently emit plain text responses for all /api/* endpoints.
There's definitely room to look at Accept and other request headers
to vary the response format. But we have to start somewhere.
Differential Revision: https://phab.mercurial-scm.org/D2834
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 13 Mar 2018 10:34:36 -0700] rev 37045
url: support suppressing Accept header
Sending this header automatically could interfere with future
testing and client behavior. Let's add a knob to disable the
behavior.
We don't have a control for User-Agent because urllib will send
it if we don't set something. I don't feel like hacking into the
bowels of urllib to figure out how to suppress that. UA shouldn't
be used for anything meaningful. So it shouldn't pose any problems
beyond non-determinism (since the header has the Mercurial version in
it).
Differential Revision: https://phab.mercurial-scm.org/D2843
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 13 Mar 2018 11:20:07 -0700] rev 37044
util: don't log low-level I/O calls for HTTP peer
`hg debugwireproto` is useful for testing HTTP interactions. Possibly
more useful than `get-with-headers.py`. But one thing that makes it
annoying for mid-level tests is that it logs all API calls, such
as readline(). This makes output - especially headers - overly
verbose.
This commit teaches our file and socket observers to not log API
calls on functions dealing with data.
We change the behavior of `hg debugwireproto` to enable this mode
by default. --debug can be added to restore the previous behavior.
As the test changes demonstrate, this makes tests much easier to
read. As a bonus, it also removes some required (glob) over lengths
in system call results.
One thing that's lacking is knowing which side sent data. But we can
fix this in a follow-up once it becomes a problem.
Differential Revision: https://phab.mercurial-scm.org/D2842
Martin von Zweigbergk <martinvonz@google.com> [Wed, 21 Mar 2018 09:56:41 -0700] rev 37043
rebase: rename conclude[memory]node() to commit[memory]node()
The functions do little more than commit at this point.
Differential Revision: https://phab.mercurial-scm.org/D2924
Martin von Zweigbergk <martinvonz@google.com> [Tue, 20 Mar 2018 22:11:10 -0700] rev 37042
rebase: pass in "user" instead of "ctx" to conclude[memory]node()
This was the only remaining part of the context object that was
needed.
Differential Revision: https://phab.mercurial-scm.org/D2923
Martin von Zweigbergk <martinvonz@google.com> [Tue, 20 Mar 2018 21:59:04 -0700] rev 37041
rebase: look up default date outside of conclude[memory]node()
Differential Revision: https://phab.mercurial-scm.org/D2922
Martin von Zweigbergk <martinvonz@google.com> [Tue, 20 Mar 2018 21:58:32 -0700] rev 37040
rebase: move config override out of conclude[memory]node()
Differential Revision: https://phab.mercurial-scm.org/D2921
Martin von Zweigbergk <martinvonz@google.com> [Tue, 20 Mar 2018 21:53:36 -0700] rev 37039
rebase: pass in entire "overrides" dict to conclude[memory]node()
As with previous patches, this was done the same way in both
functions, so let's make the caller do it instead.
Differential Revision: https://phab.mercurial-scm.org/D2920
Martin von Zweigbergk <martinvonz@google.com> [Tue, 20 Mar 2018 21:49:09 -0700] rev 37038
rebase: pass in "keepbranch" to conclude[memory]node()
Both functions calculated the same "keepbranch" value from the
"keepbranches" we passed in, so let's make the caller do it instead.
Differential Revision: https://phab.mercurial-scm.org/D2919
Martin von Zweigbergk <martinvonz@google.com> [Tue, 20 Mar 2018 21:45:48 -0700] rev 37037
rebase: inline _makextrafn() now that we have only one caller
Also avoid even creating a function since we just end up calling it
right away.
Differential Revision: https://phab.mercurial-scm.org/D2918
Martin von Zweigbergk <martinvonz@google.com> [Tue, 20 Mar 2018 21:41:53 -0700] rev 37036
rebase: pass in "extra" itself into conclude[memory]node()
We were passing in a function instead for no clear reason (probably
historical, but I haven't bothered looking).
Differential Revision: https://phab.mercurial-scm.org/D2917
Martin von Zweigbergk <martinvonz@google.com> [Wed, 21 Mar 2018 11:05:32 -0700] rev 37035
rebase: look up commit message to reuse outside of conclude[memory]node()
This was done the same way in both functions, so let's let the single
caller do it.
Differential Revision: https://phab.mercurial-scm.org/D2916
Martin von Zweigbergk <martinvonz@google.com> [Wed, 21 Mar 2018 11:04:13 -0700] rev 37034
rebase: pass in ctx, not rev, to conclude[memory]node()
They both need it and there's no locking that might make the results
different, so let's do it in one place. This also lets us move out
more common code in the following patches.
Differential Revision: https://phab.mercurial-scm.org/D2915
Martin von Zweigbergk <martinvonz@google.com> [Wed, 21 Mar 2018 11:03:31 -0700] rev 37033
rebase: extract common _concludenode()
_rebasenode() and _finishrebase() had a lot of code in common. This
commit extracts some of that. This means we will also have a single
caller of concludenode() and concludememorynode(), which gives us a
place to put code that is common between those two functions (which is
quite a bit).
Differential Revision: https://phab.mercurial-scm.org/D2914
Martin von Zweigbergk <martinvonz@google.com> [Wed, 21 Mar 2018 11:01:19 -0700] rev 37032
rebase: store rebase state after each commit
Before this patch, we stored the rebase state early in the processing
of a node, before we updated the rebase state to indicate that the
node was processed. This meant that we could redo the working copy
merge and run into conflicts. However, this only happened in the
--collapse case if the rebase was interrupted while editing the final
commit message; in the case earlier interruptions, we would instead
detect the in-process revision by finding two dirstate parents.
This patch moves the writing of the rebase state to after we have
completed the revision completely, and, importantly, after we have
updated the rebase state to mark it done. This means we'll realize
that all nodes have been rebased in the case mentioned above of
editing the final commit message of a --collapse. See change to test
case.
I also moved the writing outside of the large if/elif block in
_rebasenode(). This shouldn't matter much, but seems cleaner. One
observable effect is if rebase was interrupted just after ignoring an
obsolete node ("not rebasing ####, already in destination"), we used
to come up with the same decision after --continue too, but after this
patch we'll instead say "already rebased ###". This seems more
consistent, since that's what we would do with obsolete nodes that had
been marked done earlier in the process (not only just before the
interruption).
Differential Revision: https://phab.mercurial-scm.org/D2913
Martin von Zweigbergk <martinvonz@google.com> [Wed, 21 Mar 2018 10:46:00 -0700] rev 37031
rebase: register status file generator only once when using single transaction
rebase.storestatus() behaved differently depending on whether a
transaction is passed to it. If a transaction is passed, it registers
a "file generator" that runs when the transaction commits. If no
transaction was passed, it writes the rebase state immediately. This
imprecise timing of the writing makes it hard to reason about, so
let's make it more explicit which behavior we're getting by checking
if we have a transaction before calling it. For the single-transaction
case, move the call to storestatus(tr) early and do it only once since
it's only going to write the file (at most) once anyway.
Differential Revision: https://phab.mercurial-scm.org/D2912
Martin von Zweigbergk <martinvonz@google.com> [Sun, 18 Mar 2018 22:26:23 -0700] rev 37030
tests: add some more tests to test-rebase-collapse.t
Some of these are currently broken.
Differential Revision: https://phab.mercurial-scm.org/D2911
Martin von Zweigbergk <martinvonz@google.com> [Sun, 18 Mar 2018 15:25:23 -0700] rev 37029
tests: use drawdag in test-rebase-collapse.t
This makes the tests shorter (410 lines -> 240 lines), faster (17s ->
12s), and easier to read (the setup for each test case is just before
the test, and we can refer to commits by name).
Differential Revision: https://phab.mercurial-scm.org/D2910
Martin von Zweigbergk <martinvonz@google.com> [Tue, 20 Mar 2018 17:25:16 -0700] rev 37028
rebase: remove unused default argument values from conclude[memory]node()
It's only "commitmsg" that we don't always pass.
Differential Revision: https://phab.mercurial-scm.org/D2909
Martin von Zweigbergk <martinvonz@google.com> [Tue, 20 Mar 2018 17:30:08 -0700] rev 37027
rebase: get "inmemory" state directly from rebase runtime
As far as I can tell, rbsrt.inmemory is equivalent to
rbsrt.wctx.isinmemory(), so let's use the shorter form.
Differential Revision: https://phab.mercurial-scm.org/D2908
Pulkit Goyal <7895pulkit@gmail.com> [Mon, 25 Dec 2017 15:56:07 +0530] rev 37026
commands: use keyword arguments in update function
This will help us in having a dictionary with the values of all the arguments
and we can add more flags without adding an argument to the function.
Differential Revision: https://phab.mercurial-scm.org/D2896
Martin von Zweigbergk <martinvonz@google.com> [Tue, 20 Mar 2018 13:53:02 -0700] rev 37025
rebase: pass "inmemory" directly to _definedestmap()
We no longer reassign rbsrt.inmemory in _definedestmap(), so we don't
need to pass the whole rebase runtime instance anymore, thus
making it clear that it won't be updated.
Differential Revision: https://phab.mercurial-scm.org/D2905
Anton Shestakov <av6@dwimlabs.net> [Tue, 20 Mar 2018 14:19:28 +0800] rev 37024
hgweb: explain instabilities of unstable changesets (the rest of the themes)
Matt Harbison <matt_harbison@yahoo.com> [Tue, 20 Mar 2018 21:10:29 -0400] rev 37023
test-merge-tools: stabilize for Windows
See
fe5c4b795999.
Matt Harbison <matt_harbison@yahoo.com> [Tue, 20 Mar 2018 20:33:37 -0400] rev 37022
test-http-protocol: drop an extraneous glob for Windows
Martin von Zweigbergk <martinvonz@google.com> [Mon, 19 Mar 2018 08:07:18 -0700] rev 37021
merge with stable
Yuya Nishihara <yuya@tcha.org> [Sun, 03 Apr 2016 14:16:47 +0900] rev 37020
hgweb: evaluate the "default" value as template
Strictly speaking, everything in the map file is a template. So let's not
take out an unparsed template string.
Yuya Nishihara <yuya@tcha.org> [Fri, 16 Mar 2018 21:39:32 +0900] rev 37019
templater: use named function to expand template against mapping dict (API)
And replace __call__(t, **mapping) in favor of generate(t, mapping). I prefer
a named function here since the templater isn't a simple function-like object.
.. api::
The templater is no longer callable. Use ``templater.generate(t, mapping)``
instead of ``templater(t, **pycompat.strkwargs(mapping))``.