Gregory Szorc <gregory.szorc@gmail.com> [Tue, 10 Apr 2018 18:13:28 -0700] rev 37556
httppeer: only advertise partial-pull if capabilities are known
We don't need to be advertising client protocol parameters as part
of the capabilities request during the handshake because nothing in
version 1 of the wire protocol will use this data. i.e. the
advertisement is wasteful.
Differential Revision: https://phab.mercurial-scm.org/D3241
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 10 Apr 2018 16:53:44 -0700] rev 37555
httppeer: always add x-hg* headers to Vary header
Before, we manually updated the Vary header value for each
header contributing to it.
All X-Hg* headers are reserved for the Mercurial protocol and
could have caching implications. So it makes sense to always add
these headers to Vary.
A test revealed that X-HgArgs-Post wasn't being added to Vary.
This is only sent on POST requests. POST requests generally
aren't cacheable. However, it is possible if the server sends
the appropriate headers. Mercurial shouldn't be sending those
headers. But let's not take any chances.
Differential Revision: https://phab.mercurial-scm.org/D3240
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 10 Apr 2018 13:41:21 -0700] rev 37554
httppeer: don't accept very old media types (BC)
Versions of Mercurial older than 1.0 emitted the text/plain
and application/hg-changegroup media types in response to wire
protocol commands.
Way back in 8760d0c83b9b in 2005, the code validating these media
types was added, presumably for backwards compatibility. 0b245edec124
a short time before that commit changed things from text/plain and
application/hg-changegroup to application/mercurial-0.1 and
application/hg-0.1. 8760d0c83b9b seemed to indicate ("for now") that
the BC compatibility was temporary. But that code has lived until
this day.
It has been more than 10 years and nobody should be running pre 1.0
servers.
Pretty much the only risk to this is if there's a server somewhere
advertising the old media types or server software is interfering
and not letting Mercurial send the proper Content-Type header. I
think the chances are rare.
The wire protocol docs were created (by me) from reading existing
code. So the deletions don't constitute a spec change as much as
reflecting the reality of how things have been for years.
.. bc::
The HTTP client no longer accepts text/plain and
application/hg-changegroup Content-Type values as a valid Mercurial
command response. These should only be encountered on pre 1.0
Mercurial servers.
Differential Revision: https://phab.mercurial-scm.org/D3239
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 10 Apr 2018 13:07:13 -0700] rev 37553
httppeer: allow opener to be passed to makepeer()
This allows us to use makepeer() in `hg debugwireproto`.
Differential Revision: https://phab.mercurial-scm.org/D3238
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 10 Apr 2018 13:11:40 -0700] rev 37552
httppeer: perform capabilities request in makepeer()
Previously, we constructed an httppeer then always ran _fetchcaps()
to issue the capabilities command.
We want to issue the capabilities command before constructing a
peer instance so we can construct an appropriate peer instance
depending on the capabilities result.
With the code for making and sending requests moved out of httppeer,
it is now possible to send command requests without an httppeer.
This commit creates a new function for making the capabilities
request and calls it as part of makepeer().
This code should be functionality equivalent to what existed before.
Differential Revision: https://phab.mercurial-scm.org/D3237
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 10 Apr 2018 12:52:29 -0700] rev 37551
httppeer: extract common response handling into own function
This allows the common redirect detection, content type
validation, and decompression wrapping to be usable outside of
httppeer instances.
Differential Revision: https://phab.mercurial-scm.org/D3236
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 10 Apr 2018 12:12:07 -0700] rev 37550
httppeer: move error handling and response wrapping into sendrequest
This is common for all HTTP requests. It should be part of
sendrequest().
Differential Revision: https://phab.mercurial-scm.org/D3235
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 10 Apr 2018 10:51:12 -0700] rev 37549
httppeer: extract code for creating a request into own function
Some of this feels awkward, such as having to pass in a function
to evaluate a capability. And this code is generally pretty difficult
to read. I didn't want to perform too much refactoring as part of
the code move since it would make review more difficult.
Differential Revision: https://phab.mercurial-scm.org/D3234
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 10 Apr 2018 10:27:49 -0700] rev 37548
httppeer: extract code for performing an HTTP request
This is generic and doesn't need to live as a method of httppeer.
Differential Revision: https://phab.mercurial-scm.org/D3233
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 10 Apr 2018 10:22:26 -0700] rev 37547
httppeer: move requestbuilder defaults into makepeer() argument
Upcoming commits will move the initial ?cmd=capabilities handshake
request out of httppeer so the handshake can be performed before a
peer instance is constructed. In order to do this, we'll need to
refactor code for making HTTP requests.
The type used to construct HTTP requests is configurable. If we'll
be making HTTP requests outside of httppeer, we should be able to
use a custom request builder. So move the definition of that type
into makepeer().
Extensions can monkeypatch the function and override the argument
value.
Differential Revision: https://phab.mercurial-scm.org/D3232
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 09 Apr 2018 19:35:39 -0700] rev 37546
wireproto: move version 2 command handlers to wireprotov2server
This is relatively straightforward.
As part of this, we introduced a local @wireprotocommand that
wraps the main one and defines a v2 only policy by default.
Because the hacky HTTPv2 peer isn't using capabilities response
yet, we had to move some code around to force import of
wireprotov2server so commands are registered. This is super
hacky. But this code will go away once the HTTPv2 peer is using
the capabilities response to derive permissions.
Differential Revision: https://phab.mercurial-scm.org/D3231
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 09 Apr 2018 19:35:04 -0700] rev 37545
wireproto: extract HTTP version 2 code to own module
wireprotoserver has generic and version 1 server code. The wireproto
module also has both version 1 and version 2 command implementations.
Upcoming work I want to do will make it difficult for this code to
live in the current locations. Plus, it kind of makes sense for the
version 2 code to live in an isolated module.
This commit copies the HTTPv2 bits from wireprotoserver into a new
module. We do it as a file copy to preserve history. A future
commit will be copying wire protocol commands into this module
as well. But there is little history of that code, so it makes
sense to take history for wireprotoserver.
Differential Revision: https://phab.mercurial-scm.org/D3230
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 09 Apr 2018 16:54:20 -0700] rev 37544
wireproto: client reactor support for receiving frames
We can now feed received frames into the client reactor and it will
validate their sanity, dispatch them appropriately.
The hacky HTTP peer has been updated to use the new code. No
existing tests changed, somewhat proving the code works as
expected.
Rudimentary unit tests for the new functionality have been
implemented.
Differential Revision: https://phab.mercurial-scm.org/D3224
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 09 Apr 2018 15:32:01 -0700] rev 37543
wireproto: introduce a reactor for client-side state
We have a nice state machine of sorts for reacting to server-side
events. Now it is time to implement the client equivalent.
We introduce a "clientreactor." It allows callers to request
that commands be issued. It has multiple modes of operation to
reflect what the underlying transport supports. e.g. for SSH,
we can perform wire sends immediately but for HTTP we need to
buffer sends until all command requests are received. In addition,
SSH allows sending multiple requests as long as the connection is
open. But HTTP/1.1 only allows sending request data once.
For SSH, we'll have one reactor per connection. For HTTP, we'll
have one reactor per HTTP request. But because code that calls
wire protocol commands should not be aware of how the underlying
transport works, this will all be abstracted away by the peer
interface.
Our crude HTTP peer has been updated to use the reactor instead
of formulating frames directly. No behavior should have changed
here and tests seem to confirm that.
Basic unit tests for the reactor behavior have been added.
Differential Revision: https://phab.mercurial-scm.org/D3223
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 09 Apr 2018 14:17:57 -0700] rev 37542
tests: extract wire protocol framing tests to own file
I was lazy when I put these in test-wireproto-serverreactor.py. Let's
do it properly.
Differential Revision: https://phab.mercurial-scm.org/D3222
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 09 Apr 2018 11:33:38 -0700] rev 37541
wireproto: disallow commands handlers for multiple transport versions
I think it will be more trouble than it is worth to code version 1
and version 2 command handlers to the same interface. It will feel
awkward to shoehorn functionality into e.g. the version 1 protocol
handler interface. This would likely constrain the ability for version
2 to evolve.
Previous commits introduced a clean separation between command handlers
for version 1 and version 2 transports. This commit reinforces that
separation by dropping support for having a single command handler
service both version 1 and version 2 transports.
Differential Revision: https://phab.mercurial-scm.org/D3208
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 09 Apr 2018 11:57:12 -0700] rev 37540
wireproto: make @wireprotocommand version 1 only by default
For backwards compatibility reasons. We want extension provided
commands to opt in to version 2 rather than get inherited
automatically. This will facilitate a clean break between the
protocols.
As part of this, we duplicate some commands used in tests so
there are different command handlers per transport.
Differential Revision: https://phab.mercurial-scm.org/D3207
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 09 Apr 2018 11:54:31 -0700] rev 37539
wireproto: only expose "getbundle" and "unbundle" to v1 transports
These are the most complicated wire protocol commands. I don't want
to deal with porting them just yet. Let's disable both of them on
version 2 transports so we drive the final wedge between command
handlers and start to evolve version 2 command handlers more.
Differential Revision: https://phab.mercurial-scm.org/D3206
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 06 Apr 2018 17:48:07 -0700] rev 37538
wireproto: port lookup to wire protocol v2
This is pretty straightforward. We don't yet handle errors because we
don't have an error handling mechanism in place yet.
I'm also tempted to fold this into `known`. We'll come back to this
later.
Differential Revision: https://phab.mercurial-scm.org/D3205
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 06 Apr 2018 17:39:40 -0700] rev 37537
wireproto: port pushkey command to wire protocol version 2
It doesn't do output redirection yet. And I'd love to generally overhaul
the pushkey protocol for wire protocol version 2. But this will be a bit
of effort. Let's do it as a follow-up.
Differential Revision: https://phab.mercurial-scm.org/D3204
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 06 Apr 2018 17:21:16 -0700] rev 37536
wireproto: only expose "clonebundles" to version 1 transports
This may make a comeback in wire protocol version 2. The feature
definitely needs to be carried forward. But at this juncture, I'm
flirting with the idea of implementing this via a "redirect"
mechanism at the command response level itself rather than something
that requires one-off client support for querying and handling.
i.e. I want to make it so servers can say "fetch this first and
then come back" and clients handle that automatically. This would
not only support clone bundles, but would also support piece-meal
"pull bundles." Whatever happens, we can deal with it down the
road.
Differential Revision: https://phab.mercurial-scm.org/D3203
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 06 Apr 2018 17:14:06 -0700] rev 37535
wireproto: define and expose types of wire command arguments
Exposing the set of argument names is cool. But with wire protocol
version 2, we're using CBOR to transport arguments and this allows us
to have typing for arguments.
Typed arguments are much nicer because they will cut down on transfer
overhead and processing overhead for decoding values.
This commit teaches @wireprotocommand to accept a dictionary for
arguments. The arguments registered for version 2 transports are
canonically stored as dictionaries rather than a space-delimited string.
It is an error to defined arguments with a dictionary for commands using
version 1 transports. This reinforces my intent to fully decouple command
handlers for version 2 transports.
Differential Revision: https://phab.mercurial-scm.org/D3202
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 06 Apr 2018 16:49:57 -0700] rev 37534
wireproto: only expose "stream_out" to version 1 transports
I have plans to implement stream clone using a better mechanism than
this existing command. Let's not carry it forward to wire protocol
version 2.
Differential Revision: https://phab.mercurial-scm.org/D3201
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 09 Apr 2018 11:52:31 -0700] rev 37533
wireproto: implement capabilities for wire protocol v2
The capabilities mechanism for wire protocol version 2 represents a
clean break from version 1.
Instead of effectively exchanging a set of capabilities, we're
exchanging a rich data structure.
This data structure currently contains information about
every available command, including its accepted arguments. It also
contains information about supported compression formats.
Exposing information about supported commands will allow clients
to automatically generate bindings to the server. Clients will be
able to do things like detect when they are attempting to run a
command that isn't known to the server. Exposing the required
permissions to run a command can be used by clients to determine if
they have privileges to call a command before actually calling it.
We could potentially even have clients send credentials
preemptively without waiting for the server to deny the command
request. Lots of potential here.
The data returned by this command will likely evolve heavily. So we
shouldn't bikeshed the implementation just yet.
Differential Revision: https://phab.mercurial-scm.org/D3200
Martin von Zweigbergk <martinvonz@google.com> [Sun, 08 Apr 2018 09:45:45 -0700] rev 37532
context: add deprecation warnings for deprecated types of changeids
It's close to code freeze, and dropping support for repo['123'] and
repo ['my-bookmark'] and repo['deadbeef'] is pretty dispruptive, so
this just adds deprecation warnings so extensions can easily find the
places they need to fix.
Differential Revision: https://phab.mercurial-scm.org/D3197
Martin von Zweigbergk <martinvonz@google.com> [Sun, 08 Apr 2018 09:28:08 -0700] rev 37531
revsymbol: stop delegating to repo.__getitem__ for unhandled symbols (API)
The only remaining cases where we were delegating unhandled symbols to
repo.__getitem__ should now be when the symbol could not be found. In
that case we just delegated to repo.__getitem__ for the error
message. Let's just copy the error message instead.
If there were any cases where we got e.g. a binary nodeid or an
integer revnum into revsymbol() (e.g. via repo.lookup()), we'd now
start raising an exception instead. That is why this is marked (API).
This affects one test case, but the new behavior seems better to me. I
can't tell if the old behavior was desired or if the test was just
there to document how it happened to work.
Differential Revision: https://phab.mercurial-scm.org/D3196
Martin von Zweigbergk <martinvonz@google.com> [Sun, 08 Apr 2018 09:28:49 -0700] rev 37530
context: handle partial nodeids in revsymbol()
Similar reasoning as previous patches.
Differential Revision: https://phab.mercurial-scm.org/D3195
Martin von Zweigbergk <martinvonz@google.com> [Fri, 06 Apr 2018 23:46:17 -0700] rev 37529
context: handle namespaces in revsymbol()
Similar reasoning as previous patches.
Differential Revision: https://phab.mercurial-scm.org/D3194
Martin von Zweigbergk <martinvonz@google.com> [Fri, 06 Apr 2018 23:43:52 -0700] rev 37528
scmutil: handle full hex nodeids in revsymbol()
This is a bit unfortunate, but it enables moving other pieces out of
changectx's constructor without affecting the order in which we
look up things (e.g. hex nodeid before bookmark).
We convert nodeid to revnum before calling repo.__getitem__, even
though that will result in converting back to nodeid later. This is so
we can handle the LookupError and attempt to interpret the string as
something else (e.g. a bookmark).
We also need to start handling WdirUnsupported now, since the full hex
nodeid "ffff..." represents the working directory. The exception is
raised by the revlog layer.
Differential Revision: https://phab.mercurial-scm.org/D3193
Martin von Zweigbergk <martinvonz@google.com> [Fri, 06 Apr 2018 23:39:40 -0700] rev 37527
context: handle stringified ints in revsymbol()
This patch copies the handling of stringified ints from changectx's
constructor. It then calls repo.__getitem__ with the int. Since that
method only interprets integers as revnums the first thing it does,
this will not be redoing any of the work already done. We leave the
old code in place so we can later deprecate it instead of breaking
extensions.
Differential Revision: https://phab.mercurial-scm.org/D3146
Pulkit Goyal <7895pulkit@gmail.com> [Tue, 10 Apr 2018 19:32:08 +0530] rev 37526
py3: make sure we write bytes to file
# skip-blame because just b'' prefix
Differential Revision: https://phab.mercurial-scm.org/D3220
Pulkit Goyal <7895pulkit@gmail.com> [Tue, 10 Apr 2018 19:21:16 +0530] rev 37525
py3: workaround comparing NoneType and integers
Comparing None with integers was fine in Python 2 but returns error in Python 3
which is nice. This patch replaces None with -1 where sorting is done and some
related logic.
Differential Revision: https://phab.mercurial-scm.org/D3219
Pulkit Goyal <7895pulkit@gmail.com> [Tue, 10 Apr 2018 18:57:56 +0530] rev 37524
py3: use sys.stdout instead of print in test-mq-qpush-fail.t
Differential Revision: https://phab.mercurial-scm.org/D3218
Pulkit Goyal <7895pulkit@gmail.com> [Tue, 10 Apr 2018 18:57:32 +0530] rev 37523
py3: use '//' for integer division in tests/test-mq-qimport.t
Differential Revision: https://phab.mercurial-scm.org/D3217
Pulkit Goyal <7895pulkit@gmail.com> [Wed, 11 Apr 2018 14:35:28 +0530] rev 37522
py3: use pycompat.byterepr() instead of repr()
Differential Revision: https://phab.mercurial-scm.org/D3216
Pulkit Goyal <7895pulkit@gmail.com> [Tue, 10 Apr 2018 18:56:19 +0530] rev 37521
py3: use bytes instead of str in isinstance()
We deal with bytes internally and things should be bytes
Differential Revision: https://phab.mercurial-scm.org/D3215
Pulkit Goyal <7895pulkit@gmail.com> [Tue, 10 Apr 2018 18:55:35 +0530] rev 37520
py3: use .startswith() instead of bytes[0]
Differential Revision: https://phab.mercurial-scm.org/D3214
Pulkit Goyal <7895pulkit@gmail.com> [Tue, 10 Apr 2018 20:14:12 +0530] rev 37519
py3: add b'' prefixes in tests/test-ui-config.py
# skip-blame because just b'' prefixes
Differential Revision: https://phab.mercurial-scm.org/D3221
Matt Harbison <matt_harbison@yahoo.com> [Sat, 07 Apr 2018 22:22:20 -0400] rev 37518
lfs: infer the blob store URL from paths.default
If `lfs.url` is specified, it takes precedence. However, now that we support
serving blobs via hgweb, we shouldn't *require* this setting. Less
configuration is better (things will work out of the box once this is sorted
out), and git has similar functionality.
This is not a complete solution- it isn't able to infer the blob store from an
explicitly supplied path, and it should consider `paths.default-push` for push.
The pull solution for that is a bit hacky, and this alone is an improvement for
the vast majority of cases.
Even though there are only a handful of references to the saved remote store,
the location of them makes things complicated.
1) downloading files on demand in the revlog flag processor
2) copying to readonlyvfs with bundlerepo
3) downloading in the file prefetch hook
4) the canupload()/skipdownload() checks
5) uploading blobs
Since revlog doesn't have a repo or ui reference, we can't avoid creating a
remote store when the extension is loaded. While the long term goal is to make
sure the prefetch hook is invoked early for every command for efficiency, this
handling in the flag processor is needed as a last ditch fetch.
In order to support the clone command, the remote store needs to be created
later than when the extension loads, since `paths.default` isn't set until just
before the files are checked out. Therefore, this patch changes the prefetch
hook to ignore the saved reference, and build a new one.
The canupload()/skipdownload() checks simply check if the stored instance is a
`_nullremote`. Since this can only be set via `lfs.url` (which is reflected in
the saved reference), checking only the instance created when the extension
loaded is fine.
The blob uploading function is called from several places:
1) a prepush hook
2) when writing a new bundle
3) from infinitepush
The prepush hook gets an exchange.pushop, so it has a path to where the push is
going. The bundle writer and infinitepush don't. Further, bundle creation for
things like strip and amend are causing blobs to be uploaded. This seems wrong,
but I don't want to side track this sorting that out, so punt on trying to
handle explicit push paths or `paths.default-push`.
I also think that sending blobs to a remote store when pushing to a local repo
is wrong. This functionality predates the usercache, so perhaps that's the
reason for it. I've got some patches floating around to stop sending blobs
remotely in this case, and instead write directly to the other repo's blob
store. But the tests for corruption handling weren't happy with this change,
and I don't have time to rewrite them. So exclude filesystem based paths from
this for now.
I don't think there's much of a chance to implement `paths.remote:lfsurl` style
configs, given how early these are resolved vs how late the remote store is
created. But git has it, so I threw a TODO in there, in case anyone has ideas.
I have no idea why this is now doing http auth twice when it wasn't before. I
don't think the original blobstore's url is ever being used in these cases.
Matt Harbison <matt_harbison@yahoo.com> [Sat, 07 Apr 2018 22:40:11 -0400] rev 37517
lfs: add the ability to disable the usercache
While the usercache is important for real world uses, I've been tripped up more
than a couple of times by it in tests- thinking a file was being downloaded, but
it was simply linked from the local cache. The syntax for setting it is the
same as for setting a null remote endpoint, and like that endpoint, is left
undocumented.
This may or may not be a useful feature in the real world (I'd expect any sane
filesystem to support hardlinks at this point).
Matt Harbison <matt_harbison@yahoo.com> [Tue, 10 Apr 2018 22:57:55 -0400] rev 37516
tests: stabilize test-pull-bundle.t for Windows
See 594dd384803c.
Yuya Nishihara <yuya@tcha.org> [Sun, 01 Apr 2018 22:41:49 +0900] rev 37515
hgweb: wrap {archives} with mappinglist
No bare list of mappings should be put in a template mapping.
Yuya Nishihara <yuya@tcha.org> [Sun, 01 Apr 2018 22:40:15 +0900] rev 37514
hgweb: forward archivelist() of hgweb to webutil
self.configlist() is ui.configlist(untrusted=True), and url=None in templater
effectively means 'url' is undefined.
Yuya Nishihara <yuya@tcha.org> [Sun, 01 Apr 2018 22:37:03 +0900] rev 37513
hgweb: move archivelist() of hgwebdir to webutil
Yuya Nishihara <yuya@tcha.org> [Sun, 01 Apr 2018 22:33:55 +0900] rev 37512
hgweb: drop archivespecs from requestcontext
It's a constant.
Yuya Nishihara <yuya@tcha.org> [Sun, 01 Apr 2018 22:32:34 +0900] rev 37511
hgweb: move archivespecs to webutil
Move one step further from e38e7ea21987. Since 'archivespecs' is a constant
shared with hgweb and hgwebdir, webutil is a better place to define it.
Yuya Nishihara <yuya@tcha.org> [Sun, 01 Apr 2018 22:20:44 +0900] rev 37510
hgweb: wrap {labels} by hybridlist()
This one is a simple list of strings, which can support map operation.
Yuya Nishihara <yuya@tcha.org> [Sun, 01 Apr 2018 22:14:36 +0900] rev 37509
hgweb: wrap {pathdef} with mappinglist
No bare list of mappings should be put in a template mapping.