Gregory Szorc <gregory.szorc@gmail.com> [Tue, 10 Apr 2018 10:27:49 -0700] rev 37548
httppeer: extract code for performing an HTTP request
This is generic and doesn't need to live as a method of httppeer.
Differential Revision: https://phab.mercurial-scm.org/D3233
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 10 Apr 2018 10:22:26 -0700] rev 37547
httppeer: move requestbuilder defaults into makepeer() argument
Upcoming commits will move the initial ?cmd=capabilities handshake
request out of httppeer so the handshake can be performed before a
peer instance is constructed. In order to do this, we'll need to
refactor code for making HTTP requests.
The type used to construct HTTP requests is configurable. If we'll
be making HTTP requests outside of httppeer, we should be able to
use a custom request builder. So move the definition of that type
into makepeer().
Extensions can monkeypatch the function and override the argument
value.
Differential Revision: https://phab.mercurial-scm.org/D3232
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 09 Apr 2018 19:35:39 -0700] rev 37546
wireproto: move version 2 command handlers to wireprotov2server
This is relatively straightforward.
As part of this, we introduced a local @wireprotocommand that
wraps the main one and defines a v2 only policy by default.
Because the hacky HTTPv2 peer isn't using capabilities response
yet, we had to move some code around to force import of
wireprotov2server so commands are registered. This is super
hacky. But this code will go away once the HTTPv2 peer is using
the capabilities response to derive permissions.
Differential Revision: https://phab.mercurial-scm.org/D3231
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 09 Apr 2018 19:35:04 -0700] rev 37545
wireproto: extract HTTP version 2 code to own module
wireprotoserver has generic and version 1 server code. The wireproto
module also has both version 1 and version 2 command implementations.
Upcoming work I want to do will make it difficult for this code to
live in the current locations. Plus, it kind of makes sense for the
version 2 code to live in an isolated module.
This commit copies the HTTPv2 bits from wireprotoserver into a new
module. We do it as a file copy to preserve history. A future
commit will be copying wire protocol commands into this module
as well. But there is little history of that code, so it makes
sense to take history for wireprotoserver.
Differential Revision: https://phab.mercurial-scm.org/D3230
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 09 Apr 2018 16:54:20 -0700] rev 37544
wireproto: client reactor support for receiving frames
We can now feed received frames into the client reactor and it will
validate their sanity, dispatch them appropriately.
The hacky HTTP peer has been updated to use the new code. No
existing tests changed, somewhat proving the code works as
expected.
Rudimentary unit tests for the new functionality have been
implemented.
Differential Revision: https://phab.mercurial-scm.org/D3224
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 09 Apr 2018 15:32:01 -0700] rev 37543
wireproto: introduce a reactor for client-side state
We have a nice state machine of sorts for reacting to server-side
events. Now it is time to implement the client equivalent.
We introduce a "clientreactor." It allows callers to request
that commands be issued. It has multiple modes of operation to
reflect what the underlying transport supports. e.g. for SSH,
we can perform wire sends immediately but for HTTP we need to
buffer sends until all command requests are received. In addition,
SSH allows sending multiple requests as long as the connection is
open. But HTTP/1.1 only allows sending request data once.
For SSH, we'll have one reactor per connection. For HTTP, we'll
have one reactor per HTTP request. But because code that calls
wire protocol commands should not be aware of how the underlying
transport works, this will all be abstracted away by the peer
interface.
Our crude HTTP peer has been updated to use the reactor instead
of formulating frames directly. No behavior should have changed
here and tests seem to confirm that.
Basic unit tests for the reactor behavior have been added.
Differential Revision: https://phab.mercurial-scm.org/D3223
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 09 Apr 2018 14:17:57 -0700] rev 37542
tests: extract wire protocol framing tests to own file
I was lazy when I put these in test-wireproto-serverreactor.py. Let's
do it properly.
Differential Revision: https://phab.mercurial-scm.org/D3222
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 09 Apr 2018 11:33:38 -0700] rev 37541
wireproto: disallow commands handlers for multiple transport versions
I think it will be more trouble than it is worth to code version 1
and version 2 command handlers to the same interface. It will feel
awkward to shoehorn functionality into e.g. the version 1 protocol
handler interface. This would likely constrain the ability for version
2 to evolve.
Previous commits introduced a clean separation between command handlers
for version 1 and version 2 transports. This commit reinforces that
separation by dropping support for having a single command handler
service both version 1 and version 2 transports.
Differential Revision: https://phab.mercurial-scm.org/D3208
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 09 Apr 2018 11:57:12 -0700] rev 37540
wireproto: make @wireprotocommand version 1 only by default
For backwards compatibility reasons. We want extension provided
commands to opt in to version 2 rather than get inherited
automatically. This will facilitate a clean break between the
protocols.
As part of this, we duplicate some commands used in tests so
there are different command handlers per transport.
Differential Revision: https://phab.mercurial-scm.org/D3207
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 09 Apr 2018 11:54:31 -0700] rev 37539
wireproto: only expose "getbundle" and "unbundle" to v1 transports
These are the most complicated wire protocol commands. I don't want
to deal with porting them just yet. Let's disable both of them on
version 2 transports so we drive the final wedge between command
handlers and start to evolve version 2 command handlers more.
Differential Revision: https://phab.mercurial-scm.org/D3206
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 06 Apr 2018 17:48:07 -0700] rev 37538
wireproto: port lookup to wire protocol v2
This is pretty straightforward. We don't yet handle errors because we
don't have an error handling mechanism in place yet.
I'm also tempted to fold this into `known`. We'll come back to this
later.
Differential Revision: https://phab.mercurial-scm.org/D3205
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 06 Apr 2018 17:39:40 -0700] rev 37537
wireproto: port pushkey command to wire protocol version 2
It doesn't do output redirection yet. And I'd love to generally overhaul
the pushkey protocol for wire protocol version 2. But this will be a bit
of effort. Let's do it as a follow-up.
Differential Revision: https://phab.mercurial-scm.org/D3204
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 06 Apr 2018 17:21:16 -0700] rev 37536
wireproto: only expose "clonebundles" to version 1 transports
This may make a comeback in wire protocol version 2. The feature
definitely needs to be carried forward. But at this juncture, I'm
flirting with the idea of implementing this via a "redirect"
mechanism at the command response level itself rather than something
that requires one-off client support for querying and handling.
i.e. I want to make it so servers can say "fetch this first and
then come back" and clients handle that automatically. This would
not only support clone bundles, but would also support piece-meal
"pull bundles." Whatever happens, we can deal with it down the
road.
Differential Revision: https://phab.mercurial-scm.org/D3203
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 06 Apr 2018 17:14:06 -0700] rev 37535
wireproto: define and expose types of wire command arguments
Exposing the set of argument names is cool. But with wire protocol
version 2, we're using CBOR to transport arguments and this allows us
to have typing for arguments.
Typed arguments are much nicer because they will cut down on transfer
overhead and processing overhead for decoding values.
This commit teaches @wireprotocommand to accept a dictionary for
arguments. The arguments registered for version 2 transports are
canonically stored as dictionaries rather than a space-delimited string.
It is an error to defined arguments with a dictionary for commands using
version 1 transports. This reinforces my intent to fully decouple command
handlers for version 2 transports.
Differential Revision: https://phab.mercurial-scm.org/D3202
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 06 Apr 2018 16:49:57 -0700] rev 37534
wireproto: only expose "stream_out" to version 1 transports
I have plans to implement stream clone using a better mechanism than
this existing command. Let's not carry it forward to wire protocol
version 2.
Differential Revision: https://phab.mercurial-scm.org/D3201
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 09 Apr 2018 11:52:31 -0700] rev 37533
wireproto: implement capabilities for wire protocol v2
The capabilities mechanism for wire protocol version 2 represents a
clean break from version 1.
Instead of effectively exchanging a set of capabilities, we're
exchanging a rich data structure.
This data structure currently contains information about
every available command, including its accepted arguments. It also
contains information about supported compression formats.
Exposing information about supported commands will allow clients
to automatically generate bindings to the server. Clients will be
able to do things like detect when they are attempting to run a
command that isn't known to the server. Exposing the required
permissions to run a command can be used by clients to determine if
they have privileges to call a command before actually calling it.
We could potentially even have clients send credentials
preemptively without waiting for the server to deny the command
request. Lots of potential here.
The data returned by this command will likely evolve heavily. So we
shouldn't bikeshed the implementation just yet.
Differential Revision: https://phab.mercurial-scm.org/D3200
Martin von Zweigbergk <martinvonz@google.com> [Sun, 08 Apr 2018 09:45:45 -0700] rev 37532
context: add deprecation warnings for deprecated types of changeids
It's close to code freeze, and dropping support for repo['123'] and
repo ['my-bookmark'] and repo['deadbeef'] is pretty dispruptive, so
this just adds deprecation warnings so extensions can easily find the
places they need to fix.
Differential Revision: https://phab.mercurial-scm.org/D3197
Martin von Zweigbergk <martinvonz@google.com> [Sun, 08 Apr 2018 09:28:08 -0700] rev 37531
revsymbol: stop delegating to repo.__getitem__ for unhandled symbols (API)
The only remaining cases where we were delegating unhandled symbols to
repo.__getitem__ should now be when the symbol could not be found. In
that case we just delegated to repo.__getitem__ for the error
message. Let's just copy the error message instead.
If there were any cases where we got e.g. a binary nodeid or an
integer revnum into revsymbol() (e.g. via repo.lookup()), we'd now
start raising an exception instead. That is why this is marked (API).
This affects one test case, but the new behavior seems better to me. I
can't tell if the old behavior was desired or if the test was just
there to document how it happened to work.
Differential Revision: https://phab.mercurial-scm.org/D3196
Martin von Zweigbergk <martinvonz@google.com> [Sun, 08 Apr 2018 09:28:49 -0700] rev 37530
context: handle partial nodeids in revsymbol()
Similar reasoning as previous patches.
Differential Revision: https://phab.mercurial-scm.org/D3195
Martin von Zweigbergk <martinvonz@google.com> [Fri, 06 Apr 2018 23:46:17 -0700] rev 37529
context: handle namespaces in revsymbol()
Similar reasoning as previous patches.
Differential Revision: https://phab.mercurial-scm.org/D3194
Martin von Zweigbergk <martinvonz@google.com> [Fri, 06 Apr 2018 23:43:52 -0700] rev 37528
scmutil: handle full hex nodeids in revsymbol()
This is a bit unfortunate, but it enables moving other pieces out of
changectx's constructor without affecting the order in which we
look up things (e.g. hex nodeid before bookmark).
We convert nodeid to revnum before calling repo.__getitem__, even
though that will result in converting back to nodeid later. This is so
we can handle the LookupError and attempt to interpret the string as
something else (e.g. a bookmark).
We also need to start handling WdirUnsupported now, since the full hex
nodeid "ffff..." represents the working directory. The exception is
raised by the revlog layer.
Differential Revision: https://phab.mercurial-scm.org/D3193
Martin von Zweigbergk <martinvonz@google.com> [Fri, 06 Apr 2018 23:39:40 -0700] rev 37527
context: handle stringified ints in revsymbol()
This patch copies the handling of stringified ints from changectx's
constructor. It then calls repo.__getitem__ with the int. Since that
method only interprets integers as revnums the first thing it does,
this will not be redoing any of the work already done. We leave the
old code in place so we can later deprecate it instead of breaking
extensions.
Differential Revision: https://phab.mercurial-scm.org/D3146
Pulkit Goyal <7895pulkit@gmail.com> [Tue, 10 Apr 2018 19:32:08 +0530] rev 37526
py3: make sure we write bytes to file
# skip-blame because just b'' prefix
Differential Revision: https://phab.mercurial-scm.org/D3220
Pulkit Goyal <7895pulkit@gmail.com> [Tue, 10 Apr 2018 19:21:16 +0530] rev 37525
py3: workaround comparing NoneType and integers
Comparing None with integers was fine in Python 2 but returns error in Python 3
which is nice. This patch replaces None with -1 where sorting is done and some
related logic.
Differential Revision: https://phab.mercurial-scm.org/D3219
Pulkit Goyal <7895pulkit@gmail.com> [Tue, 10 Apr 2018 18:57:56 +0530] rev 37524
py3: use sys.stdout instead of print in test-mq-qpush-fail.t
Differential Revision: https://phab.mercurial-scm.org/D3218
Pulkit Goyal <7895pulkit@gmail.com> [Tue, 10 Apr 2018 18:57:32 +0530] rev 37523
py3: use '//' for integer division in tests/test-mq-qimport.t
Differential Revision: https://phab.mercurial-scm.org/D3217
Pulkit Goyal <7895pulkit@gmail.com> [Wed, 11 Apr 2018 14:35:28 +0530] rev 37522
py3: use pycompat.byterepr() instead of repr()
Differential Revision: https://phab.mercurial-scm.org/D3216
Pulkit Goyal <7895pulkit@gmail.com> [Tue, 10 Apr 2018 18:56:19 +0530] rev 37521
py3: use bytes instead of str in isinstance()
We deal with bytes internally and things should be bytes
Differential Revision: https://phab.mercurial-scm.org/D3215
Pulkit Goyal <7895pulkit@gmail.com> [Tue, 10 Apr 2018 18:55:35 +0530] rev 37520
py3: use .startswith() instead of bytes[0]
Differential Revision: https://phab.mercurial-scm.org/D3214
Pulkit Goyal <7895pulkit@gmail.com> [Tue, 10 Apr 2018 20:14:12 +0530] rev 37519
py3: add b'' prefixes in tests/test-ui-config.py
# skip-blame because just b'' prefixes
Differential Revision: https://phab.mercurial-scm.org/D3221
Matt Harbison <matt_harbison@yahoo.com> [Sat, 07 Apr 2018 22:22:20 -0400] rev 37518
lfs: infer the blob store URL from paths.default
If `lfs.url` is specified, it takes precedence. However, now that we support
serving blobs via hgweb, we shouldn't *require* this setting. Less
configuration is better (things will work out of the box once this is sorted
out), and git has similar functionality.
This is not a complete solution- it isn't able to infer the blob store from an
explicitly supplied path, and it should consider `paths.default-push` for push.
The pull solution for that is a bit hacky, and this alone is an improvement for
the vast majority of cases.
Even though there are only a handful of references to the saved remote store,
the location of them makes things complicated.
1) downloading files on demand in the revlog flag processor
2) copying to readonlyvfs with bundlerepo
3) downloading in the file prefetch hook
4) the canupload()/skipdownload() checks
5) uploading blobs
Since revlog doesn't have a repo or ui reference, we can't avoid creating a
remote store when the extension is loaded. While the long term goal is to make
sure the prefetch hook is invoked early for every command for efficiency, this
handling in the flag processor is needed as a last ditch fetch.
In order to support the clone command, the remote store needs to be created
later than when the extension loads, since `paths.default` isn't set until just
before the files are checked out. Therefore, this patch changes the prefetch
hook to ignore the saved reference, and build a new one.
The canupload()/skipdownload() checks simply check if the stored instance is a
`_nullremote`. Since this can only be set via `lfs.url` (which is reflected in
the saved reference), checking only the instance created when the extension
loaded is fine.
The blob uploading function is called from several places:
1) a prepush hook
2) when writing a new bundle
3) from infinitepush
The prepush hook gets an exchange.pushop, so it has a path to where the push is
going. The bundle writer and infinitepush don't. Further, bundle creation for
things like strip and amend are causing blobs to be uploaded. This seems wrong,
but I don't want to side track this sorting that out, so punt on trying to
handle explicit push paths or `paths.default-push`.
I also think that sending blobs to a remote store when pushing to a local repo
is wrong. This functionality predates the usercache, so perhaps that's the
reason for it. I've got some patches floating around to stop sending blobs
remotely in this case, and instead write directly to the other repo's blob
store. But the tests for corruption handling weren't happy with this change,
and I don't have time to rewrite them. So exclude filesystem based paths from
this for now.
I don't think there's much of a chance to implement `paths.remote:lfsurl` style
configs, given how early these are resolved vs how late the remote store is
created. But git has it, so I threw a TODO in there, in case anyone has ideas.
I have no idea why this is now doing http auth twice when it wasn't before. I
don't think the original blobstore's url is ever being used in these cases.
Matt Harbison <matt_harbison@yahoo.com> [Sat, 07 Apr 2018 22:40:11 -0400] rev 37517
lfs: add the ability to disable the usercache
While the usercache is important for real world uses, I've been tripped up more
than a couple of times by it in tests- thinking a file was being downloaded, but
it was simply linked from the local cache. The syntax for setting it is the
same as for setting a null remote endpoint, and like that endpoint, is left
undocumented.
This may or may not be a useful feature in the real world (I'd expect any sane
filesystem to support hardlinks at this point).
Matt Harbison <matt_harbison@yahoo.com> [Tue, 10 Apr 2018 22:57:55 -0400] rev 37516
tests: stabilize test-pull-bundle.t for Windows
See 594dd384803c.
Yuya Nishihara <yuya@tcha.org> [Sun, 01 Apr 2018 22:41:49 +0900] rev 37515
hgweb: wrap {archives} with mappinglist
No bare list of mappings should be put in a template mapping.
Yuya Nishihara <yuya@tcha.org> [Sun, 01 Apr 2018 22:40:15 +0900] rev 37514
hgweb: forward archivelist() of hgweb to webutil
self.configlist() is ui.configlist(untrusted=True), and url=None in templater
effectively means 'url' is undefined.
Yuya Nishihara <yuya@tcha.org> [Sun, 01 Apr 2018 22:37:03 +0900] rev 37513
hgweb: move archivelist() of hgwebdir to webutil
Yuya Nishihara <yuya@tcha.org> [Sun, 01 Apr 2018 22:33:55 +0900] rev 37512
hgweb: drop archivespecs from requestcontext
It's a constant.
Yuya Nishihara <yuya@tcha.org> [Sun, 01 Apr 2018 22:32:34 +0900] rev 37511
hgweb: move archivespecs to webutil
Move one step further from e38e7ea21987. Since 'archivespecs' is a constant
shared with hgweb and hgwebdir, webutil is a better place to define it.
Yuya Nishihara <yuya@tcha.org> [Sun, 01 Apr 2018 22:20:44 +0900] rev 37510
hgweb: wrap {labels} by hybridlist()
This one is a simple list of strings, which can support map operation.
Yuya Nishihara <yuya@tcha.org> [Sun, 01 Apr 2018 22:14:36 +0900] rev 37509
hgweb: wrap {pathdef} with mappinglist
No bare list of mappings should be put in a template mapping.
Yuya Nishihara <yuya@tcha.org> [Sun, 01 Apr 2018 22:11:58 +0900] rev 37508
hgwebdir: wrap {entries} with mappinggenerator
No bare generator should be put in a template mapping.
Martin von Zweigbergk <martinvonz@google.com> [Fri, 06 Apr 2018 09:53:17 -0700] rev 37507
eol: look up partial nodeid as partial nodeid
Similar reasoning as the previous patch. For some reason the hook
gives us a partial nodeid, so we need to resolve that to a full
nodeid.
Differential Revision: https://phab.mercurial-scm.org/D3159
Martin von Zweigbergk <martinvonz@google.com> [Fri, 06 Apr 2018 09:43:17 -0700] rev 37506
histedit: look up partial nodeid as partial nodeid
I'm about to remove support for repo[<partial hex nodeid>]. In the
verify() method, we know that self.node is always a partial or full
binary nodeid, so the most correct way to look up the revision is by
using changelog._partialmatch(), so let's do that. (It's closer to the
current code to do scmutil.revsymbol(), but that's less correct
because it will match a bookmark or tag that happens to have the same
prefix as the node.)
Differential Revision: https://phab.mercurial-scm.org/D3158
Martin von Zweigbergk <martinvonz@google.com> [Fri, 06 Apr 2018 09:41:25 -0700] rev 37505
histedit: drop unnecessary check for "self.node is not None"
We are doing hex(self.node) just a few lines up, so it shouldn't be
None. The only way it could be none is if it was reassigned in
between. The only way that can happen is if the user had put a
"ffff..." wdirhex revision in the histedit script. This code is much
older than the "ffff..." identifier, so I'm confident it's not there
to handle that case. I'll let someone else add proper checks for
"ffff..." if they care enough.
Differential Revision: https://phab.mercurial-scm.org/D3157
Martin von Zweigbergk <martinvonz@google.com> [Sun, 08 Apr 2018 08:06:34 -0700] rev 37504
context: extract partial nodeid lookup method to scmutil
We will add another caller soon, and there's a non-obvious reason to
use the unfiltered repo that we don't want to copy across the code
base.
Differential Revision: https://phab.mercurial-scm.org/D3189
Yuya Nishihara <yuya@tcha.org> [Mon, 19 Mar 2018 20:23:27 +0900] rev 37503
templatekw: fix return type of {succsandmarkers} (BC)
A hybrid object represents a list/dict of values, but {succsandmarkers}
returns a list of template mappings.
This change means old-style list templates (e.g. "start_succsandmarkers")
are no longer supported, but that should be okay since {succsandmarkers}
is still experimental and relatively new.
Yuya Nishihara <yuya@tcha.org> [Sat, 17 Mar 2018 23:34:38 +0900] rev 37502
formatter: remove template resources from nested items before generating JSON
Yuya Nishihara <yuya@tcha.org> [Thu, 15 Mar 2018 21:09:37 +0900] rev 37501
templater: add public interface returning a set of resource keys
The next patch depends on knownresourcekeys(), and there's no reason to
keep availableresourcekeys() private.