Yuya Nishihara <yuya@tcha.org> [Sun, 01 Apr 2018 23:50:34 +0900] rev 37700
hgweb: convert _siblings to a factory function of mappinggenerator
IIUC, only reason it was a class is to make the generator restartable,
which is now served by the mappinggenerator.
Yuya Nishihara <yuya@tcha.org> [Sun, 01 Apr 2018 23:47:43 +0900] rev 37699
hgweb: extract a generator function of _siblings class
_siblings will be converted to a plain function.
Yuya Nishihara <yuya@tcha.org> [Sun, 01 Apr 2018 23:40:08 +0900] rev 37698
hgweb: wrap {changenav} and {nav} with mappinglist
Yuya Nishihara <yuya@tcha.org> [Sun, 01 Apr 2018 23:34:29 +0900] rev 37697
hgweb: make revnav.gen() simply build a list of mappings by one pass
There was actually no lazy stuff.
Yuya Nishihara <yuya@tcha.org> [Sun, 01 Apr 2018 23:12:37 +0900] rev 37696
hgweb: lift {sessionvars} to a wrapped type
Since a sessionvars object is updated in-place, we can't simply wrap it by
mappinglist.
Yuya Nishihara <yuya@tcha.org> [Sun, 01 Apr 2018 23:03:58 +0900] rev 37695
hgweb: make sessionvars class less dense
Yuya Nishihara <yuya@tcha.org> [Sun, 01 Apr 2018 23:03:02 +0900] rev 37694
hgweb: prefix private variables of sessionvars with '_'
Matt Harbison <matt_harbison@yahoo.com> [Fri, 13 Apr 2018 16:32:33 -0400] rev 37693
lfs: update the HTTP status codes in error cases
I'm not bothering with validating PUT requests (for now), because the spec
doesn't explicitly call out a Content-Type (though the example transcript does
use the sensible 'application/octet-stream').
Matt Harbison <matt_harbison@yahoo.com> [Sun, 25 Feb 2018 14:07:13 -0500] rev 37692
lfs: gracefully handle aborts on the server when corrupt blobs are detected
The aborts weren't killing the server, but this seems cleaner. I'm not sure if
it matters to handle the remaining IOError in the test like this, for
consistency.
The error code still feels wrong (especially if the client is trying to download
a corrupt blob) but I don't see anything better in the RFCs, and this is already
used elsewhere because the Batch API spec specifically mentioned this as a
"Validation Error".
Matt Harbison <matt_harbison@yahoo.com> [Fri, 13 Apr 2018 14:16:30 -0400] rev 37691
lfs: fix the inferred remote store path when using a --prefix
This wasn't appending the '.git/info/lfs' path in this case.
Matt Harbison <matt_harbison@yahoo.com> [Fri, 13 Apr 2018 12:39:54 -0400] rev 37690
lfs: log information about Internal Server Errors reported in the Batch API
Reporting a 500 and then not leaving any traces on the server seems like a
receipe for frustration. There's similar log writing in hgweb.server.do_POST().
That doesn't write directly to the wsgi.errors object, so it doesn't seem worth
trying to refactor.
It does seem like earlier stack frames are missing for some reason.
Matt Harbison <matt_harbison@yahoo.com> [Sat, 07 Apr 2018 12:48:21 -0400] rev 37689
test-lfs: add tests to force server error path coverage
The tests are somewhat fragile in that the extension that forces the errors is
counting how many times some of the functions are being called, so it depends
heavily on the content of the repo. Maybe we can do something clever like load
an extension on the client, and have it send over instructions in the HTTP
header how to fail. (I'm trying to avoid killing and restarting the server,
because Windows seems to have issues with doing that a lot.) But I'd rather fix
issues than polish tests before the freeze.
Augie Fackler <augie@google.com> [Sat, 14 Apr 2018 10:43:19 -0400] rev 37688
keepalive: add ** overlooked in 83250442dc81
Caught by Yuya in D3326.
Differential Revision: https://phab.mercurial-scm.org/D3372
Yuya Nishihara <yuya@tcha.org> [Sat, 14 Apr 2018 17:27:32 +0900] rev 37687
test-check-commit: don't run hg per commit
We aren't stress testing CPU.
$ time ./run-tests.py -l test-check-commit.t --timeout 600
(orig) 162.59s user 17.98s system 101% cpu 2:58.55 total
(new) 5.85s user 0.99s system 98% cpu 6.939 total
Anton Shestakov <av6@dwimlabs.net> [Sat, 14 Apr 2018 20:34:14 +0800] rev 37686
templates: make divergentnodes in whyunstable keyword be a hybrid list
Anton Shestakov <av6@dwimlabs.net> [Mon, 26 Mar 2018 17:00:12 +0800] rev 37685
templates: add whyunstable template keyword
Augie Fackler <augie@google.com> [Sat, 14 Apr 2018 00:41:44 -0400] rev 37684
commands: drop spurious r'' on dry_run in forget
Fixes test-add.t on Python 3.
Differential Revision: https://phab.mercurial-scm.org/D3365
Augie Fackler <augie@google.com> [Sat, 14 Apr 2018 00:56:53 -0400] rev 37683
tests: port inline extension in test-http-bundle1.t to py3
# skip-blame just b prefixes
Differential Revision: https://phab.mercurial-scm.org/D3366
Augie Fackler <augie@google.com> [Sat, 14 Apr 2018 01:35:44 -0400] rev 37682
tests: add all missing b prefixes in reactor tests
Both of these tests now pass on Python 3.
# skip-blame just b prefixes. So many b prefixes.
Differential Revision: https://phab.mercurial-scm.org/D3369
Augie Fackler <augie@google.com> [Sat, 14 Apr 2018 01:34:44 -0400] rev 37681
stringutil: ast.literal_eval needs a unicode on py3
Fortunately for us, this is really only used internally, so we can be
lazy about the encoding here.
test-wireproto-framing.py now passes on Python 3.
Differential Revision: https://phab.mercurial-scm.org/D3368
Martin von Zweigbergk <martinvonz@google.com> [Fri, 13 Apr 2018 22:55:01 -0700] rev 37680
scmutil: introduce shortesthexnodeidprefix()
We have scmutil.resolvehexnodeidprefix() for resolving a prefix to a
full nodeid, so it makes sense to have the inverse method next to
it. For now it just delegates to changelog.shortest(), but it will
soon also make sure it's called on the unfiltered repo, to match
resolvehexnodeidprefix().
Note that the change in show.py also makes it so the conversion from
revnum to nodeid is done on the filtered repo, but that should be
inconsequential since the revs are all from the filtered repo anyway.
Differential Revision: https://phab.mercurial-scm.org/D3370
Martin von Zweigbergk <martinvonz@google.com> [Fri, 13 Apr 2018 11:00:30 -0700] rev 37679
scmutil: use resolvehexnodeidprefix() from revsymbol()
I should have copied this from changectx.__init__ into in 35b34202dd3b
(context: handle partial nodeids in revsymbol(), 2018-04-08).
Differential Revision: https://phab.mercurial-scm.org/D3310
Martin von Zweigbergk <martinvonz@google.com> [Fri, 13 Apr 2018 22:46:37 -0700] rev 37678
scmutil: rename resolvepartialhexnodeid() to resolvehexnodeidprefix()
I'm going to wrap revlog.shortest() in scmutil. I plan on calling it
shortesthexnodeidprefix(). resolvehexnodeidprefix() matches that
better. Also, "prefix" carries more information than "partial".
Differential Revision: https://phab.mercurial-scm.org/D3309
Martin von Zweigbergk <martinvonz@google.com> [Fri, 13 Apr 2018 09:48:22 -0700] rev 37677
scmutil: document that isrevsymbol() raises on ambiguous node prefix
Differential Revision: https://phab.mercurial-scm.org/D3308
Martin von Zweigbergk <martinvonz@google.com> [Fri, 13 Apr 2018 09:19:38 -0700] rev 37676
context: set stack level for deprecation warning
This patch makes the deprecation warning print the caller of
repo.__getitem__. That's not quite correct, since there could also be
other callers of changectx.__init__, but it seems like the most
helpful stack level in practice.
Differential Revision: https://phab.mercurial-scm.org/D3307
Martin von Zweigbergk <martinvonz@google.com> [Fri, 13 Apr 2018 14:18:26 -0700] rev 37675
util: set correct stack level on deprecation warnings
Without this patch, you'll get something like this:
<path>/mercurial/util.py:3784: DeprecationWarning: 'util.hgexecutable'
is deprecated, use 'utils.procutil.hgexecutable'
(but on one line)
Differential Revision: https://phab.mercurial-scm.org/D3331
Yuya Nishihara <yuya@tcha.org> [Sat, 14 Apr 2018 12:57:32 +0900] rev 37674
revset: pass in lookup function instead of repo (API)
And document that it's only for legacy lookup. If we have a repo, we're
likely to do more things where that shouldn't be done.
Yuya Nishihara <yuya@tcha.org> [Sat, 14 Apr 2018 12:44:40 +0900] rev 37673
revset: drop support for posttreebuilthook() (API)
AFAIK, the only user was the directaccess extension, which is in core now.
Pulkit Goyal <7895pulkit@gmail.com> [Sat, 14 Apr 2018 02:07:42 +0530] rev 37672
py3: make values bytes before passing into server.runservice()
The values of opts dict still needed to be converted to bytes.
Differential Revision: https://phab.mercurial-scm.org/D3330
Augie Fackler <augie@google.com> [Fri, 13 Apr 2018 18:44:18 -0400] rev 37671
py3: two more passing tests
Differential Revision: https://phab.mercurial-scm.org/D3338
Augie Fackler <augie@google.com> [Fri, 13 Apr 2018 18:28:58 -0400] rev 37670
localrepo: add some overlooked strkwargs love for py3
Differential Revision: https://phab.mercurial-scm.org/D3337
Augie Fackler <augie@google.com> [Fri, 13 Apr 2018 18:17:45 -0400] rev 37669
py3: paper over differences in future exception handling
It looks like Python 3's futures library lacks set_exception_info
entirely. We'll just give up and use set_exception in that case.
# no-check-commit because the underbar naming is just saner here
Differential Revision: https://phab.mercurial-scm.org/D3336
Pulkit Goyal <7895pulkit@gmail.com> [Sat, 14 Apr 2018 02:06:03 +0530] rev 37668
py3: add b'' prefixes to make values bytes
# skip-blame beacuse just b'' prefixes
Differential Revision: https://phab.mercurial-scm.org/D3329
Pulkit Goyal <7895pulkit@gmail.com> [Sat, 14 Apr 2018 02:04:19 +0530] rev 37667
py3: use b"%d" instead of str() to convert int to bytes
While I was here, I added 'and None' to suppress return values of .write()
calls.
Differential Revision: https://phab.mercurial-scm.org/D3328
Pulkit Goyal <7895pulkit@gmail.com> [Sat, 14 Apr 2018 02:03:02 +0530] rev 37666
py3: add b'' prefixes to make values bytes
# skip-blame because just b'' prefixes
Differential Revision: https://phab.mercurial-scm.org/D3327
Pulkit Goyal <7895pulkit@gmail.com> [Sat, 14 Apr 2018 02:02:11 +0530] rev 37665
py3: use str variables to check keys in request header
The values in header are of str type.
Differential Revision: https://phab.mercurial-scm.org/D3326
Pulkit Goyal <7895pulkit@gmail.com> [Sat, 14 Apr 2018 02:00:43 +0530] rev 37664
py3: make sure curses.tigetstr() first argument is a str
Differential Revision: https://phab.mercurial-scm.org/D3325
Pulkit Goyal <7895pulkit@gmail.com> [Sat, 14 Apr 2018 01:59:55 +0530] rev 37663
py3: use stringutil.forcebytestr() instead of str()
We need to convert errors to bytes using stringutil.forcebytestr()
Differential Revision: https://phab.mercurial-scm.org/D3324
Pulkit Goyal <7895pulkit@gmail.com> [Sat, 14 Apr 2018 01:58:44 +0530] rev 37662
py3: iterate over a copy of dict while changing it
Differential Revision: https://phab.mercurial-scm.org/D3323
Pulkit Goyal <7895pulkit@gmail.com> [Sat, 14 Apr 2018 00:56:44 +0530] rev 37661
py3: use urllib.parse.unquote_plus instead of urllib.unquote_plus
The later is not present in Python 3.
Differential Revision: https://phab.mercurial-scm.org/D3322
Pulkit Goyal <7895pulkit@gmail.com> [Fri, 13 Apr 2018 18:41:56 +0530] rev 37660
py3: add b'' prefixes to tests/test-status-inprocess.py
# skip-blame because just b'' prefixes
Differential Revision: https://phab.mercurial-scm.org/D3321
Pulkit Goyal <7895pulkit@gmail.com> [Fri, 13 Apr 2018 16:31:02 +0530] rev 37659
lock: don't use 'file' as a variable name
Differential Revision: https://phab.mercurial-scm.org/D3320
Pulkit Goyal <7895pulkit@gmail.com> [Fri, 13 Apr 2018 16:30:27 +0530] rev 37658
py3: use b"%d" instead of str() to convert integers to bytes
Differential Revision: https://phab.mercurial-scm.org/D3319
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 13 Apr 2018 12:50:47 -0700] rev 37657
wireproto: expose repository formats via capabilities
Servers need to expose their set of repository storage requirements
in order to facilitate streaming clones (clients need to know
if they are capable of reading the raw storage files that the
server exposes).
Differential Revision: https://phab.mercurial-scm.org/D3335
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 13 Apr 2018 12:49:47 -0700] rev 37656
wireprotoframing: record when new stream is encountered
Without this, we choke after receiving the 2nd frame in a stream.
Not sure how we made it this far without finding this bug.
Differential Revision: https://phab.mercurial-scm.org/D3334
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 12 Apr 2018 13:11:29 -0700] rev 37655
wireprotoframing: use value passed into function
Oops.
Differential Revision: https://phab.mercurial-scm.org/D3333
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 12 Apr 2018 13:08:33 -0700] rev 37654
httppeer: handle error response from client reactor
With this in place, we're now seeing useful errors when running
tests with the new wire protocol enabled!
Differential Revision: https://phab.mercurial-scm.org/D3332
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 13 Apr 2018 12:31:56 -0700] rev 37653
wireproto: add media type to version 2 capabilities response
This is useful to advertise because servers reject unsupported
media types. A client may wish to speak multiple media types and
choose the one the server supports.
I doubt we'll ever use multiple media types or negotiation in core.
But during the course of developing this protocol, I may end up
making extensions that backport and forward port protocol support
as needed to support Mercurial deploys in the wild. e.g. I may
deploy support for an older protocol on a server so old clients
can continue using it.
It's worth pursuing changing the SSH protocol's upgrade mechanism
to support multiple media types as well...
Differential Revision: https://phab.mercurial-scm.org/D3299
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 13 Apr 2018 12:31:34 -0700] rev 37652
debugcommands: use command executor for invoking commands
Now that all peers support the command executor interface, we
can use it in place of peer._call() (which isn't part of the public
API).
Tests changed because we are now returning the decoded response
instead of the raw bytes. The raw bytes over the wire are already
logged. So we're not losing any test coverage. In fact, we're adding
test coverage because we're testing decoding of those command
responses as well.
Differential Revision: https://phab.mercurial-scm.org/D3298
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 13 Apr 2018 12:30:04 -0700] rev 37651
httppeer: implement command executor for version 2 peer
Now that we have a new API for issuing commands which is compatible
with wire protocol version 2, we can start using it with wire protocol
version 2.
This commit replaces our hacky implementation of _call() with something
a bit more robust based on the new command executor interface.
We now have proper support for issuing multiple commands per HTTP
request. Each HTTP request maintains its own client reactor.
The implementation is similar to the one in the legacy wire protocol.
We use a ThreadPoolExecutor for spinning up a thread to read the HTTP
response in the background. This allows responses to resolve in any
order. While not implemented on the server yet, a client could use
concurrent.futures.as_completed() with a collection of futures and
handle responses as they arrive from the server.
The return value from issued commands is still a simple list of raw
or decoded CBOR data. This is still super hacky. We will want a rich
data type for representing command responses. But at least this
commit gets us one step closer to a proper peer implementation.
Differential Revision: https://phab.mercurial-scm.org/D3297
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 13 Apr 2018 11:54:13 -0700] rev 37650
repository: remove ipeercommands from ipeerbase
AFAICT all callers in core have moved to the commandexecutor
interface for invoking wire protocol commands. Or at least they
aren't using the named methods on ipeercommands to invoke them.
This means we can drop ipeercommands from the ipeerbase interface.
As far as interface based programming goes, it is now illegal to call
an ipeercommands method for issuing wire protocol commands. However,
the methods are still there, so they will still work. At some
point we will want to break that API...
Differential Revision: https://phab.mercurial-scm.org/D3318
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 13 Apr 2018 12:13:42 -0700] rev 37649
wireproto: properly call clonebundles command
We should not be using _call() to make wire protocol calls because
it isn't part of the peer API.
But clonebundles wasn't part of the supported commands in the
peer API!
So this commit defines that command in the commands interface,
implements it, and teaches the one caller in core to call it using
the command executor interface.
Differential Revision: https://phab.mercurial-scm.org/D3317
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 13 Apr 2018 11:37:37 -0700] rev 37648
exchange: use command executor for getbundle
The code consuming the bundle has been moved to inside the
context manager, as that is supposed to be part of the API.
(Although it doesn't matter for version 1 peers.)
Differential Revision: https://phab.mercurial-scm.org/D3316
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 13 Apr 2018 11:45:38 -0700] rev 37647
exchange: use command executor for pushkey
Differential Revision: https://phab.mercurial-scm.org/D3315
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 13 Apr 2018 11:30:44 -0700] rev 37646
wireproto: use command executor for unbundle
This also required unifying the name of the argument because the
new API always passes arguments by keyword. I decided to change
implementations to "bundle" instead of the interface to "cg"
because "bundle" is more appropriate in a modern world.
Differential Revision: https://phab.mercurial-scm.org/D3314
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 12 Apr 2018 12:33:07 -0700] rev 37645
debugcommands: perform handshake when obtaining httpv2 peer
If we obtain an httpv2peer directly, the instance doesn't have
an API descriptor and therefore doesn't know about the remote's
commands, feature support, etc. This doesn't matter now. But when
we implement the peer so it consults the API descriptor as part
of sending commands, it will.
So we change the logic for obtaining an http version 2 peer to
go through makepeer() so the peer will perform the handshake and
pass the API descriptor to the httpv2peer instance.
Tests changed because we now perform a ?cmd=capabilities when
obtaining version 2 peers.
The Content-Length header is globbed because compression info
will lack zstandard for pure builds.
Differential Revision: https://phab.mercurial-scm.org/D3296
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 11 Apr 2018 18:15:51 -0700] rev 37644
wireproto: rename HTTPV2 so it less like HTTP/2
Per review suggestion on D3230 from Augie.
Differential Revision: https://phab.mercurial-scm.org/D3295
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 13 Apr 2018 11:21:55 -0700] rev 37643
bundlerepo: use command executor for wire protocol commands
Differential Revision: https://phab.mercurial-scm.org/D3294
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 11 Apr 2018 17:51:40 -0700] rev 37642
bundlerepo: rename "other" to "peer"
Differential Revision: https://phab.mercurial-scm.org/D3293
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 13 Apr 2018 11:19:39 -0700] rev 37641
bookmarks: use command executor for wire protocol commands
And change the name of a variable to reflect that is is a peer.
Differential Revision: https://phab.mercurial-scm.org/D3292
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 13 Apr 2018 11:17:45 -0700] rev 37640
hg: use command executor for wire protocol commands
As part of this, I realized that some uses of lookup in a loop
could be converted to use a batch request. But I didn't change
behavior and left in a todo to track potentially changing this.
Differential Revision: https://phab.mercurial-scm.org/D3291
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 13 Apr 2018 11:14:54 -0700] rev 37639
logexchange: use command executor for wire protocol commands
Differential Revision: https://phab.mercurial-scm.org/D3290
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 13 Apr 2018 11:14:19 -0700] rev 37638
streamclone: use command executor for wire protocol commands
Differential Revision: https://phab.mercurial-scm.org/D3289
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 13 Apr 2018 11:13:05 -0700] rev 37637
discovery: use command executor interface
We're trying to port all wire protocol code to use the new
interface so we can implement wire protocol version 2 clients.
Differential Revision: https://phab.mercurial-scm.org/D3288
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 11 Apr 2018 17:24:43 -0700] rev 37636
discovery: don't redundantly call branchmap
We were calling the remote command twice without mutation the
remote in between. Derp.
Differential Revision: https://phab.mercurial-scm.org/D3287
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 13 Apr 2018 11:12:19 -0700] rev 37635
wireproto: convert legacy commands to command executor
Calls to the legacy commands "changegroup" and "changegroupsubset" have
been ported to the new command executor interface.
Because we always pass arguments by name and not position, some
inconsistent names throughout the code base have been unified.
As part of this change, we no longer had any remaining callers
of the legacy command methods {between, branches, changegroup,
changegroupsubset}. So, these interfaces/methods have been dropped
from peer interfaces. We still have an interface declaring these
methods. But that interface is implemented on the concrete peer
types and isn't part of the generic peer interface. (The
implementations of the command executor continue to call these
methods.)
The ultimate goal is to remove the per-command methods from the
generic peer interface: the only interface-conforming way to
call a command will be with the new executor API. At some point,
we may want to move the methods outside of the peer classes and
change the executor implementations to not call methods directly
on a peer instance.
Differential Revision: https://phab.mercurial-scm.org/D3273
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 13 Apr 2018 11:10:59 -0700] rev 37634
treediscovery: switch to command executor interface
We now have a new interface for requesting that commands run.
Switch to it.
Differential Revision: https://phab.mercurial-scm.org/D3272
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 11 Apr 2018 16:18:26 -0700] rev 37633
wireproto: remove iterbatch() from peer interface (API)
Good riddance.
Some tests have been ported to the new API. This probably should
have been done in earlier commits. But duplicating the test coverage
would have been difficult. It was easier this way.
.. api::
The wire protocol peer's ``iterbatch()`` for bulk executing commands
has been remove.d Use ``peer.commandexecutor()`` instead.
Differential Revision: https://phab.mercurial-scm.org/D3271
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 13 Apr 2018 11:08:46 -0700] rev 37632
largefiles: use command executor for batch operation
This is the only other user of iterbatch() in core.
Tests changed because the new command executor is smart enough
to not send a "batch" command over the wire if only 1 command
was requested.
There is still coverage for the "batch" command in this test
though.
Differential Revision: https://phab.mercurial-scm.org/D3270
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 13 Apr 2018 11:02:34 -0700] rev 37631
wireproto: implement batching on peer executor interface
This is a bit more complicated than non-batch requests because we
need to buffer sends until the last request arrives *and* we need
to support resolving futures as data arrives from the remote.
In a classical concurrent.futures executor model, the future
"starts" as soon as it is submitted. However, we have nothing to
start until the last command is submitted.
If we did nothing, calling result() would deadlock, since the future
hasn't "started." So in the case where we queue the command, we return
a special future type whose result() will trigger sendcommands().
This eliminates the deadlock potential. It also serves as a check
against callers who may be calling result() prematurely, as it will
prevent any subsequent callcommands() from working. This behavior
is slightly annoying and a bit restrictive. But it's the world
that half duplex connections forces on us.
In order to support streaming responses, we were previously using
a generator. But with a futures-based API, we're using futures
and not generators. So in order to get streaming, we need a
background thread to read data from the server.
The approach taken in this patch is to leverage the ThreadPoolExecutor
from concurrent.futures for managing a background thread. We create
an executor and future that resolves when all response data is
processed (or an error occurs). When exiting the context manager,
we wait on that background reading before returning.
I was hoping we could manually spin up a threading.Thread and this
would be simple. But I ran into a few deadlocks when implementing.
After looking at the source code to concurrent.futures, I figured
it would just be easier to use a ThreadPoolExecutor than implement
all the code needed to manually manage a thread.
To prove this works, a use of the batch API in discovery has been
updated.
Differential Revision: https://phab.mercurial-scm.org/D3269
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 13 Apr 2018 10:51:23 -0700] rev 37630
wireproto: implement command executor interface for version 1 peers
Now that we've defined our new interface for issuing commands,
let's implement it.
We add the interface to the base peer interface. This means all
peer types must implement it.
The only peer types that we have are the local peer in localrepo
and a shared wire peer for version 1 of the wire protocol.
The local peer implementation is pretty straightforward. We
don't do anything fancy and just return a resolved future with
the result of a method call. This is similar to what
localiterbatcher does.
The wire protocol version 1 implementation is a bit more complicated
and is a more robust implementation.
The wire executor queues commands by default. And because the new
executor interface always allows multiple commands but not all version
1 commands are @batchable, it has to check that the requested commands
are batchable if multiple commands are being requested.
The wire executor currently only supports executing a single command.
This is for simplicity reasons. Support for multiple commands will
be added in a separate commit.
To prove the new interface works, a call to the "known" command
during discovery has been updated to use the new API.
It's worth noting that both implementations require a method having
the command name to exist on the peer. There is at least one caller
in core that don't have a method calls peer._call() directly. We
may need to shore up the requirements later...
Differential Revision: https://phab.mercurial-scm.org/D3268
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 13 Apr 2018 10:23:05 -0700] rev 37629
repository: define new interface for running commands
Today, the peer interface exposes methods for each command that can
be executed. In addition, there is an iterbatch() API that allows
commands to be issued in batches and provides an iterator over the
results. This is a glorified wrapper around the "batch" wire command.
Wire protocol version 2 supports nicer things (such as batching
any command and out-of-order replies). It will require a more
flexible API for executing commands.
This commit introduces a new peer interface for making command
requests. In the new world, you can't simply call a method on the
peer to execute a command: you need to obtain an object to be used
for executing commands. That object can be used to issue a single
command or it can batch multiple requests. In the case of full duplex
peers, the command may even be sent out over the wire immediately.
There are no per-command methods. Instead, there is a generic
method to call a command. The implementation can then perform domain
specific processing for specific commands. This includes passing
data via a specially named argument.
Arguments are also passed as a dictionary instead of using **kwargs.
While **kwargs is nicer to use, we've historically gotten into
trouble using it because there will inevitably be a conflict between
the name of an argument to a wire protocol command and an argument
we want to pass into a function.
Instead of a command returning a value, it returns a future which
will resolve to a value. This opens the door for out-of-order
response handling and concurrent response handling in the version
2 protocol.
Differential Revision: https://phab.mercurial-scm.org/D3267
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 09 Apr 2018 12:28:57 -0700] rev 37628
pycompat: export a handle on concurrent.futures
On Python 3, we use the built-in version in the standard library. Else
we use our vendored backport.
Differential Revision: https://phab.mercurial-scm.org/D3266
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 09 Apr 2018 12:27:52 -0700] rev 37627
setup: add packages for concurrent.futures
We conceivably don't need to distribute this package on Python 3
since we will use the version in the standard library. However,
we want installs to be usable of multiple versions of Python. So
it is best to always have it.
Differential Revision: https://phab.mercurial-scm.org/D3265
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 09 Apr 2018 12:23:48 -0700] rev 37626
futures: switch to absolute and relative imports
This makes the package conform with our importing policy,
silencing a number of warnings. It also makes the package usable
when it isn't named "concurrent.futures."
Differential Revision: https://phab.mercurial-scm.org/D3264
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 09 Apr 2018 12:22:31 -0700] rev 37625
tests: silence pyflakes for thirdparty/concurrent
It is complaining about unused imports.
Differential Revision: https://phab.mercurial-scm.org/D3263
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 09 Apr 2018 12:19:37 -0700] rev 37624
futures: get rid of extend_path
This is used so mutliple directories can provide a package. We don't
need it when vendoring.
Differential Revision: https://phab.mercurial-scm.org/D3262
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 11 Apr 2018 14:48:24 -0700] rev 37623
thirdparty: vendor futures 3.2.0
Python 3 has a concurrent.futures package in the standard library
for representing futures. The "futures" package on PyPI is a backport
of this package to work with Python 2.
The wire protocol code today has its own future concept for handling
of "batch" requests. The frame-based protocol will also want to
use futures.
I've heavily used the "futures" package on Python 2 in other projects
and it is pretty nice. It even has a built-in thread and process pool
for running functions in parallel. I've used this heavily for concurrent
I/O and other GIL-less activities.
The existing futures API in the wire protocol code is not as nice as
concurrent.futures. Since concurrent.futures is in the Python standard
library and will presumably be the long-term future for futures in our
code base, let's vendor the backport so we can use proper futures today.
# no-check-commit because of style violations
Differential Revision: https://phab.mercurial-scm.org/D3261
Pulkit Goyal <7895pulkit@gmail.com> [Thu, 12 Apr 2018 15:05:49 +0530] rev 37622
py3: make sure decode() first argument is str
Uses pycompat.sysstr() to make sure we uses bytes on Python 2 and unicodes on
Python 3.
Differential Revision: https://phab.mercurial-scm.org/D3279
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 12 Apr 2018 23:14:38 -0700] rev 37621
patch: make extract() a context manager (API)
Previously, this function was creating a temporary file and relying
on callers to unlink it. Yuck.
We convert the function to a context manager and tie the lifetime of
the temporary file to that of the context manager. This changed
indentation not only from the context manager, but also from the
elination of try blocks. It was just easier to split the heart of
extract() into its own function.
The single consumer of this function has been refactored to use it as
a context manager. Code for cleaning up the file in tryimportone()
has also been removed.
.. api::
``patch.extract()`` is now a context manager. Callers no longer have
to worry about deleting the temporary file it creates, as the file is
tied to the lifetime of the context manager.
Differential Revision: https://phab.mercurial-scm.org/D3306
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 12 Apr 2018 23:06:27 -0700] rev 37620
cmdutil: pass in parsed patch to tryimportone() (API)
Previously, we parsed the patch in tryimportone(). This assumes the
input is in a patch format that needs to be parsed. We want to support
feeding in data from other formats. So let's let the caller handle the
parsing.
One wonky thing about patch parsing is that patch.extract() creates
a temp file to hold the diffs and it is up to tryimportone() to
unlink that temp file. I'll improve this in a subsequent commit.
Differential Revision: https://phab.mercurial-scm.org/D3305
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 12 Apr 2018 20:42:42 -0700] rev 37619
stringutil: support more types with pprint()
bytearray wasn't working. Integers and floats were not being
formatted.
I /think/ %f is portable across both Python 2 and 3, as it should
default to 6 decimal points on each.
Differential Revision: https://phab.mercurial-scm.org/D3302
Augie Fackler <augie@google.com> [Thu, 12 Apr 2018 14:27:13 -0400] rev 37618
fix: port most of the way to python 3
Only most of the way because we now have to decide: if we want to keep
the current .format() interface for the config in hgrc, we have to use
unicodes to do formatting in Python 3, rather than bytes. I'm
basically fine with that, so a follow-up patch will do so.
Differential Revision: https://phab.mercurial-scm.org/D3300
Matt Harbison <matt_harbison@yahoo.com> [Thu, 12 Apr 2018 17:24:55 -0700] rev 37617
lfs: teach the blob server to handle --prefix
Matt Harbison <matt_harbison@yahoo.com> [Thu, 05 Apr 2018 15:42:40 -0400] rev 37616
hgweb: fallback to checking wsgireq.env for REPO_NAME for 3rd party hosting
Starting with d7fd203e36cc, SCM Manager began to 404 any repository access.
What's happening is that it is generating a python script that creates an hgweb
application (not hgwebdir), and launches hgweb via wsgicgi. It must be setting
REPO_NAME in the process environment before launching this script, which gets
picked up and put into wsgireq.env when wsgicgi launches the hgweb application.
>From there, other variables (notably 'apppath' and 'dispatchpath') are
constructed differently.
d7fd203e36cc^ (working):
apppath: /hg/eng/devsetup
dispatchpath:
pathinfo: /eng/devsetup
reponame: eng/devsetup
d7fd203e36cc:
apppath: /hg
dispatchpath: eng/devsetup
pathinfo: /eng/devsetup
reponame: None
REPO_NAME: eng/devsetup
Rather than having an existing installation break when Mercurial is upgraded,
just resume checking the environment. I have no idea how many other hosting
solutions would break without restoring this.
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 11 Apr 2018 12:51:09 -0700] rev 37615
peer: scatter module to the wind (API)
peer.py hardly contained any code. The code it did contain was
generic to the version 1 peer interface or specific to the
local repository peer.
So code has been moved to wireprotov1peer and localrepo, as
appropriate.
Differential Revision: https://phab.mercurial-scm.org/D3260
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 11 Apr 2018 12:49:08 -0700] rev 37614
wireproto: move version 1 peer functionality to standalone module (API)
wireproto.py contains code for both the client and the server. There
*should* be a somewhat strong separation between the two.
This commit extracts the client-side code from wireproto.py into a new
module - wireprotov1peer.
Differential Revision: https://phab.mercurial-scm.org/D3259
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 11 Apr 2018 10:51:38 -0700] rev 37613
wireproto: move gboptsmap to wireprototypes and rename (API)
This is also shared between client and server and will need to
exist in a shared module when that code is split into different
modules.
Differential Revision: https://phab.mercurial-scm.org/D3258
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 11 Apr 2018 10:50:58 -0700] rev 37612
wireproto: move value encoding functions to wireprototypes (API)
These functions should live in the same place. I plan to separate
client from server code in upcoming commits. wireprototypes is
where we are putting shared code like this.
Differential Revision: https://phab.mercurial-scm.org/D3257
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 10 Apr 2018 19:09:35 -0700] rev 37611
httppeer: basic implementation of capabilities interface
This is a bit crude. The capabilities mechanism for version 2 of
the wire protocol is a bit different from version 1. And code
in core is relying on strings passed to capable() matching strings
advertised by the "capabilities" wire protocol command. I may
refactor the internal checking mechanism to be a bit more
abstract or based on interfaces. Time will tell...
Differential Revision: https://phab.mercurial-scm.org/D3256
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 10 Apr 2018 18:47:09 -0700] rev 37610
repository: split capabilities methods into separate interface
So we can implement them without having to implement support for
every command.
Differential Revision: https://phab.mercurial-scm.org/D3255
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 11 Apr 2018 11:03:45 -0700] rev 37609
httppeer: implement ipeerconnection
This is low hanging fruit. We might as well start somewhere.
Differential Revision: https://phab.mercurial-scm.org/D3254
Augie Fackler <augie@google.com> [Thu, 12 Apr 2018 13:25:54 -0400] rev 37608
py3: whitelist another six passing tests
Differential Revision: https://phab.mercurial-scm.org/D3286
Augie Fackler <augie@google.com> [Wed, 11 Apr 2018 17:43:00 -0400] rev 37607
py3: whitelist another nine passing tests
Differential Revision: https://phab.mercurial-scm.org/D3253
Augie Fackler <augie@google.com> [Wed, 11 Apr 2018 14:01:37 -0400] rev 37606
hgweb: use our forked wsgiheaders module instead of stdlib one
Now we use bytes for headers, rather than native strings.
Differential Revision: https://phab.mercurial-scm.org/D2854
Augie Fackler <augie@google.com> [Thu, 12 Apr 2018 10:00:09 -0700] rev 37605
wsgiheaders: import a bytes-ified fork of wsgiref.headers from cpython@46f5072
This will let us restore Python 3 compatibility for tests that do http things.
Differential Revision: https://phab.mercurial-scm.org/D3245