spectral <spectral@google.com> [Tue, 02 Oct 2018 18:55:07 -0700] rev 40038
treemanifests: extract _loaddifflazy from _diff, use in _filesnotin
Differential Revision: https://phab.mercurial-scm.org/D4873
Valentin Gatien-Baron <vgatien-baron@janestreet.com> [Wed, 03 Oct 2018 18:07:49 -0400] rev 40037
identify: show remote bookmarks in `hg id url -Tjson -B`
I didn't display bookmarks when `default and not ui.quiet`: it seems
strange for templates to depend on --id or -q, and it would take more
code for `hg id url -T {node}` to not request remote bookmarks.
An alternative I thought of was providing lazy data to the formatter,
`fm.data(bookmarks=lambda: fm.formatlist(getbms(), name='bookmark'))`.
The plainformatter would naturally not compute it, the
templateformatter would compute only what it needs, and the other ones
would compute everything, but that's not supported (or I don't see
how), so I abandoned this idea.
Differential Revision: https://phab.mercurial-scm.org/D4872
Augie Fackler <augie@google.com> [Wed, 03 Oct 2018 16:03:16 -0400] rev 40036
showstack: also handle SIGALRM
This is looking *very* handy when debugging mysterious hangs in a
test: you can wrap a hanging invocation in
`perl -e 'alarm shift @ARGV; exec @ARGV' 1`
for example, a hanging `hg pull` becomes
`perl -e 'alarm shift @ARGV; exec @ARGV' 1 hg pull`
where the `1` is the timeout in seconds before the process will be hit
with SIGALRM. After making that edit to the test file, you can then
use --extra-config-opt on run-tests.py to globaly enable showstack
during the test run, so you'll get full stack traces as you force your
hg to exit.
I wonder (but only a little, not enough to take action just yet) if we
should wire up some scaffolding in run-tests itself to automatically
wrap all commands in alarm(3) somehow to avoid hangs in the future?
Differential Revision: https://phab.mercurial-scm.org/D4870
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 03 Oct 2018 13:54:31 -0700] rev 40035
exchangev2: add progress bar around manifest scanning
This can take a long time on large repositories. Let's add a progress
bar so we don't have long periods where it isn't obvious what is
going on.
Differential Revision: https://phab.mercurial-scm.org/D4859
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 01 Oct 2018 13:17:38 -0700] rev 40034
httppeer: report http statistics
Now that keepalive.py records HTTP request count and the
number of bytes sent and received as part of performing those
requests, we can easily print a report on the activity when
closing a peer instance!
Exact byte counts are globbed in tests because they are influenced
by non-deterministic things, such as hostnames and port numbers.
Plus, the exact byte count isn't too important anyway.
I feel obliged to note that printing the byte count could have
security implications. e.g. if sending a password via HTTP basic
auth, the length of that password will influence the byte count
and the reporting of the byte count could be a side-channel leak
of the password length. I /think/ this is beyond our threshold
for concern. But if we think it poses a problem, we can teach the
byte count logging code to e.g. ignore sensitive HTTP request
headers. We could also consider not reporting the byte count of
request headers altogether. But since the wire protocol uses HTTP
headers for sending command arguments, it is kind of important to
report their size.
Differential Revision: https://phab.mercurial-scm.org/D4858
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 01 Oct 2018 12:30:32 -0700] rev 40033
keepalive: track number of bytes received from an HTTP response
We also bubble the byte count up to the HTTPConnection instance and its
parent opener at read time. Unlike sending, there isn't a clear
"end of response" signal we can intercept to defer updating the
accounting.
Differential Revision: https://phab.mercurial-scm.org/D4857
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 01 Oct 2018 12:02:54 -0700] rev 40032
keepalive: track request count and bytes sent
I want wire protocol interactions to report the number of
requests made and bytes transferred.
This commit teaches the very low-level custom HTTPConnection class
to track the number of bytes sent to the socket. This may vary from
the number of bytes that go on the wire due to e.g. TLS. That's OK.
KeepAliveHandler is taught to track the total number of requests
and total number of bytes sent across all requests.
Differential Revision: https://phab.mercurial-scm.org/D4856
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 01 Oct 2018 12:06:36 -0700] rev 40031
url: have httpsconnection inherit from our custom HTTPConnection
This will ensure that any customizations we perform to HTTPConnection
will be available to httpsconnection.
Differential Revision: https://phab.mercurial-scm.org/D4855
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 03 Oct 2018 09:43:01 -0700] rev 40030
cborutil: change buffering strategy
Profiling revealed that we were spending a lot of time on the
line that was concatenating the old buffer with the incoming data
when attempting to decode long byte strings, such as manifest
revisions.
Essentially, we were feeding N chunks of size len(X) << len(Y) into
decode() and continuously allocating a new, larger buffer to hold
the undecoded input. This created substantial memory churn and
slowed down execution.
Changing the code to aggregate pending chunks in a list until we
have enough data to fully decode the next atom makes things much
more efficient.
I don't have exact data, but I recall the old code spending >1s
on manifest fulltexts from the mozilla-unified repo. The new code
doesn't significantly appear in profile output.
Differential Revision: https://phab.mercurial-scm.org/D4854
Martin von Zweigbergk <martinvonz@google.com> [Wed, 03 Oct 2018 10:27:44 -0700] rev 40029
cleanup: some Yoda conditions, this patch removes
It seems the factor 20 is less than the frequency of " < \d" compared
to " \d > ".
Differential Revision: https://phab.mercurial-scm.org/D4862
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 02 Oct 2018 12:43:54 -0700] rev 40028
streamclone: don't support stream clone unless repo feature present
This change means custom repository types must opt in to enabling
stream clone. This seems reasonable, as stream clones are a very
low-level feature that has historically assumed the use of revlogs
and the layout of .hg/ that they entail.
Differential Revision: https://phab.mercurial-scm.org/D4853
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 02 Oct 2018 12:40:39 -0700] rev 40027
localrepo: add repository feature when repo can be stream cloned
Right now, the wire protocol server assumes all repository objects can
be stream cloned (unless the stream clone feature is disabled via
config option).
But not all storage backends or repository objects may support stream
clone.
This commit defines a repository feature denoting whether stream clone
is supported. The feature is defined for revlog-based repositories,
which should currently be "all repositories."
Differential Revision: https://phab.mercurial-scm.org/D4852
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 26 Sep 2018 18:08:08 -0700] rev 40026
wireprotov2: client support for following content redirects
And with the server actually sending content redirects, it is finally
time to implement client support for following them!
When a redirect response is seen, we wait until all data for that
request has been received (it should be nearly immediate since no
data is expected to follow the redirect message). Then we use
a URL opener to make a request. We stuff that response into the
client handler and construct a new response object to track it.
When readdata() is called for servicing requests, we attempt to
read data from the first redirected response. During data reading,
data is processed similarly to as if it came from a frame payload.
The existing test for the functionality demonstrates the client
transparently following the redirect and obtaining the command
response data from an alternate URL!
There is still plenty of work to do here, including shoring up
testing. I'm not convinced things will work in the presence of
multiple redirect responses. And we don't yet implement support
for integrity verification or configuring server certificates
to validate the connection. But it's a start. And it should enable
us to start experimenting with "real" caches.
Differential Revision: https://phab.mercurial-scm.org/D4778
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 26 Sep 2018 18:07:55 -0700] rev 40025
wireprotov2: server support for sending content redirects
A "content redirect" can be sent in place of inline response content.
In terms of code, we model a content redirect as a special type of
response object holding the attributes describing that redirect.
Sending a content redirect thus becomes as simple as the object
emission layer sending an instance of that type. A cacher using
externally-addressable content storage could replace the outgoing
object stream with an object advertising its location.
The bulk of the code in this commit is teaching the output layer
which handles the object stream to recognize alternate location
objects. The rules are that if an alternate location object is
present, it must be the first and only object in the object stream.
Otherwise the server emits an error.
Differential Revision: https://phab.mercurial-scm.org/D4777
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 26 Sep 2018 15:02:19 -0700] rev 40024
wireprotov2: client support for advertising redirect targets
With the server now able to emit a redirect target descriptor, we can
start to teach the client to recognize it.
This commit implements support for filtering the advertised
redirect targets against supported features and for advertising
compatible redirect targets as part of command requests. It also
adds the minimal boilerplate required to fail when a content
redirect is seen.
The server doesn't yet do anything with the advertised redirect
targets. And the client can't yet follow redirects if it did. But
at least we're putting bytes on the wire.
Differential Revision: https://phab.mercurial-scm.org/D4776
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 26 Sep 2018 17:46:48 -0700] rev 40023
wireprotov2: advertise redirect targets in capabilities
This is pretty straightforward.
Redirect targets will require an extension to support. So we've added
a function that can be wrapped to define redirect targets.
To test this, we teach our simple cache test extension to read
redirect targets from a file. It's a bit hacky. But it gets the
job done.
Differential Revision: https://phab.mercurial-scm.org/D4775