Gregory Szorc <gregory.szorc@gmail.com> [Mon, 01 Oct 2018 13:17:38 -0700] rev 40034
httppeer: report http statistics
Now that keepalive.py records HTTP request count and the
number of bytes sent and received as part of performing those
requests, we can easily print a report on the activity when
closing a peer instance!
Exact byte counts are globbed in tests because they are influenced
by non-deterministic things, such as hostnames and port numbers.
Plus, the exact byte count isn't too important anyway.
I feel obliged to note that printing the byte count could have
security implications. e.g. if sending a password via HTTP basic
auth, the length of that password will influence the byte count
and the reporting of the byte count could be a side-channel leak
of the password length. I /think/ this is beyond our threshold
for concern. But if we think it poses a problem, we can teach the
byte count logging code to e.g. ignore sensitive HTTP request
headers. We could also consider not reporting the byte count of
request headers altogether. But since the wire protocol uses HTTP
headers for sending command arguments, it is kind of important to
report their size.
Differential Revision: https://phab.mercurial-scm.org/D4858
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 01 Oct 2018 12:30:32 -0700] rev 40033
keepalive: track number of bytes received from an HTTP response
We also bubble the byte count up to the HTTPConnection instance and its
parent opener at read time. Unlike sending, there isn't a clear
"end of response" signal we can intercept to defer updating the
accounting.
Differential Revision: https://phab.mercurial-scm.org/D4857
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 01 Oct 2018 12:02:54 -0700] rev 40032
keepalive: track request count and bytes sent
I want wire protocol interactions to report the number of
requests made and bytes transferred.
This commit teaches the very low-level custom HTTPConnection class
to track the number of bytes sent to the socket. This may vary from
the number of bytes that go on the wire due to e.g. TLS. That's OK.
KeepAliveHandler is taught to track the total number of requests
and total number of bytes sent across all requests.
Differential Revision: https://phab.mercurial-scm.org/D4856
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 01 Oct 2018 12:06:36 -0700] rev 40031
url: have httpsconnection inherit from our custom HTTPConnection
This will ensure that any customizations we perform to HTTPConnection
will be available to httpsconnection.
Differential Revision: https://phab.mercurial-scm.org/D4855
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 03 Oct 2018 09:43:01 -0700] rev 40030
cborutil: change buffering strategy
Profiling revealed that we were spending a lot of time on the
line that was concatenating the old buffer with the incoming data
when attempting to decode long byte strings, such as manifest
revisions.
Essentially, we were feeding N chunks of size len(X) << len(Y) into
decode() and continuously allocating a new, larger buffer to hold
the undecoded input. This created substantial memory churn and
slowed down execution.
Changing the code to aggregate pending chunks in a list until we
have enough data to fully decode the next atom makes things much
more efficient.
I don't have exact data, but I recall the old code spending >1s
on manifest fulltexts from the mozilla-unified repo. The new code
doesn't significantly appear in profile output.
Differential Revision: https://phab.mercurial-scm.org/D4854
Martin von Zweigbergk <martinvonz@google.com> [Wed, 03 Oct 2018 10:27:44 -0700] rev 40029
cleanup: some Yoda conditions, this patch removes
It seems the factor 20 is less than the frequency of " < \d" compared
to " \d > ".
Differential Revision: https://phab.mercurial-scm.org/D4862
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 02 Oct 2018 12:43:54 -0700] rev 40028
streamclone: don't support stream clone unless repo feature present
This change means custom repository types must opt in to enabling
stream clone. This seems reasonable, as stream clones are a very
low-level feature that has historically assumed the use of revlogs
and the layout of .hg/ that they entail.
Differential Revision: https://phab.mercurial-scm.org/D4853
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 02 Oct 2018 12:40:39 -0700] rev 40027
localrepo: add repository feature when repo can be stream cloned
Right now, the wire protocol server assumes all repository objects can
be stream cloned (unless the stream clone feature is disabled via
config option).
But not all storage backends or repository objects may support stream
clone.
This commit defines a repository feature denoting whether stream clone
is supported. The feature is defined for revlog-based repositories,
which should currently be "all repositories."
Differential Revision: https://phab.mercurial-scm.org/D4852