Tue, 02 Oct 2018 18:55:07 -0700 treemanifests: extract _loaddifflazy from _diff, use in _filesnotin
spectral <spectral@google.com> [Tue, 02 Oct 2018 18:55:07 -0700] rev 40038
treemanifests: extract _loaddifflazy from _diff, use in _filesnotin Differential Revision: https://phab.mercurial-scm.org/D4873
Wed, 03 Oct 2018 18:07:49 -0400 identify: show remote bookmarks in `hg id url -Tjson -B`
Valentin Gatien-Baron <vgatien-baron@janestreet.com> [Wed, 03 Oct 2018 18:07:49 -0400] rev 40037
identify: show remote bookmarks in `hg id url -Tjson -B` I didn't display bookmarks when `default and not ui.quiet`: it seems strange for templates to depend on --id or -q, and it would take more code for `hg id url -T {node}` to not request remote bookmarks. An alternative I thought of was providing lazy data to the formatter, `fm.data(bookmarks=lambda: fm.formatlist(getbms(), name='bookmark'))`. The plainformatter would naturally not compute it, the templateformatter would compute only what it needs, and the other ones would compute everything, but that's not supported (or I don't see how), so I abandoned this idea. Differential Revision: https://phab.mercurial-scm.org/D4872
Wed, 03 Oct 2018 16:03:16 -0400 showstack: also handle SIGALRM
Augie Fackler <augie@google.com> [Wed, 03 Oct 2018 16:03:16 -0400] rev 40036
showstack: also handle SIGALRM This is looking *very* handy when debugging mysterious hangs in a test: you can wrap a hanging invocation in `perl -e 'alarm shift @ARGV; exec @ARGV' 1` for example, a hanging `hg pull` becomes `perl -e 'alarm shift @ARGV; exec @ARGV' 1 hg pull` where the `1` is the timeout in seconds before the process will be hit with SIGALRM. After making that edit to the test file, you can then use --extra-config-opt on run-tests.py to globaly enable showstack during the test run, so you'll get full stack traces as you force your hg to exit. I wonder (but only a little, not enough to take action just yet) if we should wire up some scaffolding in run-tests itself to automatically wrap all commands in alarm(3) somehow to avoid hangs in the future? Differential Revision: https://phab.mercurial-scm.org/D4870
Wed, 03 Oct 2018 13:54:31 -0700 exchangev2: add progress bar around manifest scanning
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 03 Oct 2018 13:54:31 -0700] rev 40035
exchangev2: add progress bar around manifest scanning This can take a long time on large repositories. Let's add a progress bar so we don't have long periods where it isn't obvious what is going on. Differential Revision: https://phab.mercurial-scm.org/D4859
Mon, 01 Oct 2018 13:17:38 -0700 httppeer: report http statistics
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 01 Oct 2018 13:17:38 -0700] rev 40034
httppeer: report http statistics Now that keepalive.py records HTTP request count and the number of bytes sent and received as part of performing those requests, we can easily print a report on the activity when closing a peer instance! Exact byte counts are globbed in tests because they are influenced by non-deterministic things, such as hostnames and port numbers. Plus, the exact byte count isn't too important anyway. I feel obliged to note that printing the byte count could have security implications. e.g. if sending a password via HTTP basic auth, the length of that password will influence the byte count and the reporting of the byte count could be a side-channel leak of the password length. I /think/ this is beyond our threshold for concern. But if we think it poses a problem, we can teach the byte count logging code to e.g. ignore sensitive HTTP request headers. We could also consider not reporting the byte count of request headers altogether. But since the wire protocol uses HTTP headers for sending command arguments, it is kind of important to report their size. Differential Revision: https://phab.mercurial-scm.org/D4858
Mon, 01 Oct 2018 12:30:32 -0700 keepalive: track number of bytes received from an HTTP response
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 01 Oct 2018 12:30:32 -0700] rev 40033
keepalive: track number of bytes received from an HTTP response We also bubble the byte count up to the HTTPConnection instance and its parent opener at read time. Unlike sending, there isn't a clear "end of response" signal we can intercept to defer updating the accounting. Differential Revision: https://phab.mercurial-scm.org/D4857
Mon, 01 Oct 2018 12:02:54 -0700 keepalive: track request count and bytes sent
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 01 Oct 2018 12:02:54 -0700] rev 40032
keepalive: track request count and bytes sent I want wire protocol interactions to report the number of requests made and bytes transferred. This commit teaches the very low-level custom HTTPConnection class to track the number of bytes sent to the socket. This may vary from the number of bytes that go on the wire due to e.g. TLS. That's OK. KeepAliveHandler is taught to track the total number of requests and total number of bytes sent across all requests. Differential Revision: https://phab.mercurial-scm.org/D4856
Mon, 01 Oct 2018 12:06:36 -0700 url: have httpsconnection inherit from our custom HTTPConnection
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 01 Oct 2018 12:06:36 -0700] rev 40031
url: have httpsconnection inherit from our custom HTTPConnection This will ensure that any customizations we perform to HTTPConnection will be available to httpsconnection. Differential Revision: https://phab.mercurial-scm.org/D4855
Wed, 03 Oct 2018 09:43:01 -0700 cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 03 Oct 2018 09:43:01 -0700] rev 40030
cborutil: change buffering strategy Profiling revealed that we were spending a lot of time on the line that was concatenating the old buffer with the incoming data when attempting to decode long byte strings, such as manifest revisions. Essentially, we were feeding N chunks of size len(X) << len(Y) into decode() and continuously allocating a new, larger buffer to hold the undecoded input. This created substantial memory churn and slowed down execution. Changing the code to aggregate pending chunks in a list until we have enough data to fully decode the next atom makes things much more efficient. I don't have exact data, but I recall the old code spending >1s on manifest fulltexts from the mozilla-unified repo. The new code doesn't significantly appear in profile output. Differential Revision: https://phab.mercurial-scm.org/D4854
Wed, 03 Oct 2018 10:27:44 -0700 cleanup: some Yoda conditions, this patch removes
Martin von Zweigbergk <martinvonz@google.com> [Wed, 03 Oct 2018 10:27:44 -0700] rev 40029
cleanup: some Yoda conditions, this patch removes It seems the factor 20 is less than the frequency of " < \d" compared to " \d > ". Differential Revision: https://phab.mercurial-scm.org/D4862
Tue, 02 Oct 2018 12:43:54 -0700 streamclone: don't support stream clone unless repo feature present
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 02 Oct 2018 12:43:54 -0700] rev 40028
streamclone: don't support stream clone unless repo feature present This change means custom repository types must opt in to enabling stream clone. This seems reasonable, as stream clones are a very low-level feature that has historically assumed the use of revlogs and the layout of .hg/ that they entail. Differential Revision: https://phab.mercurial-scm.org/D4853
Tue, 02 Oct 2018 12:40:39 -0700 localrepo: add repository feature when repo can be stream cloned
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 02 Oct 2018 12:40:39 -0700] rev 40027
localrepo: add repository feature when repo can be stream cloned Right now, the wire protocol server assumes all repository objects can be stream cloned (unless the stream clone feature is disabled via config option). But not all storage backends or repository objects may support stream clone. This commit defines a repository feature denoting whether stream clone is supported. The feature is defined for revlog-based repositories, which should currently be "all repositories." Differential Revision: https://phab.mercurial-scm.org/D4852
Wed, 26 Sep 2018 18:08:08 -0700 wireprotov2: client support for following content redirects
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 26 Sep 2018 18:08:08 -0700] rev 40026
wireprotov2: client support for following content redirects And with the server actually sending content redirects, it is finally time to implement client support for following them! When a redirect response is seen, we wait until all data for that request has been received (it should be nearly immediate since no data is expected to follow the redirect message). Then we use a URL opener to make a request. We stuff that response into the client handler and construct a new response object to track it. When readdata() is called for servicing requests, we attempt to read data from the first redirected response. During data reading, data is processed similarly to as if it came from a frame payload. The existing test for the functionality demonstrates the client transparently following the redirect and obtaining the command response data from an alternate URL! There is still plenty of work to do here, including shoring up testing. I'm not convinced things will work in the presence of multiple redirect responses. And we don't yet implement support for integrity verification or configuring server certificates to validate the connection. But it's a start. And it should enable us to start experimenting with "real" caches. Differential Revision: https://phab.mercurial-scm.org/D4778
Wed, 26 Sep 2018 18:07:55 -0700 wireprotov2: server support for sending content redirects
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 26 Sep 2018 18:07:55 -0700] rev 40025
wireprotov2: server support for sending content redirects A "content redirect" can be sent in place of inline response content. In terms of code, we model a content redirect as a special type of response object holding the attributes describing that redirect. Sending a content redirect thus becomes as simple as the object emission layer sending an instance of that type. A cacher using externally-addressable content storage could replace the outgoing object stream with an object advertising its location. The bulk of the code in this commit is teaching the output layer which handles the object stream to recognize alternate location objects. The rules are that if an alternate location object is present, it must be the first and only object in the object stream. Otherwise the server emits an error. Differential Revision: https://phab.mercurial-scm.org/D4777
Wed, 26 Sep 2018 15:02:19 -0700 wireprotov2: client support for advertising redirect targets
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 26 Sep 2018 15:02:19 -0700] rev 40024
wireprotov2: client support for advertising redirect targets With the server now able to emit a redirect target descriptor, we can start to teach the client to recognize it. This commit implements support for filtering the advertised redirect targets against supported features and for advertising compatible redirect targets as part of command requests. It also adds the minimal boilerplate required to fail when a content redirect is seen. The server doesn't yet do anything with the advertised redirect targets. And the client can't yet follow redirects if it did. But at least we're putting bytes on the wire. Differential Revision: https://phab.mercurial-scm.org/D4776
Wed, 26 Sep 2018 17:46:48 -0700 wireprotov2: advertise redirect targets in capabilities
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 26 Sep 2018 17:46:48 -0700] rev 40023
wireprotov2: advertise redirect targets in capabilities This is pretty straightforward. Redirect targets will require an extension to support. So we've added a function that can be wrapped to define redirect targets. To test this, we teach our simple cache test extension to read redirect targets from a file. It's a bit hacky. But it gets the job done. Differential Revision: https://phab.mercurial-scm.org/D4775
(0) -30000 -10000 -3000 -1000 -300 -100 -16 +16 +100 +300 +1000 +3000 +10000 tip