cborutil: implement support for streaming encoding, bytestring decoding The vendored cbor2 package is... a bit disappointing. On the encoding side, it insists that you pass it something with a write() to send data to. That means if you want to emit data to a generator, you have to construct an e.g. io.BytesIO(), write() to it, then get the data back out. There can be non-trivial overhead involved. The encoder also doesn't support indefinite types - bytestrings, arrays, and maps that don't have a known length. Again, this is really unfortunate because it requires you to buffer the entire source and destination in memory to encode large things. On the decoding side, it supports reading indefinite length types. But it buffers them completely before returning. More sadness. This commit implements "streaming" encoders for various CBOR types. Encoding emits a generator of hunks. So you can efficiently stream encoded data elsewhere. It also implements support for emitting indefinite length bytestrings, arrays, and maps. On the decoding side, we only implement support for decoding an indefinite length bytestring from a file object. It will emit a generator of raw chunks from the source. I didn't want to reinvent so many wheels. But profiling the wire protocol revealed that the overhead of constructing io.BytesIO() instances to temporarily hold results has a non-trivial overhead. We're talking >15% of execution time for operations like "transfer the fulltexts of all files in a revision." So I can justify this effort. Fortunately, CBOR is a relatively straightforward format. And we have a reference implementation in the repo we can test against. Differential Revision: https://phab.mercurial-scm.org/D3303

configitems: register server.zstdlevel Somehow, I managed to trigger a devel-warn running `hg serve` outside the test suite on one of the repos generated by running test-lfs-serve-access.t --keep. (I'm not hitting it now after doing a `make local`.) The only reference to this in all of the history is the help text added in e75463e3179f, and the translations. (It looks like the string is built dynamically with '%slevel', which is probably how this was missed. I wonder if this isn't getting routed to error.log for some reason.) In any event, server.zliblevel is registered, and that's required to pick up the documented default.

scmutil: make shortesthexnodeidprefix() take a full binary nodeid The shortest() template function depended on the behavior of revlog._partialmatch() for these types of inputs: * non-hex strings * ambiguous strings * too long strings revlog._partialmatch() seems to return the input unchanged in these cases, but we shouldn't depend on such a low-level function to match the behavior we want in the user-facing template function. Instead, let's handle these cases in the template function and always pass a binary nodeid to _partialmatch(). Differential Revision: https://phab.mercurial-scm.org/D3371

scmutil: make shortesthexnodeidprefix() use unfiltered repo Both callers were doing this, and resolvehexnodeidprefix() was also working on the unfiltered repo, so it makes more sense to have it all in one place. Differential Revision: https://phab.mercurial-scm.org/D3313

templates: adjust white space amount in the output of {whyunstable} There used to be 2 spaces between divergent nodes (when not using custom template for divergentnodes) because divergentnodes is a hybrid list, which means it gets ' '.join()ed, but formatnode() already had a space. Now it doesn't, which requires extra effort in writing custom templates for whyunstable, but at least it looks correctly by default. Test output needs to be sorted for stability.

tests: split long templates that use {whyunstable} and put them in hgrc

tests: make custom templates that use {whyunstable} terser These templates demonstrate that {whyunstable} is fully template-friendly, but they don't need to produce such long lines of output.

tests: hg log shouldn't need --hidden to show whyunstable template keyword

makefile: add Ubuntu Bionic docker targets (.deb and ppa) Bionic Beaver is an LTS release, supported until 2023-04.

makefile: reformat .PHONY targets The list of make targets is likely to evolve over time. This reformats streamlines future patches.

makefile: mkdir is not needed on templatized docker builds This imitates e63dfbbdbd07 and is a small addition to 231690dba9b4 and 5c1283713293

hgweb: convert _siblings to a factory function of mappinggenerator IIUC, only reason it was a class is to make the generator restartable, which is now served by the mappinggenerator.

hgweb: extract a generator function of _siblings class _siblings will be converted to a plain function.

hgweb: wrap {changenav} and {nav} with mappinglist

hgweb: make revnav.gen() simply build a list of mappings by one pass There was actually no lazy stuff.

hgweb: lift {sessionvars} to a wrapped type Since a sessionvars object is updated in-place, we can't simply wrap it by mappinglist.

hgweb: make sessionvars class less dense

hgweb: prefix private variables of sessionvars with '_'

lfs: update the HTTP status codes in error cases I'm not bothering with validating PUT requests (for now), because the spec doesn't explicitly call out a Content-Type (though the example transcript does use the sensible 'application/octet-stream').

lfs: gracefully handle aborts on the server when corrupt blobs are detected The aborts weren't killing the server, but this seems cleaner. I'm not sure if it matters to handle the remaining IOError in the test like this, for consistency. The error code still feels wrong (especially if the client is trying to download a corrupt blob) but I don't see anything better in the RFCs, and this is already used elsewhere because the Batch API spec specifically mentioned this as a "Validation Error".

lfs: fix the inferred remote store path when using a --prefix This wasn't appending the '.git/info/lfs' path in this case.

lfs: log information about Internal Server Errors reported in the Batch API Reporting a 500 and then not leaving any traces on the server seems like a receipe for frustration. There's similar log writing in hgweb.server.do_POST(). That doesn't write directly to the wsgi.errors object, so it doesn't seem worth trying to refactor. It does seem like earlier stack frames are missing for some reason.

test-lfs: add tests to force server error path coverage The tests are somewhat fragile in that the extension that forces the errors is counting how many times some of the functions are being called, so it depends heavily on the content of the repo. Maybe we can do something clever like load an extension on the client, and have it send over instructions in the HTTP header how to fail. (I'm trying to avoid killing and restarting the server, because Windows seems to have issues with doing that a lot.) But I'd rather fix issues than polish tests before the freeze.

keepalive: add ** overlooked in 83250442dc81 Caught by Yuya in D3326. Differential Revision: https://phab.mercurial-scm.org/D3372

test-check-commit: don't run hg per commit We aren't stress testing CPU. $ time ./run-tests.py -l test-check-commit.t --timeout 600 (orig) 162.59s user 17.98s system 101% cpu 2:58.55 total (new) 5.85s user 0.99s system 98% cpu 6.939 total

templates: make divergentnodes in whyunstable keyword be a hybrid list

templates: add whyunstable template keyword

commands: drop spurious r'' on dry_run in forget Fixes test-add.t on Python 3. Differential Revision: https://phab.mercurial-scm.org/D3365