debugcommands: ability to suppress logging of handshake The tests for calling wire protocol commands were getting quite verbose because they included the results of the capabilities request. Furthermore, it was annoying to have to update several tests every time the capabilities response changed. The only tests that really care about the low-level details of the capabilities requests are those testing the protocol handshake. And those are mostly not instantiating peer instances or are contained to limited files. This commit adds an option to `hg debugwireproto` to suppress logging of the handshake. The shell helper function to perform HTTP tests has been updated to use this by default. Lots of excessive test output has gone away. Differential Revision: https://phab.mercurial-scm.org/D3378

hg: pass command intents to repo/peer creation (API) The previous commit introduced a mechanism to declare command intents. This commit changes the repository and peer instantiation mechanism so the intents are passed down to each repository and peer type so they can do with them whatever they please. Currently, nobody does anything with any intent. Differential Revision: https://phab.mercurial-scm.org/D3377

registrar: replace "cmdtype" with an intent-based mechanism (API) Commands perform varied actions and repositories vary in their capabilities. Historically, the .hg/requires file has been used to lock out clients lacking a requirement. But this is a very heavy-handed approach and is typically reserved for cases where the on-disk storage format changes and we want to prevent incompatible clients from operating on a repo. Outside of the .hg/requires file, we tend to deal with things like optional, extension-provided features via checking at call sites. We'll either have checks in core or extensions will monkeypatch functions in core disabling incompatible features, enabling new features, etc. Things are somewhat tolerable today. But once we introduce alternate storage backends with varying support for repository features and vastly different modes of behavior, the current model will quickly grow unwieldy. For example, the implementation of the "simple store" required a lot of hacks to deal with stripping and verify because various parts of core assume things are implemented a certain way. Partial clone will require new ways of modeling file data retrieval, because we can no longer assume that all file data is already local. In this new world, some commands might not make any sense for certain types of repositories. What we need is a mechanism to affect the construction of repository (and eventually peer) instances so the requirements/capabilities needed for the current operation can be taken into account. "Current operation" can almost certainly be defined by a command. So it makes sense for commands to declare their intended actions. This commit introduces the "intents" concept on the command registrar. "intents" captures a set of strings that declare actions that are anticipated to be taken, requirements the repository must possess, etc. These intents will be passed into hg.repo(), which will pass them into localrepository, where they can be used to influence the object being created. Some use cases for this include: * For read-only intents, constructing a repository object that doesn't expose methods that can mutate the repository. Its VFS instances don't even allow opening a file with write access. * For read-only intents, constructing a repository object without cache invalidation logic. If the repo never changes during its lifetime, nothing ever needs to be invalidated and we don't need to do expensive things like verify the changelog's hidden revisions state is accurate every time we access repo.changelog. * We can automatically hide commands from `hg help` when the current repository doesn't provide that command. For example, an alternate storage backend may not support `hg commit`, so we can hide that command or anything else that would perform local commits. We already kind of had an "intents" mechanism on the registrar in the form of "cmdtype." However, it was never used. And it was limited to a single value. We really need something that supports multiple intents. And because intents may be defined by extensions and at this point are advisory, I think it is best to define them in a set rather than as separate arguments/attributes on the command. Differential Revision: https://phab.mercurial-scm.org/D3376

cleanup: polyfill assertRaisesRegex so we can avoid assertRaisesRegexp The latter is deprecated on Python 3.7 and causes our tests to fail due to the warning. Differential Revision: https://phab.mercurial-scm.org/D3375

tests: add b prefixes to test-hg-parseurl.py Now passes on Python 3. Differential Revision: https://phab.mercurial-scm.org/D3374

tests: port test-hg-parseurl.py to unittest Differential Revision: https://phab.mercurial-scm.org/D3373

hgwebdir: un-bytes the env dict before re-parsing env Not the most elegant, but it restores test-subrepo-deep-nested-change.t to passing on Python 3. Differential Revision: https://phab.mercurial-scm.org/D3367

cborutil: implement support for streaming encoding, bytestring decoding The vendored cbor2 package is... a bit disappointing. On the encoding side, it insists that you pass it something with a write() to send data to. That means if you want to emit data to a generator, you have to construct an e.g. io.BytesIO(), write() to it, then get the data back out. There can be non-trivial overhead involved. The encoder also doesn't support indefinite types - bytestrings, arrays, and maps that don't have a known length. Again, this is really unfortunate because it requires you to buffer the entire source and destination in memory to encode large things. On the decoding side, it supports reading indefinite length types. But it buffers them completely before returning. More sadness. This commit implements "streaming" encoders for various CBOR types. Encoding emits a generator of hunks. So you can efficiently stream encoded data elsewhere. It also implements support for emitting indefinite length bytestrings, arrays, and maps. On the decoding side, we only implement support for decoding an indefinite length bytestring from a file object. It will emit a generator of raw chunks from the source. I didn't want to reinvent so many wheels. But profiling the wire protocol revealed that the overhead of constructing io.BytesIO() instances to temporarily hold results has a non-trivial overhead. We're talking >15% of execution time for operations like "transfer the fulltexts of all files in a revision." So I can justify this effort. Fortunately, CBOR is a relatively straightforward format. And we have a reference implementation in the repo we can test against. Differential Revision: https://phab.mercurial-scm.org/D3303

configitems: register server.zstdlevel Somehow, I managed to trigger a devel-warn running `hg serve` outside the test suite on one of the repos generated by running test-lfs-serve-access.t --keep. (I'm not hitting it now after doing a `make local`.) The only reference to this in all of the history is the help text added in e75463e3179f, and the translations. (It looks like the string is built dynamically with '%slevel', which is probably how this was missed. I wonder if this isn't getting routed to error.log for some reason.) In any event, server.zliblevel is registered, and that's required to pick up the documented default.

scmutil: make shortesthexnodeidprefix() take a full binary nodeid The shortest() template function depended on the behavior of revlog._partialmatch() for these types of inputs: * non-hex strings * ambiguous strings * too long strings revlog._partialmatch() seems to return the input unchanged in these cases, but we shouldn't depend on such a low-level function to match the behavior we want in the user-facing template function. Instead, let's handle these cases in the template function and always pass a binary nodeid to _partialmatch(). Differential Revision: https://phab.mercurial-scm.org/D3371

scmutil: make shortesthexnodeidprefix() use unfiltered repo Both callers were doing this, and resolvehexnodeidprefix() was also working on the unfiltered repo, so it makes more sense to have it all in one place. Differential Revision: https://phab.mercurial-scm.org/D3313

templates: adjust white space amount in the output of {whyunstable} There used to be 2 spaces between divergent nodes (when not using custom template for divergentnodes) because divergentnodes is a hybrid list, which means it gets ' '.join()ed, but formatnode() already had a space. Now it doesn't, which requires extra effort in writing custom templates for whyunstable, but at least it looks correctly by default. Test output needs to be sorted for stability.

tests: split long templates that use {whyunstable} and put them in hgrc

tests: make custom templates that use {whyunstable} terser These templates demonstrate that {whyunstable} is fully template-friendly, but they don't need to produce such long lines of output.

tests: hg log shouldn't need --hidden to show whyunstable template keyword

makefile: add Ubuntu Bionic docker targets (.deb and ppa) Bionic Beaver is an LTS release, supported until 2023-04.

makefile: reformat .PHONY targets The list of make targets is likely to evolve over time. This reformats streamlines future patches.

makefile: mkdir is not needed on templatized docker builds This imitates e63dfbbdbd07 and is a small addition to 231690dba9b4 and 5c1283713293

hgweb: convert _siblings to a factory function of mappinggenerator IIUC, only reason it was a class is to make the generator restartable, which is now served by the mappinggenerator.

hgweb: extract a generator function of _siblings class _siblings will be converted to a plain function.

hgweb: wrap {changenav} and {nav} with mappinglist

hgweb: make revnav.gen() simply build a list of mappings by one pass There was actually no lazy stuff.

hgweb: lift {sessionvars} to a wrapped type Since a sessionvars object is updated in-place, we can't simply wrap it by mappinglist.

hgweb: make sessionvars class less dense

hgweb: prefix private variables of sessionvars with '_'

lfs: update the HTTP status codes in error cases I'm not bothering with validating PUT requests (for now), because the spec doesn't explicitly call out a Content-Type (though the example transcript does use the sensible 'application/octet-stream').

lfs: gracefully handle aborts on the server when corrupt blobs are detected The aborts weren't killing the server, but this seems cleaner. I'm not sure if it matters to handle the remaining IOError in the test like this, for consistency. The error code still feels wrong (especially if the client is trying to download a corrupt blob) but I don't see anything better in the RFCs, and this is already used elsewhere because the Batch API spec specifically mentioned this as a "Validation Error".

lfs: fix the inferred remote store path when using a --prefix This wasn't appending the '.git/info/lfs' path in this case.

lfs: log information about Internal Server Errors reported in the Batch API Reporting a 500 and then not leaving any traces on the server seems like a receipe for frustration. There's similar log writing in hgweb.server.do_POST(). That doesn't write directly to the wsgi.errors object, so it doesn't seem worth trying to refactor. It does seem like earlier stack frames are missing for some reason.

test-lfs: add tests to force server error path coverage The tests are somewhat fragile in that the extension that forces the errors is counting how many times some of the functions are being called, so it depends heavily on the content of the repo. Maybe we can do something clever like load an extension on the client, and have it send over instructions in the HTTP header how to fail. (I'm trying to avoid killing and restarting the server, because Windows seems to have issues with doing that a lot.) But I'd rather fix issues than polish tests before the freeze.

keepalive: add ** overlooked in 83250442dc81 Caught by Yuya in D3326. Differential Revision: https://phab.mercurial-scm.org/D3372

test-check-commit: don't run hg per commit We aren't stress testing CPU. $ time ./run-tests.py -l test-check-commit.t --timeout 600 (orig) 162.59s user 17.98s system 101% cpu 2:58.55 total (new) 5.85s user 0.99s system 98% cpu 6.939 total

templates: make divergentnodes in whyunstable keyword be a hybrid list

templates: add whyunstable template keyword

commands: drop spurious r'' on dry_run in forget Fixes test-add.t on Python 3. Differential Revision: https://phab.mercurial-scm.org/D3365

tests: port inline extension in test-http-bundle1.t to py3 # skip-blame just b prefixes Differential Revision: https://phab.mercurial-scm.org/D3366

tests: add all missing b prefixes in reactor tests Both of these tests now pass on Python 3. # skip-blame just b prefixes. So many b prefixes. Differential Revision: https://phab.mercurial-scm.org/D3369

stringutil: ast.literal_eval needs a unicode on py3 Fortunately for us, this is really only used internally, so we can be lazy about the encoding here. test-wireproto-framing.py now passes on Python 3. Differential Revision: https://phab.mercurial-scm.org/D3368

scmutil: introduce shortesthexnodeidprefix() We have scmutil.resolvehexnodeidprefix() for resolving a prefix to a full nodeid, so it makes sense to have the inverse method next to it. For now it just delegates to changelog.shortest(), but it will soon also make sure it's called on the unfiltered repo, to match resolvehexnodeidprefix(). Note that the change in show.py also makes it so the conversion from revnum to nodeid is done on the filtered repo, but that should be inconsequential since the revs are all from the filtered repo anyway. Differential Revision: https://phab.mercurial-scm.org/D3370

scmutil: use resolvehexnodeidprefix() from revsymbol() I should have copied this from changectx.__init__ into in 35b34202dd3b (context: handle partial nodeids in revsymbol(), 2018-04-08). Differential Revision: https://phab.mercurial-scm.org/D3310

scmutil: rename resolvepartialhexnodeid() to resolvehexnodeidprefix() I'm going to wrap revlog.shortest() in scmutil. I plan on calling it shortesthexnodeidprefix(). resolvehexnodeidprefix() matches that better. Also, "prefix" carries more information than "partial". Differential Revision: https://phab.mercurial-scm.org/D3309

scmutil: document that isrevsymbol() raises on ambiguous node prefix Differential Revision: https://phab.mercurial-scm.org/D3308

context: set stack level for deprecation warning This patch makes the deprecation warning print the caller of repo.__getitem__. That's not quite correct, since there could also be other callers of changectx.__init__, but it seems like the most helpful stack level in practice. Differential Revision: https://phab.mercurial-scm.org/D3307

util: set correct stack level on deprecation warnings Without this patch, you'll get something like this: <path>/mercurial/util.py:3784: DeprecationWarning: 'util.hgexecutable' is deprecated, use 'utils.procutil.hgexecutable' (but on one line) Differential Revision: https://phab.mercurial-scm.org/D3331

revset: pass in lookup function instead of repo (API) And document that it's only for legacy lookup. If we have a repo, we're likely to do more things where that shouldn't be done.

revset: drop support for posttreebuilthook() (API) AFAIK, the only user was the directaccess extension, which is in core now.

py3: make values bytes before passing into server.runservice() The values of opts dict still needed to be converted to bytes. Differential Revision: https://phab.mercurial-scm.org/D3330

py3: two more passing tests Differential Revision: https://phab.mercurial-scm.org/D3338

localrepo: add some overlooked strkwargs love for py3 Differential Revision: https://phab.mercurial-scm.org/D3337

py3: paper over differences in future exception handling It looks like Python 3's futures library lacks set_exception_info entirely. We'll just give up and use set_exception in that case. # no-check-commit because the underbar naming is just saner here Differential Revision: https://phab.mercurial-scm.org/D3336

py3: add b'' prefixes to make values bytes # skip-blame beacuse just b'' prefixes Differential Revision: https://phab.mercurial-scm.org/D3329

py3: use b"%d" instead of str() to convert int to bytes While I was here, I added 'and None' to suppress return values of .write() calls. Differential Revision: https://phab.mercurial-scm.org/D3328

py3: add b'' prefixes to make values bytes # skip-blame because just b'' prefixes Differential Revision: https://phab.mercurial-scm.org/D3327

py3: use str variables to check keys in request header The values in header are of str type. Differential Revision: https://phab.mercurial-scm.org/D3326

py3: make sure curses.tigetstr() first argument is a str Differential Revision: https://phab.mercurial-scm.org/D3325

py3: use stringutil.forcebytestr() instead of str() We need to convert errors to bytes using stringutil.forcebytestr() Differential Revision: https://phab.mercurial-scm.org/D3324

py3: iterate over a copy of dict while changing it Differential Revision: https://phab.mercurial-scm.org/D3323

py3: use urllib.parse.unquote_plus instead of urllib.unquote_plus The later is not present in Python 3. Differential Revision: https://phab.mercurial-scm.org/D3322

py3: add b'' prefixes to tests/test-status-inprocess.py # skip-blame because just b'' prefixes Differential Revision: https://phab.mercurial-scm.org/D3321

lock: don't use 'file' as a variable name Differential Revision: https://phab.mercurial-scm.org/D3320

py3: use b"%d" instead of str() to convert integers to bytes Differential Revision: https://phab.mercurial-scm.org/D3319

wireproto: expose repository formats via capabilities Servers need to expose their set of repository storage requirements in order to facilitate streaming clones (clients need to know if they are capable of reading the raw storage files that the server exposes). Differential Revision: https://phab.mercurial-scm.org/D3335

wireprotoframing: record when new stream is encountered Without this, we choke after receiving the 2nd frame in a stream. Not sure how we made it this far without finding this bug. Differential Revision: https://phab.mercurial-scm.org/D3334

wireprotoframing: use value passed into function Oops. Differential Revision: https://phab.mercurial-scm.org/D3333