hgweb: inform hgweb.hgweb() entrypoint that paths should be bytes Differential Revision: https://phab.mercurial-scm.org/D3354

tests: port inline cgi script in test-largefiles.t to python 3 Differential Revision: https://phab.mercurial-scm.org/D3353

lfcommands: use %d on known-int in format string Differential Revision: https://phab.mercurial-scm.org/D3352

py3: fix test-shelve.t on Python 3 We get a better error message out of the abort here, which is fine. Differential Revision: https://phab.mercurial-scm.org/D3351

tests: fix up a couple of minor bytes inconsistencies in run-tests.py Only in the automatic bisection code, so fortunately nothing major is amiss. Fixes test-run-tests.t under Python 3. Differential Revision: https://phab.mercurial-scm.org/D3350

tests: make sure test-run-tests.t actually runs run-tests.py under Python 3 I'm fairly certain it hasn't been until now. Mercifully, there doesn't appear to be any ninja breakage. Differential Revision: https://phab.mercurial-scm.org/D3349

py3: another three passing Differential Revision: https://phab.mercurial-scm.org/D3348

httppeer: work around API differences on urllib Request objects Since this is only a problem in httppeer, I'd rather keep this a local-to-the-module kludge rather than pile more on pycompat. We'll still find it easily to clean up later because it checks pycompat.ispy3. Differential Revision: https://phab.mercurial-scm.org/D3347

httppeer: no matter what Python 3 might think, http headers are bytes Differential Revision: https://phab.mercurial-scm.org/D3346

httppeer: fix debug prints to work on Python 3 Differential Revision: https://phab.mercurial-scm.org/D3345

url: some bytes/str cleanup where we interface with stdlib funcs Differential Revision: https://phab.mercurial-scm.org/D3344

hgweb: these strings should be sysstrs, not bytes Differential Revision: https://phab.mercurial-scm.org/D3343

tests: port inline extensions in test-http.t to Python 3 Differential Revision: https://phab.mercurial-scm.org/D3342

patch: implement a new worddiff algorithm The previous worddiff algorithm has many problems. The major problem is it does a "similarity check" that selects a subset of matched lines to do inline diffs. It is a bad idea because: - The "similarity check" is non-obvious to users. For example, a simple change from "long long x" to "int64_t x" will fail the similarity check and won't be diff-ed as expected. - Selecting "lines" to diff won't work as people expect if there are line wrapping changes. - It has a sad time complexity if lines do not match, could be O(N^2)-ish. There are other problems in implementation details. - Lines can match across distant hunks (if the next hunk does not have "-" lines). - "difflib" is slow. The solution would be removing the "similarity check", and just diff all words in a same hunk. So no content will be missed and everything will be diff-ed as expected. This is similar to what code review tool like Phabricator does. This diff implements the word diff algorithm as described above. It also avoids difflib to be faster. Note about colors: To be consistent, "changed inserted" parts and "purely insertion blocks" should have a same color, since they do not exist in the previous version. Instead of highlighting differences, this patch chooses to dim common parts. This is also more consistent with Phabricator or GitHub webpage. That said, the labels are defined in a way that people can still highlight changed parts and leave purely inserted/deleted hunks use the "non-highlighted" color. As one example, running: hg log -pr df50b87d8f736aff8dc281f816bddcd6f306930c mercurial/commands.py \ --config experimental.worddiff=1 --color=debug --config diff.unified=0 The previous algorithm outputs: [diff.file_a|--- a/mercurial/commands.py Fri Mar 09 15:53:41 2018 +0100] [diff.file_b|+++ b/mercurial/commands.py Sat Mar 10 12:33:19 2018 +0530] [diff.hunk|@@ -2039,1 +2039,4 @@] [diff.deleted|-][diff.deleted.highlight|@command('^forget',][diff.deleted| ][diff.deleted.highlight|walkopts,][diff.deleted| _('[OPTION]... FILE...'), inferrepo=True)] [diff.inserted|+@command(] [diff.inserted|+ '^forget',] [diff.inserted|+ walkopts + dryrunopts,] [diff.inserted|+ ][diff.inserted.highlight| ][diff.inserted| _('[OPTION]... FILE...'), inferrepo=True)] [diff.hunk|@@ -2074,1 +2077,3 @@] [diff.deleted|- rejected = cmdutil.forget(ui, repo, m, prefix="",][diff.deleted.highlight| explicitonly=False)[0]] [diff.inserted|+ dryrun = opts.get(r'dry_run')] [diff.inserted|+ rejected = cmdutil.forget(ui, repo, m, prefix="",] [diff.inserted|+ explicitonly=False, dryrun=dryrun)[0]] The new algorithm outputs: [diff.file_a|--- a/mercurial/commands.py Fri Mar 09 15:53:41 2018 +0100] [diff.file_b|+++ b/mercurial/commands.py Sat Mar 10 12:33:19 2018 +0530] [diff.hunk|@@ -2039,1 +2039,4 @@] [diff.deleted|-][diff.deleted.unchanged|@command(][diff.deleted.unchanged|'^forget',][diff.deleted.unchanged| ][diff.deleted.changed|walkopts][diff.deleted.unchanged|,][diff.deleted.changed| ][diff.deleted.unchanged|_('[OPTION]... FILE...'), inferrepo=True)] [diff.inserted|+][diff.inserted.unchanged|@command(] [diff.inserted|+][diff.inserted.changed| ][diff.inserted.unchanged|'^forget',] [diff.inserted|+][diff.inserted.changed| walkopts][diff.inserted.unchanged| ][diff.inserted.changed|+ dryrunopts][diff.inserted.unchanged|,] [diff.inserted|+][diff.inserted.changed| ][diff.inserted.unchanged|_('[OPTION]... FILE...'), inferrepo=True)] [diff.hunk|@@ -2074,1 +2077,3 @@] [diff.deleted|-][diff.deleted.unchanged| rejected = cmdutil.forget(ui, repo, m, prefix="",][diff.deleted.changed| ][diff.deleted.unchanged|explicitonly=False][diff.deleted.unchanged|)[0]] [diff.inserted|+][diff.inserted.changed| dryrun = opts.get(r'dry_run')] [diff.inserted|+][diff.inserted.unchanged| rejected = cmdutil.forget(ui, repo, m, prefix="",] [diff.inserted|+][diff.inserted.changed| ][diff.inserted.unchanged|explicitonly=False][diff.inserted.changed|, dryrun=dryrun][diff.inserted.unchanged|)[0]] Practically, when diffing a 8k line change, the time spent on worddiff reduces from 4 seconds to 0.14 seconds. Differential Revision: https://phab.mercurial-scm.org/D3212

patch: buffer lines for a same hunk Instead of yielding tokens directly, buffer them if they belong to a same hunk. This makes it easier for the upcoming new worddiff algorithm to only focus on the diff hunk, instead of having to worry about other contents. This breaks how the existing experimental worddiff algorithm works, so the algorithm was removed, and related tests are disabled for now. The next patch will add a new worddiff algorithm. Differential Revision: https://phab.mercurial-scm.org/D3211

patch: move yielding "\n" to the end of loop The original logic makes it harder to reason about - it yields the "\n" character belonging to the last line in the next loop iteration. The new code is in theory a little bit slower. But is more readable. It makes the following changes easier to read. Differential Revision: https://phab.mercurial-scm.org/D3210

context: clarify deprecation warning message I had one developer report that they couldn't find the message. This patch should make it clear where to find it. Differential Revision: https://phab.mercurial-scm.org/D3389

wireprotov2: add support for more response types This adds types to represent error and generator responses from server commands. Differential Revision: https://phab.mercurial-scm.org/D3388

wireprotov2: remove support for sending bytes response We recently declared that all responses must be CBOR. So remove support for sending a type that isn't CBOR data. Differential Revision: https://phab.mercurial-scm.org/D3387

wireprotov2: change behavior of error frame Now that we have a leading CBOR map in command response frames to indicate overall command result status, we don't need to use the error response frame to represent command errors. Instead, we can reserve it for protocol and server level errors. And for the special case of a command error that occurred after command response frames were emitted. The code for error handling still needs a ton of work. But we're slowly going in the right direction... Differential Revision: https://phab.mercurial-scm.org/D3386

wireprotov2: change command response protocol to include a leading map The error handling mechanism for the new wire protocol isn't very well-defined. This commit takes us a step in the right direction by introducing a leading CBOR map for command responses. This map will contain an overall result of the command. Currently, the map indicates whether the command was overall successful or if an error occurred. And if an error occurred, that error is present in the map. There is still a dedicated error frame. My intent is to use that for protocol-level errors and for errors that are encountered after the initial response frame has been sent. This will be clarified in a later commit. Differential Revision: https://phab.mercurial-scm.org/D3385

wireprotov2: change frame type and name for command response There was hole at frame type value 3. And the frame is better named as a command response. Differential Revision: https://phab.mercurial-scm.org/D3384

wireprotov2: change frame type value for command data When we dropped the dedicated command argument frame type, this left a hole in our frame type numbering. Let's start plugging that hole. The command data frame is now type value 2 instead of 3. There was limited test fallout because a) we do a good job of using the constants to refer to frame types b) not many tests are sending command data frames. Bumping the media type will be performed in a later commit, once all type value adjustment has been performed. Differential Revision: https://phab.mercurial-scm.org/D3383

wireprotov2: define response data as CBOR Previously, response data was defined as a stream of bytes. We had the option to declare it as CBOR using a frame flag. We've converged all wire protocol commands exposed on version 2 to CBOR. I think consistency is important. The overhead to encoding things with CBOR is minimal. Even a very large bytestring can be efficiently encoded using an indefinite length bytestring. Now, there are limitations with consumers not being able to efficiently stream large CBOR values. But these feel like solvable problems. This commit removes the "is CBOR" frame flag from command response frames and defines the frame as always consisting of a stream of CBOR values. The framing protocol media type has been bumped to reflect this BC change. Differential Revision: https://phab.mercurial-scm.org/D3382

wireprotov2: decode responses to their expected types Callers of established wire protocol commands expect the response from that command to be decoded into a data structure. It's not very useful if callers get back a stream of bytes and don't know how they should be interpreted - especially since that stream of bytes varies by wire protocol and even the transport within that protocol version. This commit establishes decoding functions for various command responses so callers of those commands get the response type they expect. In theory, this should make the version 2 HTTP peer usable for various operations. But I haven't tested to confirm. Differential Revision: https://phab.mercurial-scm.org/D3381

wireprotov2: establish a type for representing command response It will be desirable to have a higher-level type for representing command responses. This will allow us to do nicer things. For now, the instance encapsulates existing logic. It is still a bit primitive. But we're slowly making things better. Version 1 protocols have a wrapping layer that decodes the raw string data into a data structure and that data structure is sent to the future. Version 2 doesn't yet have this layer and the future is receiving the raw wire response. Hence why debugcommands needed to be taught about the response type. Differential Revision: https://phab.mercurial-scm.org/D3380

wireprotov2: move response handling out of httppeer And fix some bugs while we're here. The code for processing response data from the unified framing protocol is mostly peer agnostic. The peer-specific bits are the configuration of the client reactor and how I/O is performed. I initially implemented things in httppeer for expediency. This commit establishes a module for holding the peer API level code for the framing based protocol. Inside this module we have a class to help coordinate higher-level activities, such as managing response object. The client handler bits could be rolled into clientreactor. However, I want clientreactor to be sans I/O and I want it to only be concerned with protocol-level details, not higher-level concepts like how protocol events are converted into peer API concepts. I want clientreactor to receive a frame and then tell the caller what should probably be done about it. If we start putting things like future resolution into clientreactor, we'll constrain how the protocol can be used (e.g. by requiring futures). The new code is loosely based on what was in httppeer before. I changed things a bit around response handling. We now buffer the entire response "body" and then handle it as one atomic unit. This fixed a bug around decoding CBOR data that spanned multiple frames. I also fixed an off-by-one bug where we failed to read a single byte CBOR value at the end of the stream. That's why tests have changed. The new state of httppeer is much cleaner. It is largely agnostic about framing protocol implementation details. That's how it should be: the framing protocol is designed to be largely transport agnostic. We want peers merely putting bytes on the wire and telling the framing protocol where to read response data from. There's still a bit of work to be done here, especially for representing responses. But at least we're a step closer to having a higher-level peer interface that can be plugged into the SSH peer someday. I initially added this class to wireprotoframing. However, we'll eventually need version 2 specific functions to convert CBOR responses into data structures expected by the code calling commands. This needs to live somewhere. Since that code would be shared across peers, we need a common module. We have wireprotov1peer for the equivalent version 1 code. So I decided to establish wireprotov2peer. Differential Revision: https://phab.mercurial-scm.org/D3379

debugcommands: ability to suppress logging of handshake The tests for calling wire protocol commands were getting quite verbose because they included the results of the capabilities request. Furthermore, it was annoying to have to update several tests every time the capabilities response changed. The only tests that really care about the low-level details of the capabilities requests are those testing the protocol handshake. And those are mostly not instantiating peer instances or are contained to limited files. This commit adds an option to `hg debugwireproto` to suppress logging of the handshake. The shell helper function to perform HTTP tests has been updated to use this by default. Lots of excessive test output has gone away. Differential Revision: https://phab.mercurial-scm.org/D3378

hg: pass command intents to repo/peer creation (API) The previous commit introduced a mechanism to declare command intents. This commit changes the repository and peer instantiation mechanism so the intents are passed down to each repository and peer type so they can do with them whatever they please. Currently, nobody does anything with any intent. Differential Revision: https://phab.mercurial-scm.org/D3377

registrar: replace "cmdtype" with an intent-based mechanism (API) Commands perform varied actions and repositories vary in their capabilities. Historically, the .hg/requires file has been used to lock out clients lacking a requirement. But this is a very heavy-handed approach and is typically reserved for cases where the on-disk storage format changes and we want to prevent incompatible clients from operating on a repo. Outside of the .hg/requires file, we tend to deal with things like optional, extension-provided features via checking at call sites. We'll either have checks in core or extensions will monkeypatch functions in core disabling incompatible features, enabling new features, etc. Things are somewhat tolerable today. But once we introduce alternate storage backends with varying support for repository features and vastly different modes of behavior, the current model will quickly grow unwieldy. For example, the implementation of the "simple store" required a lot of hacks to deal with stripping and verify because various parts of core assume things are implemented a certain way. Partial clone will require new ways of modeling file data retrieval, because we can no longer assume that all file data is already local. In this new world, some commands might not make any sense for certain types of repositories. What we need is a mechanism to affect the construction of repository (and eventually peer) instances so the requirements/capabilities needed for the current operation can be taken into account. "Current operation" can almost certainly be defined by a command. So it makes sense for commands to declare their intended actions. This commit introduces the "intents" concept on the command registrar. "intents" captures a set of strings that declare actions that are anticipated to be taken, requirements the repository must possess, etc. These intents will be passed into hg.repo(), which will pass them into localrepository, where they can be used to influence the object being created. Some use cases for this include: * For read-only intents, constructing a repository object that doesn't expose methods that can mutate the repository. Its VFS instances don't even allow opening a file with write access. * For read-only intents, constructing a repository object without cache invalidation logic. If the repo never changes during its lifetime, nothing ever needs to be invalidated and we don't need to do expensive things like verify the changelog's hidden revisions state is accurate every time we access repo.changelog. * We can automatically hide commands from `hg help` when the current repository doesn't provide that command. For example, an alternate storage backend may not support `hg commit`, so we can hide that command or anything else that would perform local commits. We already kind of had an "intents" mechanism on the registrar in the form of "cmdtype." However, it was never used. And it was limited to a single value. We really need something that supports multiple intents. And because intents may be defined by extensions and at this point are advisory, I think it is best to define them in a set rather than as separate arguments/attributes on the command. Differential Revision: https://phab.mercurial-scm.org/D3376

cleanup: polyfill assertRaisesRegex so we can avoid assertRaisesRegexp The latter is deprecated on Python 3.7 and causes our tests to fail due to the warning. Differential Revision: https://phab.mercurial-scm.org/D3375

tests: add b prefixes to test-hg-parseurl.py Now passes on Python 3. Differential Revision: https://phab.mercurial-scm.org/D3374

tests: port test-hg-parseurl.py to unittest Differential Revision: https://phab.mercurial-scm.org/D3373

hgwebdir: un-bytes the env dict before re-parsing env Not the most elegant, but it restores test-subrepo-deep-nested-change.t to passing on Python 3. Differential Revision: https://phab.mercurial-scm.org/D3367

cborutil: implement support for streaming encoding, bytestring decoding The vendored cbor2 package is... a bit disappointing. On the encoding side, it insists that you pass it something with a write() to send data to. That means if you want to emit data to a generator, you have to construct an e.g. io.BytesIO(), write() to it, then get the data back out. There can be non-trivial overhead involved. The encoder also doesn't support indefinite types - bytestrings, arrays, and maps that don't have a known length. Again, this is really unfortunate because it requires you to buffer the entire source and destination in memory to encode large things. On the decoding side, it supports reading indefinite length types. But it buffers them completely before returning. More sadness. This commit implements "streaming" encoders for various CBOR types. Encoding emits a generator of hunks. So you can efficiently stream encoded data elsewhere. It also implements support for emitting indefinite length bytestrings, arrays, and maps. On the decoding side, we only implement support for decoding an indefinite length bytestring from a file object. It will emit a generator of raw chunks from the source. I didn't want to reinvent so many wheels. But profiling the wire protocol revealed that the overhead of constructing io.BytesIO() instances to temporarily hold results has a non-trivial overhead. We're talking >15% of execution time for operations like "transfer the fulltexts of all files in a revision." So I can justify this effort. Fortunately, CBOR is a relatively straightforward format. And we have a reference implementation in the repo we can test against. Differential Revision: https://phab.mercurial-scm.org/D3303

configitems: register server.zstdlevel Somehow, I managed to trigger a devel-warn running `hg serve` outside the test suite on one of the repos generated by running test-lfs-serve-access.t --keep. (I'm not hitting it now after doing a `make local`.) The only reference to this in all of the history is the help text added in e75463e3179f, and the translations. (It looks like the string is built dynamically with '%slevel', which is probably how this was missed. I wonder if this isn't getting routed to error.log for some reason.) In any event, server.zliblevel is registered, and that's required to pick up the documented default.

scmutil: make shortesthexnodeidprefix() take a full binary nodeid The shortest() template function depended on the behavior of revlog._partialmatch() for these types of inputs: * non-hex strings * ambiguous strings * too long strings revlog._partialmatch() seems to return the input unchanged in these cases, but we shouldn't depend on such a low-level function to match the behavior we want in the user-facing template function. Instead, let's handle these cases in the template function and always pass a binary nodeid to _partialmatch(). Differential Revision: https://phab.mercurial-scm.org/D3371

scmutil: make shortesthexnodeidprefix() use unfiltered repo Both callers were doing this, and resolvehexnodeidprefix() was also working on the unfiltered repo, so it makes more sense to have it all in one place. Differential Revision: https://phab.mercurial-scm.org/D3313

templates: adjust white space amount in the output of {whyunstable} There used to be 2 spaces between divergent nodes (when not using custom template for divergentnodes) because divergentnodes is a hybrid list, which means it gets ' '.join()ed, but formatnode() already had a space. Now it doesn't, which requires extra effort in writing custom templates for whyunstable, but at least it looks correctly by default. Test output needs to be sorted for stability.

tests: split long templates that use {whyunstable} and put them in hgrc

tests: make custom templates that use {whyunstable} terser These templates demonstrate that {whyunstable} is fully template-friendly, but they don't need to produce such long lines of output.

tests: hg log shouldn't need --hidden to show whyunstable template keyword

makefile: add Ubuntu Bionic docker targets (.deb and ppa) Bionic Beaver is an LTS release, supported until 2023-04.

makefile: reformat .PHONY targets The list of make targets is likely to evolve over time. This reformats streamlines future patches.

makefile: mkdir is not needed on templatized docker builds This imitates e63dfbbdbd07 and is a small addition to 231690dba9b4 and 5c1283713293

hgweb: convert _siblings to a factory function of mappinggenerator IIUC, only reason it was a class is to make the generator restartable, which is now served by the mappinggenerator.

hgweb: extract a generator function of _siblings class _siblings will be converted to a plain function.

hgweb: wrap {changenav} and {nav} with mappinglist

hgweb: make revnav.gen() simply build a list of mappings by one pass There was actually no lazy stuff.

hgweb: lift {sessionvars} to a wrapped type Since a sessionvars object is updated in-place, we can't simply wrap it by mappinglist.

hgweb: make sessionvars class less dense

hgweb: prefix private variables of sessionvars with '_'

lfs: update the HTTP status codes in error cases I'm not bothering with validating PUT requests (for now), because the spec doesn't explicitly call out a Content-Type (though the example transcript does use the sensible 'application/octet-stream').

lfs: gracefully handle aborts on the server when corrupt blobs are detected The aborts weren't killing the server, but this seems cleaner. I'm not sure if it matters to handle the remaining IOError in the test like this, for consistency. The error code still feels wrong (especially if the client is trying to download a corrupt blob) but I don't see anything better in the RFCs, and this is already used elsewhere because the Batch API spec specifically mentioned this as a "Validation Error".

lfs: fix the inferred remote store path when using a --prefix This wasn't appending the '.git/info/lfs' path in this case.

lfs: log information about Internal Server Errors reported in the Batch API Reporting a 500 and then not leaving any traces on the server seems like a receipe for frustration. There's similar log writing in hgweb.server.do_POST(). That doesn't write directly to the wsgi.errors object, so it doesn't seem worth trying to refactor. It does seem like earlier stack frames are missing for some reason.

test-lfs: add tests to force server error path coverage The tests are somewhat fragile in that the extension that forces the errors is counting how many times some of the functions are being called, so it depends heavily on the content of the repo. Maybe we can do something clever like load an extension on the client, and have it send over instructions in the HTTP header how to fail. (I'm trying to avoid killing and restarting the server, because Windows seems to have issues with doing that a lot.) But I'd rather fix issues than polish tests before the freeze.

keepalive: add ** overlooked in 83250442dc81 Caught by Yuya in D3326. Differential Revision: https://phab.mercurial-scm.org/D3372

test-check-commit: don't run hg per commit We aren't stress testing CPU. $ time ./run-tests.py -l test-check-commit.t --timeout 600 (orig) 162.59s user 17.98s system 101% cpu 2:58.55 total (new) 5.85s user 0.99s system 98% cpu 6.939 total

templates: make divergentnodes in whyunstable keyword be a hybrid list

templates: add whyunstable template keyword

commands: drop spurious r'' on dry_run in forget Fixes test-add.t on Python 3. Differential Revision: https://phab.mercurial-scm.org/D3365

tests: port inline extension in test-http-bundle1.t to py3 # skip-blame just b prefixes Differential Revision: https://phab.mercurial-scm.org/D3366

tests: add all missing b prefixes in reactor tests Both of these tests now pass on Python 3. # skip-blame just b prefixes. So many b prefixes. Differential Revision: https://phab.mercurial-scm.org/D3369

stringutil: ast.literal_eval needs a unicode on py3 Fortunately for us, this is really only used internally, so we can be lazy about the encoding here. test-wireproto-framing.py now passes on Python 3. Differential Revision: https://phab.mercurial-scm.org/D3368

scmutil: introduce shortesthexnodeidprefix() We have scmutil.resolvehexnodeidprefix() for resolving a prefix to a full nodeid, so it makes sense to have the inverse method next to it. For now it just delegates to changelog.shortest(), but it will soon also make sure it's called on the unfiltered repo, to match resolvehexnodeidprefix(). Note that the change in show.py also makes it so the conversion from revnum to nodeid is done on the filtered repo, but that should be inconsequential since the revs are all from the filtered repo anyway. Differential Revision: https://phab.mercurial-scm.org/D3370

scmutil: use resolvehexnodeidprefix() from revsymbol() I should have copied this from changectx.__init__ into in 35b34202dd3b (context: handle partial nodeids in revsymbol(), 2018-04-08). Differential Revision: https://phab.mercurial-scm.org/D3310

scmutil: rename resolvepartialhexnodeid() to resolvehexnodeidprefix() I'm going to wrap revlog.shortest() in scmutil. I plan on calling it shortesthexnodeidprefix(). resolvehexnodeidprefix() matches that better. Also, "prefix" carries more information than "partial". Differential Revision: https://phab.mercurial-scm.org/D3309

scmutil: document that isrevsymbol() raises on ambiguous node prefix Differential Revision: https://phab.mercurial-scm.org/D3308

context: set stack level for deprecation warning This patch makes the deprecation warning print the caller of repo.__getitem__. That's not quite correct, since there could also be other callers of changectx.__init__, but it seems like the most helpful stack level in practice. Differential Revision: https://phab.mercurial-scm.org/D3307

util: set correct stack level on deprecation warnings Without this patch, you'll get something like this: <path>/mercurial/util.py:3784: DeprecationWarning: 'util.hgexecutable' is deprecated, use 'utils.procutil.hgexecutable' (but on one line) Differential Revision: https://phab.mercurial-scm.org/D3331

revset: pass in lookup function instead of repo (API) And document that it's only for legacy lookup. If we have a repo, we're likely to do more things where that shouldn't be done.

revset: drop support for posttreebuilthook() (API) AFAIK, the only user was the directaccess extension, which is in core now.

py3: make values bytes before passing into server.runservice() The values of opts dict still needed to be converted to bytes. Differential Revision: https://phab.mercurial-scm.org/D3330

py3: two more passing tests Differential Revision: https://phab.mercurial-scm.org/D3338

localrepo: add some overlooked strkwargs love for py3 Differential Revision: https://phab.mercurial-scm.org/D3337

py3: paper over differences in future exception handling It looks like Python 3's futures library lacks set_exception_info entirely. We'll just give up and use set_exception in that case. # no-check-commit because the underbar naming is just saner here Differential Revision: https://phab.mercurial-scm.org/D3336

py3: add b'' prefixes to make values bytes # skip-blame beacuse just b'' prefixes Differential Revision: https://phab.mercurial-scm.org/D3329

py3: use b"%d" instead of str() to convert int to bytes While I was here, I added 'and None' to suppress return values of .write() calls. Differential Revision: https://phab.mercurial-scm.org/D3328

py3: add b'' prefixes to make values bytes # skip-blame because just b'' prefixes Differential Revision: https://phab.mercurial-scm.org/D3327

py3: use str variables to check keys in request header The values in header are of str type. Differential Revision: https://phab.mercurial-scm.org/D3326

py3: make sure curses.tigetstr() first argument is a str Differential Revision: https://phab.mercurial-scm.org/D3325

py3: use stringutil.forcebytestr() instead of str() We need to convert errors to bytes using stringutil.forcebytestr() Differential Revision: https://phab.mercurial-scm.org/D3324

py3: iterate over a copy of dict while changing it Differential Revision: https://phab.mercurial-scm.org/D3323

py3: use urllib.parse.unquote_plus instead of urllib.unquote_plus The later is not present in Python 3. Differential Revision: https://phab.mercurial-scm.org/D3322

py3: add b'' prefixes to tests/test-status-inprocess.py # skip-blame because just b'' prefixes Differential Revision: https://phab.mercurial-scm.org/D3321

lock: don't use 'file' as a variable name Differential Revision: https://phab.mercurial-scm.org/D3320

py3: use b"%d" instead of str() to convert integers to bytes Differential Revision: https://phab.mercurial-scm.org/D3319

wireproto: expose repository formats via capabilities Servers need to expose their set of repository storage requirements in order to facilitate streaming clones (clients need to know if they are capable of reading the raw storage files that the server exposes). Differential Revision: https://phab.mercurial-scm.org/D3335

wireprotoframing: record when new stream is encountered Without this, we choke after receiving the 2nd frame in a stream. Not sure how we made it this far without finding this bug. Differential Revision: https://phab.mercurial-scm.org/D3334

wireprotoframing: use value passed into function Oops. Differential Revision: https://phab.mercurial-scm.org/D3333

httppeer: handle error response from client reactor With this in place, we're now seeing useful errors when running tests with the new wire protocol enabled! Differential Revision: https://phab.mercurial-scm.org/D3332

wireproto: add media type to version 2 capabilities response This is useful to advertise because servers reject unsupported media types. A client may wish to speak multiple media types and choose the one the server supports. I doubt we'll ever use multiple media types or negotiation in core. But during the course of developing this protocol, I may end up making extensions that backport and forward port protocol support as needed to support Mercurial deploys in the wild. e.g. I may deploy support for an older protocol on a server so old clients can continue using it. It's worth pursuing changing the SSH protocol's upgrade mechanism to support multiple media types as well... Differential Revision: https://phab.mercurial-scm.org/D3299

debugcommands: use command executor for invoking commands Now that all peers support the command executor interface, we can use it in place of peer._call() (which isn't part of the public API). Tests changed because we are now returning the decoded response instead of the raw bytes. The raw bytes over the wire are already logged. So we're not losing any test coverage. In fact, we're adding test coverage because we're testing decoding of those command responses as well. Differential Revision: https://phab.mercurial-scm.org/D3298

httppeer: implement command executor for version 2 peer Now that we have a new API for issuing commands which is compatible with wire protocol version 2, we can start using it with wire protocol version 2. This commit replaces our hacky implementation of _call() with something a bit more robust based on the new command executor interface. We now have proper support for issuing multiple commands per HTTP request. Each HTTP request maintains its own client reactor. The implementation is similar to the one in the legacy wire protocol. We use a ThreadPoolExecutor for spinning up a thread to read the HTTP response in the background. This allows responses to resolve in any order. While not implemented on the server yet, a client could use concurrent.futures.as_completed() with a collection of futures and handle responses as they arrive from the server. The return value from issued commands is still a simple list of raw or decoded CBOR data. This is still super hacky. We will want a rich data type for representing command responses. But at least this commit gets us one step closer to a proper peer implementation. Differential Revision: https://phab.mercurial-scm.org/D3297

repository: remove ipeercommands from ipeerbase AFAICT all callers in core have moved to the commandexecutor interface for invoking wire protocol commands. Or at least they aren't using the named methods on ipeercommands to invoke them. This means we can drop ipeercommands from the ipeerbase interface. As far as interface based programming goes, it is now illegal to call an ipeercommands method for issuing wire protocol commands. However, the methods are still there, so they will still work. At some point we will want to break that API... Differential Revision: https://phab.mercurial-scm.org/D3318

wireproto: properly call clonebundles command We should not be using _call() to make wire protocol calls because it isn't part of the peer API. But clonebundles wasn't part of the supported commands in the peer API! So this commit defines that command in the commands interface, implements it, and teaches the one caller in core to call it using the command executor interface. Differential Revision: https://phab.mercurial-scm.org/D3317

exchange: use command executor for getbundle The code consuming the bundle has been moved to inside the context manager, as that is supposed to be part of the API. (Although it doesn't matter for version 1 peers.) Differential Revision: https://phab.mercurial-scm.org/D3316

exchange: use command executor for pushkey Differential Revision: https://phab.mercurial-scm.org/D3315

wireproto: use command executor for unbundle This also required unifying the name of the argument because the new API always passes arguments by keyword. I decided to change implementations to "bundle" instead of the interface to "cg" because "bundle" is more appropriate in a modern world. Differential Revision: https://phab.mercurial-scm.org/D3314

debugcommands: perform handshake when obtaining httpv2 peer If we obtain an httpv2peer directly, the instance doesn't have an API descriptor and therefore doesn't know about the remote's commands, feature support, etc. This doesn't matter now. But when we implement the peer so it consults the API descriptor as part of sending commands, it will. So we change the logic for obtaining an http version 2 peer to go through makepeer() so the peer will perform the handshake and pass the API descriptor to the httpv2peer instance. Tests changed because we now perform a ?cmd=capabilities when obtaining version 2 peers. The Content-Length header is globbed because compression info will lack zstandard for pure builds. Differential Revision: https://phab.mercurial-scm.org/D3296

wireproto: rename HTTPV2 so it less like HTTP/2 Per review suggestion on D3230 from Augie. Differential Revision: https://phab.mercurial-scm.org/D3295

bundlerepo: use command executor for wire protocol commands Differential Revision: https://phab.mercurial-scm.org/D3294

bundlerepo: rename "other" to "peer" Differential Revision: https://phab.mercurial-scm.org/D3293

bookmarks: use command executor for wire protocol commands And change the name of a variable to reflect that is is a peer. Differential Revision: https://phab.mercurial-scm.org/D3292

hg: use command executor for wire protocol commands As part of this, I realized that some uses of lookup in a loop could be converted to use a batch request. But I didn't change behavior and left in a todo to track potentially changing this. Differential Revision: https://phab.mercurial-scm.org/D3291

logexchange: use command executor for wire protocol commands Differential Revision: https://phab.mercurial-scm.org/D3290

streamclone: use command executor for wire protocol commands Differential Revision: https://phab.mercurial-scm.org/D3289

discovery: use command executor interface We're trying to port all wire protocol code to use the new interface so we can implement wire protocol version 2 clients. Differential Revision: https://phab.mercurial-scm.org/D3288

discovery: don't redundantly call branchmap We were calling the remote command twice without mutation the remote in between. Derp. Differential Revision: https://phab.mercurial-scm.org/D3287

wireproto: convert legacy commands to command executor Calls to the legacy commands "changegroup" and "changegroupsubset" have been ported to the new command executor interface. Because we always pass arguments by name and not position, some inconsistent names throughout the code base have been unified. As part of this change, we no longer had any remaining callers of the legacy command methods {between, branches, changegroup, changegroupsubset}. So, these interfaces/methods have been dropped from peer interfaces. We still have an interface declaring these methods. But that interface is implemented on the concrete peer types and isn't part of the generic peer interface. (The implementations of the command executor continue to call these methods.) The ultimate goal is to remove the per-command methods from the generic peer interface: the only interface-conforming way to call a command will be with the new executor API. At some point, we may want to move the methods outside of the peer classes and change the executor implementations to not call methods directly on a peer instance. Differential Revision: https://phab.mercurial-scm.org/D3273

treediscovery: switch to command executor interface We now have a new interface for requesting that commands run. Switch to it. Differential Revision: https://phab.mercurial-scm.org/D3272

wireproto: remove iterbatch() from peer interface (API) Good riddance. Some tests have been ported to the new API. This probably should have been done in earlier commits. But duplicating the test coverage would have been difficult. It was easier this way. .. api:: The wire protocol peer's ``iterbatch()`` for bulk executing commands has been remove.d Use ``peer.commandexecutor()`` instead. Differential Revision: https://phab.mercurial-scm.org/D3271

largefiles: use command executor for batch operation This is the only other user of iterbatch() in core. Tests changed because the new command executor is smart enough to not send a "batch" command over the wire if only 1 command was requested. There is still coverage for the "batch" command in this test though. Differential Revision: https://phab.mercurial-scm.org/D3270

wireproto: implement batching on peer executor interface This is a bit more complicated than non-batch requests because we need to buffer sends until the last request arrives *and* we need to support resolving futures as data arrives from the remote. In a classical concurrent.futures executor model, the future "starts" as soon as it is submitted. However, we have nothing to start until the last command is submitted. If we did nothing, calling result() would deadlock, since the future hasn't "started." So in the case where we queue the command, we return a special future type whose result() will trigger sendcommands(). This eliminates the deadlock potential. It also serves as a check against callers who may be calling result() prematurely, as it will prevent any subsequent callcommands() from working. This behavior is slightly annoying and a bit restrictive. But it's the world that half duplex connections forces on us. In order to support streaming responses, we were previously using a generator. But with a futures-based API, we're using futures and not generators. So in order to get streaming, we need a background thread to read data from the server. The approach taken in this patch is to leverage the ThreadPoolExecutor from concurrent.futures for managing a background thread. We create an executor and future that resolves when all response data is processed (or an error occurs). When exiting the context manager, we wait on that background reading before returning. I was hoping we could manually spin up a threading.Thread and this would be simple. But I ran into a few deadlocks when implementing. After looking at the source code to concurrent.futures, I figured it would just be easier to use a ThreadPoolExecutor than implement all the code needed to manually manage a thread. To prove this works, a use of the batch API in discovery has been updated. Differential Revision: https://phab.mercurial-scm.org/D3269

wireproto: implement command executor interface for version 1 peers Now that we've defined our new interface for issuing commands, let's implement it. We add the interface to the base peer interface. This means all peer types must implement it. The only peer types that we have are the local peer in localrepo and a shared wire peer for version 1 of the wire protocol. The local peer implementation is pretty straightforward. We don't do anything fancy and just return a resolved future with the result of a method call. This is similar to what localiterbatcher does. The wire protocol version 1 implementation is a bit more complicated and is a more robust implementation. The wire executor queues commands by default. And because the new executor interface always allows multiple commands but not all version 1 commands are @batchable, it has to check that the requested commands are batchable if multiple commands are being requested. The wire executor currently only supports executing a single command. This is for simplicity reasons. Support for multiple commands will be added in a separate commit. To prove the new interface works, a call to the "known" command during discovery has been updated to use the new API. It's worth noting that both implementations require a method having the command name to exist on the peer. There is at least one caller in core that don't have a method calls peer._call() directly. We may need to shore up the requirements later... Differential Revision: https://phab.mercurial-scm.org/D3268

repository: define new interface for running commands Today, the peer interface exposes methods for each command that can be executed. In addition, there is an iterbatch() API that allows commands to be issued in batches and provides an iterator over the results. This is a glorified wrapper around the "batch" wire command. Wire protocol version 2 supports nicer things (such as batching any command and out-of-order replies). It will require a more flexible API for executing commands. This commit introduces a new peer interface for making command requests. In the new world, you can't simply call a method on the peer to execute a command: you need to obtain an object to be used for executing commands. That object can be used to issue a single command or it can batch multiple requests. In the case of full duplex peers, the command may even be sent out over the wire immediately. There are no per-command methods. Instead, there is a generic method to call a command. The implementation can then perform domain specific processing for specific commands. This includes passing data via a specially named argument. Arguments are also passed as a dictionary instead of using **kwargs. While **kwargs is nicer to use, we've historically gotten into trouble using it because there will inevitably be a conflict between the name of an argument to a wire protocol command and an argument we want to pass into a function. Instead of a command returning a value, it returns a future which will resolve to a value. This opens the door for out-of-order response handling and concurrent response handling in the version 2 protocol. Differential Revision: https://phab.mercurial-scm.org/D3267

pycompat: export a handle on concurrent.futures On Python 3, we use the built-in version in the standard library. Else we use our vendored backport. Differential Revision: https://phab.mercurial-scm.org/D3266

setup: add packages for concurrent.futures We conceivably don't need to distribute this package on Python 3 since we will use the version in the standard library. However, we want installs to be usable of multiple versions of Python. So it is best to always have it. Differential Revision: https://phab.mercurial-scm.org/D3265

futures: switch to absolute and relative imports This makes the package conform with our importing policy, silencing a number of warnings. It also makes the package usable when it isn't named "concurrent.futures." Differential Revision: https://phab.mercurial-scm.org/D3264