scmutil: teach the file prefetch hook to handle multiple commits The remainder of the commands that need prefetch deal with multiple revisions. I initially coded this as a separate hook, but then it needed a list of files to handle `diff` and `grep`, so it didn't seem worth keeping them separate. Not every matcher will emit bad file messages (some are built from a list of files that are known to exist). But it seems better to filter this in one place than to push this on either each caller or each hook implementation.

tests: arrange for a server in wireproto-command-capabilities.t to be killed The stray servers were piling up after the test harness exited. On Windows, this means the *.pyd files can't be rebuilt, which is why the build warning count dropped to 1 recently.

revset: skip legacy lookup for revspec wrapped in 'revset(...)' Currently, multiple labels can take forms that can be confused with revset (eg: "rev(0)" is a valid tag). Since we look up for tags before evaluating revset, this means a tag can shadow a valid revset at any time. We now enforce the strict revset parsing when wrapped with 'revset(...)'. For now, This only work on a whole revspec (but can be used within the revset without effect). This might change in the future if we improve the implementation. The feature is undocumented for now, keeping it in the experimental namespace. In case a better approach to achieve the same goal is found. The syntax looks like a revset but is not implemented as such for now. Since the goal is to avoid some preprocessing that happens before revset parsing, we cannot simply implement it as a revset predicate. There was other approaches discussed over the mailing-list but they were less convincing. Having a configuration flag to disable legacy lookup have been considered but discarded. There are too many common uses of ambiguous identifier (eg: '+', '-' or '..') to have the legacy lookup mechanism turned off. In addition, the approach can control the parsing of each revset, making it more flexible. For example, a revset used as the value of an existing configuration option (eg: pushrev) could enforce its resolution as a revset (by using the prefix) while user inputs would still use the legacy lookup. In addition of offering a way to unambiguously input a revset, this prefix allow skipping the name lookup providing a significant speedup in some case.

revset: add more test to show current behaviors with label looking like revset There are very few constraints on what character can be put into tags and other labels. We add more tests showing some of extreme cases that user can currently use.

commands: use command executor interface Differential Revision: https://phab.mercurial-scm.org/D3391

exchange: use command executor interface for calling listkeys So the requests are compatible with version 2 peers. Differential Revision: https://phab.mercurial-scm.org/D3390

forget: add --confirm option Also added confirmopts in cmdutil.py Differential Revision: https://phab.mercurial-scm.org/D2934

largefiles: opts appears to already be bytes in this instance test-largefiles.t now passes. Differential Revision: https://phab.mercurial-scm.org/D3364

tests: port inline extensions in test-hook.t to py3 This test *almost* passes now, but some import errors print very differently in ways that seem at least somewhat important. Differential Revision: https://phab.mercurial-scm.org/D3363

tests: manually print list in test-hook.t Changes the output a bit, but not in an important way. Differential Revision: https://phab.mercurial-scm.org/D3362

hook: also use pprint on lists for stable output on py2/3 Differential Revision: https://phab.mercurial-scm.org/D3361

hook: use stringutil.pprint instead of reinventing it Differential Revision: https://phab.mercurial-scm.org/D3360

stringutil: make b prefixes on string output optional I need this to preserve some behavior in hook.py. Differential Revision: https://phab.mercurial-scm.org/D3359

stringutil: teach pprint how to format None Differential Revision: https://phab.mercurial-scm.org/D3358

tests: update inline extensions in test-bundle2-exchange.t to py3 The test doesn't pass for what superficially look like good reasons. We'll need to come back to it later. # skip-blame because it's b prefixes and a couple of b'N' instead of str(N) Differential Revision: https://phab.mercurial-scm.org/D3357

wsgicgi: un-do some prior porting work that is now wrong The Python 3 WSGI behavior is that the environ dict should be full of unicodes. We previously tried Too Hard here, so we unwind that bit of porting. Also add some bytesurl() encodes on status and headers. test-clone-cgi.t now passes. Differential Revision: https://phab.mercurial-scm.org/D3356

hgweb_mod: inform hgweb class about paths actually being bytes Differential Revision: https://phab.mercurial-scm.org/D3355

hgweb: inform hgweb.hgweb() entrypoint that paths should be bytes Differential Revision: https://phab.mercurial-scm.org/D3354

tests: port inline cgi script in test-largefiles.t to python 3 Differential Revision: https://phab.mercurial-scm.org/D3353

lfcommands: use %d on known-int in format string Differential Revision: https://phab.mercurial-scm.org/D3352

py3: fix test-shelve.t on Python 3 We get a better error message out of the abort here, which is fine. Differential Revision: https://phab.mercurial-scm.org/D3351

tests: fix up a couple of minor bytes inconsistencies in run-tests.py Only in the automatic bisection code, so fortunately nothing major is amiss. Fixes test-run-tests.t under Python 3. Differential Revision: https://phab.mercurial-scm.org/D3350

tests: make sure test-run-tests.t actually runs run-tests.py under Python 3 I'm fairly certain it hasn't been until now. Mercifully, there doesn't appear to be any ninja breakage. Differential Revision: https://phab.mercurial-scm.org/D3349

py3: another three passing Differential Revision: https://phab.mercurial-scm.org/D3348

httppeer: work around API differences on urllib Request objects Since this is only a problem in httppeer, I'd rather keep this a local-to-the-module kludge rather than pile more on pycompat. We'll still find it easily to clean up later because it checks pycompat.ispy3. Differential Revision: https://phab.mercurial-scm.org/D3347

httppeer: no matter what Python 3 might think, http headers are bytes Differential Revision: https://phab.mercurial-scm.org/D3346

httppeer: fix debug prints to work on Python 3 Differential Revision: https://phab.mercurial-scm.org/D3345

url: some bytes/str cleanup where we interface with stdlib funcs Differential Revision: https://phab.mercurial-scm.org/D3344

hgweb: these strings should be sysstrs, not bytes Differential Revision: https://phab.mercurial-scm.org/D3343

tests: port inline extensions in test-http.t to Python 3 Differential Revision: https://phab.mercurial-scm.org/D3342

patch: implement a new worddiff algorithm The previous worddiff algorithm has many problems. The major problem is it does a "similarity check" that selects a subset of matched lines to do inline diffs. It is a bad idea because: - The "similarity check" is non-obvious to users. For example, a simple change from "long long x" to "int64_t x" will fail the similarity check and won't be diff-ed as expected. - Selecting "lines" to diff won't work as people expect if there are line wrapping changes. - It has a sad time complexity if lines do not match, could be O(N^2)-ish. There are other problems in implementation details. - Lines can match across distant hunks (if the next hunk does not have "-" lines). - "difflib" is slow. The solution would be removing the "similarity check", and just diff all words in a same hunk. So no content will be missed and everything will be diff-ed as expected. This is similar to what code review tool like Phabricator does. This diff implements the word diff algorithm as described above. It also avoids difflib to be faster. Note about colors: To be consistent, "changed inserted" parts and "purely insertion blocks" should have a same color, since they do not exist in the previous version. Instead of highlighting differences, this patch chooses to dim common parts. This is also more consistent with Phabricator or GitHub webpage. That said, the labels are defined in a way that people can still highlight changed parts and leave purely inserted/deleted hunks use the "non-highlighted" color. As one example, running: hg log -pr df50b87d8f736aff8dc281f816bddcd6f306930c mercurial/commands.py \ --config experimental.worddiff=1 --color=debug --config diff.unified=0 The previous algorithm outputs: [diff.file_a|--- a/mercurial/commands.py Fri Mar 09 15:53:41 2018 +0100] [diff.file_b|+++ b/mercurial/commands.py Sat Mar 10 12:33:19 2018 +0530] [diff.hunk|@@ -2039,1 +2039,4 @@] [diff.deleted|-][diff.deleted.highlight|@command('^forget',][diff.deleted| ][diff.deleted.highlight|walkopts,][diff.deleted| _('[OPTION]... FILE...'), inferrepo=True)] [diff.inserted|+@command(] [diff.inserted|+ '^forget',] [diff.inserted|+ walkopts + dryrunopts,] [diff.inserted|+ ][diff.inserted.highlight| ][diff.inserted| _('[OPTION]... FILE...'), inferrepo=True)] [diff.hunk|@@ -2074,1 +2077,3 @@] [diff.deleted|- rejected = cmdutil.forget(ui, repo, m, prefix="",][diff.deleted.highlight| explicitonly=False)[0]] [diff.inserted|+ dryrun = opts.get(r'dry_run')] [diff.inserted|+ rejected = cmdutil.forget(ui, repo, m, prefix="",] [diff.inserted|+ explicitonly=False, dryrun=dryrun)[0]] The new algorithm outputs: [diff.file_a|--- a/mercurial/commands.py Fri Mar 09 15:53:41 2018 +0100] [diff.file_b|+++ b/mercurial/commands.py Sat Mar 10 12:33:19 2018 +0530] [diff.hunk|@@ -2039,1 +2039,4 @@] [diff.deleted|-][diff.deleted.unchanged|@command(][diff.deleted.unchanged|'^forget',][diff.deleted.unchanged| ][diff.deleted.changed|walkopts][diff.deleted.unchanged|,][diff.deleted.changed| ][diff.deleted.unchanged|_('[OPTION]... FILE...'), inferrepo=True)] [diff.inserted|+][diff.inserted.unchanged|@command(] [diff.inserted|+][diff.inserted.changed| ][diff.inserted.unchanged|'^forget',] [diff.inserted|+][diff.inserted.changed| walkopts][diff.inserted.unchanged| ][diff.inserted.changed|+ dryrunopts][diff.inserted.unchanged|,] [diff.inserted|+][diff.inserted.changed| ][diff.inserted.unchanged|_('[OPTION]... FILE...'), inferrepo=True)] [diff.hunk|@@ -2074,1 +2077,3 @@] [diff.deleted|-][diff.deleted.unchanged| rejected = cmdutil.forget(ui, repo, m, prefix="",][diff.deleted.changed| ][diff.deleted.unchanged|explicitonly=False][diff.deleted.unchanged|)[0]] [diff.inserted|+][diff.inserted.changed| dryrun = opts.get(r'dry_run')] [diff.inserted|+][diff.inserted.unchanged| rejected = cmdutil.forget(ui, repo, m, prefix="",] [diff.inserted|+][diff.inserted.changed| ][diff.inserted.unchanged|explicitonly=False][diff.inserted.changed|, dryrun=dryrun][diff.inserted.unchanged|)[0]] Practically, when diffing a 8k line change, the time spent on worddiff reduces from 4 seconds to 0.14 seconds. Differential Revision: https://phab.mercurial-scm.org/D3212

patch: buffer lines for a same hunk Instead of yielding tokens directly, buffer them if they belong to a same hunk. This makes it easier for the upcoming new worddiff algorithm to only focus on the diff hunk, instead of having to worry about other contents. This breaks how the existing experimental worddiff algorithm works, so the algorithm was removed, and related tests are disabled for now. The next patch will add a new worddiff algorithm. Differential Revision: https://phab.mercurial-scm.org/D3211

patch: move yielding "\n" to the end of loop The original logic makes it harder to reason about - it yields the "\n" character belonging to the last line in the next loop iteration. The new code is in theory a little bit slower. But is more readable. It makes the following changes easier to read. Differential Revision: https://phab.mercurial-scm.org/D3210

context: clarify deprecation warning message I had one developer report that they couldn't find the message. This patch should make it clear where to find it. Differential Revision: https://phab.mercurial-scm.org/D3389

wireprotov2: add support for more response types This adds types to represent error and generator responses from server commands. Differential Revision: https://phab.mercurial-scm.org/D3388

wireprotov2: remove support for sending bytes response We recently declared that all responses must be CBOR. So remove support for sending a type that isn't CBOR data. Differential Revision: https://phab.mercurial-scm.org/D3387

wireprotov2: change behavior of error frame Now that we have a leading CBOR map in command response frames to indicate overall command result status, we don't need to use the error response frame to represent command errors. Instead, we can reserve it for protocol and server level errors. And for the special case of a command error that occurred after command response frames were emitted. The code for error handling still needs a ton of work. But we're slowly going in the right direction... Differential Revision: https://phab.mercurial-scm.org/D3386

wireprotov2: change command response protocol to include a leading map The error handling mechanism for the new wire protocol isn't very well-defined. This commit takes us a step in the right direction by introducing a leading CBOR map for command responses. This map will contain an overall result of the command. Currently, the map indicates whether the command was overall successful or if an error occurred. And if an error occurred, that error is present in the map. There is still a dedicated error frame. My intent is to use that for protocol-level errors and for errors that are encountered after the initial response frame has been sent. This will be clarified in a later commit. Differential Revision: https://phab.mercurial-scm.org/D3385

wireprotov2: change frame type and name for command response There was hole at frame type value 3. And the frame is better named as a command response. Differential Revision: https://phab.mercurial-scm.org/D3384

wireprotov2: change frame type value for command data When we dropped the dedicated command argument frame type, this left a hole in our frame type numbering. Let's start plugging that hole. The command data frame is now type value 2 instead of 3. There was limited test fallout because a) we do a good job of using the constants to refer to frame types b) not many tests are sending command data frames. Bumping the media type will be performed in a later commit, once all type value adjustment has been performed. Differential Revision: https://phab.mercurial-scm.org/D3383

wireprotov2: define response data as CBOR Previously, response data was defined as a stream of bytes. We had the option to declare it as CBOR using a frame flag. We've converged all wire protocol commands exposed on version 2 to CBOR. I think consistency is important. The overhead to encoding things with CBOR is minimal. Even a very large bytestring can be efficiently encoded using an indefinite length bytestring. Now, there are limitations with consumers not being able to efficiently stream large CBOR values. But these feel like solvable problems. This commit removes the "is CBOR" frame flag from command response frames and defines the frame as always consisting of a stream of CBOR values. The framing protocol media type has been bumped to reflect this BC change. Differential Revision: https://phab.mercurial-scm.org/D3382

wireprotov2: decode responses to their expected types Callers of established wire protocol commands expect the response from that command to be decoded into a data structure. It's not very useful if callers get back a stream of bytes and don't know how they should be interpreted - especially since that stream of bytes varies by wire protocol and even the transport within that protocol version. This commit establishes decoding functions for various command responses so callers of those commands get the response type they expect. In theory, this should make the version 2 HTTP peer usable for various operations. But I haven't tested to confirm. Differential Revision: https://phab.mercurial-scm.org/D3381

wireprotov2: establish a type for representing command response It will be desirable to have a higher-level type for representing command responses. This will allow us to do nicer things. For now, the instance encapsulates existing logic. It is still a bit primitive. But we're slowly making things better. Version 1 protocols have a wrapping layer that decodes the raw string data into a data structure and that data structure is sent to the future. Version 2 doesn't yet have this layer and the future is receiving the raw wire response. Hence why debugcommands needed to be taught about the response type. Differential Revision: https://phab.mercurial-scm.org/D3380

wireprotov2: move response handling out of httppeer And fix some bugs while we're here. The code for processing response data from the unified framing protocol is mostly peer agnostic. The peer-specific bits are the configuration of the client reactor and how I/O is performed. I initially implemented things in httppeer for expediency. This commit establishes a module for holding the peer API level code for the framing based protocol. Inside this module we have a class to help coordinate higher-level activities, such as managing response object. The client handler bits could be rolled into clientreactor. However, I want clientreactor to be sans I/O and I want it to only be concerned with protocol-level details, not higher-level concepts like how protocol events are converted into peer API concepts. I want clientreactor to receive a frame and then tell the caller what should probably be done about it. If we start putting things like future resolution into clientreactor, we'll constrain how the protocol can be used (e.g. by requiring futures). The new code is loosely based on what was in httppeer before. I changed things a bit around response handling. We now buffer the entire response "body" and then handle it as one atomic unit. This fixed a bug around decoding CBOR data that spanned multiple frames. I also fixed an off-by-one bug where we failed to read a single byte CBOR value at the end of the stream. That's why tests have changed. The new state of httppeer is much cleaner. It is largely agnostic about framing protocol implementation details. That's how it should be: the framing protocol is designed to be largely transport agnostic. We want peers merely putting bytes on the wire and telling the framing protocol where to read response data from. There's still a bit of work to be done here, especially for representing responses. But at least we're a step closer to having a higher-level peer interface that can be plugged into the SSH peer someday. I initially added this class to wireprotoframing. However, we'll eventually need version 2 specific functions to convert CBOR responses into data structures expected by the code calling commands. This needs to live somewhere. Since that code would be shared across peers, we need a common module. We have wireprotov1peer for the equivalent version 1 code. So I decided to establish wireprotov2peer. Differential Revision: https://phab.mercurial-scm.org/D3379

debugcommands: ability to suppress logging of handshake The tests for calling wire protocol commands were getting quite verbose because they included the results of the capabilities request. Furthermore, it was annoying to have to update several tests every time the capabilities response changed. The only tests that really care about the low-level details of the capabilities requests are those testing the protocol handshake. And those are mostly not instantiating peer instances or are contained to limited files. This commit adds an option to `hg debugwireproto` to suppress logging of the handshake. The shell helper function to perform HTTP tests has been updated to use this by default. Lots of excessive test output has gone away. Differential Revision: https://phab.mercurial-scm.org/D3378

hg: pass command intents to repo/peer creation (API) The previous commit introduced a mechanism to declare command intents. This commit changes the repository and peer instantiation mechanism so the intents are passed down to each repository and peer type so they can do with them whatever they please. Currently, nobody does anything with any intent. Differential Revision: https://phab.mercurial-scm.org/D3377

registrar: replace "cmdtype" with an intent-based mechanism (API) Commands perform varied actions and repositories vary in their capabilities. Historically, the .hg/requires file has been used to lock out clients lacking a requirement. But this is a very heavy-handed approach and is typically reserved for cases where the on-disk storage format changes and we want to prevent incompatible clients from operating on a repo. Outside of the .hg/requires file, we tend to deal with things like optional, extension-provided features via checking at call sites. We'll either have checks in core or extensions will monkeypatch functions in core disabling incompatible features, enabling new features, etc. Things are somewhat tolerable today. But once we introduce alternate storage backends with varying support for repository features and vastly different modes of behavior, the current model will quickly grow unwieldy. For example, the implementation of the "simple store" required a lot of hacks to deal with stripping and verify because various parts of core assume things are implemented a certain way. Partial clone will require new ways of modeling file data retrieval, because we can no longer assume that all file data is already local. In this new world, some commands might not make any sense for certain types of repositories. What we need is a mechanism to affect the construction of repository (and eventually peer) instances so the requirements/capabilities needed for the current operation can be taken into account. "Current operation" can almost certainly be defined by a command. So it makes sense for commands to declare their intended actions. This commit introduces the "intents" concept on the command registrar. "intents" captures a set of strings that declare actions that are anticipated to be taken, requirements the repository must possess, etc. These intents will be passed into hg.repo(), which will pass them into localrepository, where they can be used to influence the object being created. Some use cases for this include: * For read-only intents, constructing a repository object that doesn't expose methods that can mutate the repository. Its VFS instances don't even allow opening a file with write access. * For read-only intents, constructing a repository object without cache invalidation logic. If the repo never changes during its lifetime, nothing ever needs to be invalidated and we don't need to do expensive things like verify the changelog's hidden revisions state is accurate every time we access repo.changelog. * We can automatically hide commands from `hg help` when the current repository doesn't provide that command. For example, an alternate storage backend may not support `hg commit`, so we can hide that command or anything else that would perform local commits. We already kind of had an "intents" mechanism on the registrar in the form of "cmdtype." However, it was never used. And it was limited to a single value. We really need something that supports multiple intents. And because intents may be defined by extensions and at this point are advisory, I think it is best to define them in a set rather than as separate arguments/attributes on the command. Differential Revision: https://phab.mercurial-scm.org/D3376

cleanup: polyfill assertRaisesRegex so we can avoid assertRaisesRegexp The latter is deprecated on Python 3.7 and causes our tests to fail due to the warning. Differential Revision: https://phab.mercurial-scm.org/D3375