mdiff: use str.startswith/endswith() instead of slicing

patch: unify check_binary and binary flags Follows up 079b27b5a869. If opts.text=True, check_binary is ignored, so we can just pass the binary flag to unidiff(). perfunidiff now takes any inputs as text files, which I think is a desired behavior.

py3: make sure we open the file in bytes mode # skip-blame because we are just adding b Differential Revision: https://phab.mercurial-scm.org/D2077

py3: use bytes instead of str Differential Revision: https://phab.mercurial-scm.org/D2076

py3: add b'' prefixes to string literals in test files # skip-blame because we are just adding b'' Differential Revision: https://phab.mercurial-scm.org/D2075

py3: use range instead of xrange xrange is not available on Python 3. Differential Revision: https://phab.mercurial-scm.org/D2074

py3: use open() instead of file() file() is not present in Python 3. Differential Revision: https://phab.mercurial-scm.org/D2073

py3: use pycompat.bytestr to convert _b85chars to bytes The tranformer does append b'' to the value and make that a bytes but bytes in Python 3 returns the ascii value on getting characters using indexing. Characters of this string are queried using indexing multiple times in the file and to support that we use pycompat.bytestr which returns the bytechrs using indexing. Differential Revision: https://phab.mercurial-scm.org/D2072

py3: use pycompat.bytechr instead of chr Differential Revision: https://phab.mercurial-scm.org/D2071

py3: use pycompat.ziplist instead of zip zip returns a zip object instead of a list on Python 3. Differential Revision: https://phab.mercurial-scm.org/D2070

patches: move assignment outside the conditional Having this movement in its own patch will make the next patch clearer.

sshpeer: remove support for connecting to <0.9.1 servers (BC) 197d10e157ce made this change for the HTTP peer. Let's do the same for the SSH peer. Test output changes as expected. A redundant test has been dropped. .. bc:: Support for connecting to Mercurial servers older than 0.9.1 has been removed. Differential Revision: https://phab.mercurial-scm.org/D2036

sshpeer: document the handshake mechanism The mechanism by which SSH peers establish connections with remotes is wonky and requires a bit of code archeology to understand. While it is already documented in `hg help internals.wireproto`, it helps to have documentation in the code as well. Differential Revision: https://phab.mercurial-scm.org/D2035

sshpeer: move handshake outside of sshpeer With the handshake now performed before a peer class is instantiated, we can now instantiate a different peer class depending on the results of the handshake. Our test extension had to change to cope with the new API. Because we now issue the command via raw I/O calls and don't call _callstream(), we no longer have to register the fake command. (_callstream() uses the command registration to see what args to send). Differential Revision: https://phab.mercurial-scm.org/D2034

sshpeer: inline I/O into _validaterepo() We want to move the handshake code out of the peer class so the peer factory function can perform the handshake and instantiate a proper class depending on the results. To make that refactor easier to read, we first inline I/O functionality into _validaterepo(). Test output for low-level protocol tests didn't change, thus hopefully demonstrating that this refactor didn't change any material behavior. Because we no longer call _callstream(), our test extension for monkeypatching the peer had to change its hook point. Differential Revision: https://phab.mercurial-scm.org/D2033

sshpeer: clean up API for sshpeer.__init__ (API) Our refactoring left the state of sshpeer.__init__ in a poor state. "create" was no longer used. Process/pipe arguments were passed poorly. "name" was really a URL. This commit cleans all that up. .. api:: sshpeer.sshpeer.__init__ now receives arguments describing an existing connection instead of creating a connection itself. Differential Revision: https://phab.mercurial-scm.org/D2032

sshpeer: establish SSH connection before class instantiation We want to move the handshake to before peers are created so we can instantiate a different peer class depending on the results of the handshake. This necessitates moving the SSH process invocation to outside the peer class. As part of the code move, some variables were renamed for clarity. util.popen4() returns stdin, stdout, and stderr in their typical file descriptor order. However, stdin and stdout were being mapped to "pipeo" and "pipei" respectively. "o" for "stdin" and "i" for "stdout" is a bit confusing. Although it does make sense for "output" and "input" from the perspective of the client. But in the context of the new function, it makes sense to refer to these as their file descriptor names. In addition, the last use of self._path disappeared, so we stop setting that attribute and we can delete the redundant URL parsing necessary to set it. Differential Revision: https://phab.mercurial-scm.org/D2031

sshpeer: remove frivolous call to _cleanup() _validaterepo() is called once during __init__. _cleanup() no-ops if the self._pipe* attributes aren't set. These attributes are set during _validaterepo(). So the call to _cleanup() isn't necessary. But just to be on the safe side, we add an assertion. Differential Revision: https://phab.mercurial-scm.org/D2030

sshpeer: extract pipe cleanup logic to own function So it can be used outside of instantiated classes. This is needed to support pipe creation before __init__ is called. Differential Revision: https://phab.mercurial-scm.org/D2029

sshpeer: move ssh command and repo creation logic out of __init__ It was easier to move both of these at once because repository creation requires various variables and I didn't want to add tons of arguments and code to __init__ that will soon be deleted anyway. We do add an extra argument so we can proxy values to the _validaterepo() call. But this is minimally invasive. Some callers of self._abort() were converted to just raise. Like before, the _abort() call wasn't necessary because self._pipe* aren't populated this early in the object's lifetime. As part of this, various private attributes derived from the parsed URL are no longer used. So we no longer set them. Differential Revision: https://phab.mercurial-scm.org/D2028

sshpeer: move URL validation out of sshpeer.__init__ We will soon have another SSH peer class to support the new version of the SSH protocol. However, we won't know which peer class to instantiate until we perform a handshake on an active connection. This means that we need to move connection establishment and handshake code out of sshpeer.__init__. This commit starts the process of migrating peer creation code out of sshpeer.__init__ into instance(), which is the API for creating peers. The moved code no longer calls _abort(). _abort() runs _cleanup() and raises. _cleanup() only performs actions on self._pipe*. These objects aren't instantiated until we actually connect to the peer. So _abort() was not necessary in the old code. To keep the API the same, __init__() now makes a redundant call to util.url(). This will be fixed in subsequent commits. Differential Revision: https://phab.mercurial-scm.org/D2027

tests: add low-level SSH protocol tests We don't really have good low-level tests for the behavior of the SSH wire protocol. This commit attempts to establish some. The added tests consist of a mixture of starting a server with `hg serve --stdio` and sending bytes to it and using `hg debugpeer` to go through the official client code. Having insight into what raw bytes are exchanged as well as what the peer does is useful. We also introduce a test extension for modifying the behavior of the SSH server and peer. For example, we change the server to not recognize the "hello" command, simulating behavior of <0.9.1 servers. These tests are generally useful to have. But the impetus for creating them now is they will be needed for verifying behavior of old clients and servers when a new SSH protocol is introduced. Differential Revision: https://phab.mercurial-scm.org/D2026 # no-check-commit because of serve_forever()

debugcommands: introduce debugpeer command `hg debugpeer <path>` will establish a connection to a peer repository and print information about it. If you add --debug, it will log low-level protocol request info. This will be useful for upcoming tests around protocol handshaking. Differential Revision: https://phab.mercurial-scm.org/D2025

sshpeer: make "instance" a function The API is that peer modules must provide an "instance" symbol that is callable to return a peer. Making "instance" a function instead of an alias to "sshpeer" makes it easier to monkeypatch the "sshpeer" type. It will also make it possible to turn instance() into a factory function of sorts that returns different types based on connection properties. Differential Revision: https://phab.mercurial-scm.org/D2024

lfs: deduplicate oids in the transfer Apparently, we can't rely on the server to deduplicate for us. Sadly, the pointer object isn't hashable, so it can't be reduced by converting it to a set. In order to be hashable, it needs to be immutable. I had a bunch of code to change it to composition and forward the readonly dict methods to a member dict. But the pointer is updated via __setitem__() when creating the pointer file. So it didn't see worth adding all of that code to the class.

cat: call the storage prefetch hook It's not important to call in the case of a single file, but maybe it's better to do so for consistency.

archive: call the storage prefetch hook

lfs: prefetch lfs blobs during revert The revert command oddly prints out what it will do before requesting the files to be prefetched. But the 'need to transfer' line indicates the blobs are being grouped.

cmdutil: convert _revertprefetch() to a generic stored file hook (API) This will be used by LFS to fetch required files in a group for multiple commands, prior to being accessed. That avoids the one-at-a-time fetch when the filelog wrapper goes to access it, and it is missing locally (which costs two round trips to the server.) The core command list that needs this is probably at least: - annotate - archive (which is also used by extdiff) - cat - diff - export - grep - verify (sadly) - anything that has the '{data}' template There are no core users of the revert prefetch hook, and never have been since it was introduced in 45e02cfad4bd for remotefilelog. Thanks to Yuya for figuring out a way to reliably trigger the deprecated warning. Unfortunately, it wanted to blame the caller of revert. Passing along an adjusted stack level seemed the least bad choice (although it still blames a core function). One thing to note is that the store lock isn't being held when this is called. I'm not at all familiar with remotefilelog or its locking requirements, so this may not be a big deal. Currently, LFS doesn't hold a lock when downloading files. Even though largefiles doesn't either, I'm starting to think it should, and maybe the .hg/store/lock isn't good enough to cover the globally shared cache. .. api:: The cmdutil._revertprefetch() hook point for prefetching stored files has been replaced by the command agnostic cmdutil._prefetchfiles(). The new function takes a list of files, instead of a list of lists of files.

lfs: prefetch lfs blobs when applying merge updates In addition to merge, this method ultimately gets called by many commands: - backout - bisect - clone - fetch - graft - import (without --bypass) - pull -u - rebase - strip - share - transplant - unbundle - update Additionally, it's also called by histedit, shelve, unshelve, and split, but it seems that the related blobs should always be available locally for these. For `hg update`, it happens after the normal argument checking and pre-update hook processing, and remote corruption is detected prior to manipulating the working directory. Other commands could use this treatment (archive, cat, revert, etc), but this covers so many of the frequently used bulk commands, it seems like a good starting point. Losing the verbose message that prints the file name before a corrupt blob aborts the command is a little sad, because there's no easy way to go from oid to file name. I'd like to change that message to list the file name so it looks cleaner and less cryptic, but the pointer object is nowhere near where it needs to be to do this. So punt on that for now.