cmdutil: ensure PatchError messages are coerded to bytes, not str Differential Revision: https://phab.mercurial-scm.org/D2613

tests: fix bytes literals in test-fncache.t # skip-blame just b prefixes Differential Revision: https://phab.mercurial-scm.org/D2612

scmutil: avoid using basestring and add explicit handling of unicodes This resolves some Python 3 defects, and I don't think it is a meaningful behavior change in Python 2. Differential Revision: https://phab.mercurial-scm.org/D2611

tests: fix inline extension bytes in test-ssh-proto-unbundle.t # skip-blame just b prefixes Differential Revision: https://phab.mercurial-scm.org/D2610

hghave: fix up clang-libfuzzer regex to be bytes Fixes this check on Python 3. # skip-blame just a b prefix Differential Revision: https://phab.mercurial-scm.org/D2607

py3: accept both unicode and byte strings as filename carried by IOError Follows up 77f98867538f. We could assume there's no bytes filename in our codebase, but it's probably better to not raise UnicodeError because of a unicode filename.

py3: back out c77c925987d7 to store bytes filename in IOError Appears that this is how Python 3 works.

largefiles: headers and values need to be sysstrs, add r prefixes # skip-blame just some r prefixes Differential Revision: https://phab.mercurial-scm.org/D2606

cext: accept arguments as Py_buffer The s*/y* value formatters receive a Py_buffer instead of a char *. This value format is more flexible in the types that it allows. We change bdiff() to accept any object that conforms to the buffer protocol. We validate the buffers are contiguous and have a single dimension. This allows memoryview instances to be handled by the function, so we revert a recent change to cast arguments to bytes before calling this function. Differential Revision: https://phab.mercurial-scm.org/D2587

cext: refactor cleanup code in bdiff() A future commit will need to introduce additional cleanup code. We refactor the cleanup code to check NULL before calling free(). We also initialize these variables as NULL. We set the out of memory exception explicitly, so we can just return result. Differential Revision: https://phab.mercurial-scm.org/D2586

py3: use pycompat.bytestr() to convert error messages to bytes Differential Revision: https://phab.mercurial-scm.org/D2535

url: more bytes/unicodes fussing in url.py around auth handling Once again, these methods are a little annoying to handle because they can get unicodes or bytes depending on who's calling. I think we can probably clean this up a TON once we can run something like pytype and do typechecking of our Python, but until then this is going to be the easy way out. This fixes test-http-bundle1.t. Differential Revision: https://phab.mercurial-scm.org/D2599

httpconnection: convert url to bytes in readauthforuri This method is sometimes called by the stdlib, so we just need to accept both bytes and unicodes here. Awesome. Differential Revision: https://phab.mercurial-scm.org/D2598

tests: prevent enormous output spew in test-lfs-largefiles.t This test currently fails on Python 3, but in a super-slow way. Adding this `head` invocation costs us nothing, but will prevent failures in this area from being super expensive. Differential Revision: https://phab.mercurial-scm.org/D2600

py3: fix formatting of path-auditing errors

py3: make os.curdir a bytes

py3: make os.pardir a bytes

py3: fix slicing of bytes in patch.iterhunks()

tests: fix various test-check-module-imports.t violations Somehow these are only caught when running the test under Python 3. Differential Revision: https://phab.mercurial-scm.org/D2580

pycompat: prevent encoding or decoding values if not required pycompat.py has functions strurl and bytesurl which decodes and encodes the url passed on Python 3 respectively. In some cases, strurl gets a url which is already str and bytesurl gets a url which is already bytes. Let's prevent encoding or decoding the values again if not required. Differential Revision: https://phab.mercurial-scm.org/D2472

py3: add some b'' to make test-lock-badness.t happy

py3: fix formatting of lock error message

py3: fix some unicode madness in global exception catcher

py3: pass a system-string filename to sub-classes of IOError

py3: fix some membership tests on linkrev adjustment

py3: make test-basic.t pass on Python 3

py3: silence the final IOError by closing stdout/err slightly early Fixes the following test failure: $ hg status >/dev/full abort: No space left on device Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' ... OSError: [Errno 28] No space left on device [120]

py3: conditionalize initialization of stdio flags Since Python 3 doesn't depend on the stdio of libc, there should be no need to set O_BINARY flag on Windows.

test-command-template: glob out detailed "invalid escape" message Python 3 also reports the position where an invalid escape found.

templater: byte-stringify dict/list values before passing to default format bytestr() is applied only when no custom format string like '%d' is specified.

templater: allow dynamically switching the default dict/list formatting '%s' doesn't work nicely on Python 3 because many Python types don't implement __bytes__().

py3: use util.forcevytestr to convert error to bytes Differential Revision: https://phab.mercurial-scm.org/D2585

util: report integer result from write() Python 2 sometimes returns None from write() calls. Python 3 doesn't. This will make test output inconsistent between Python 2 and 3. So let's paper over the differences by converting None to the length of the result string. Differential Revision: https://phab.mercurial-scm.org/D2584

util: log readinto() I/O Differential Revision: https://phab.mercurial-scm.org/D2583

util: teach escapedata() about bytearray re.map doesn't seem to know about bytearray (at least in Python 2). Cast bytearray to a bytes to work around this inconvenience. Differential Revision: https://phab.mercurial-scm.org/D2582

sshpeer: don't read(0) read(0) is essentially a no-op. Let's avoid the function call, overhead, and extra test output by not performing it. Differential Revision: https://phab.mercurial-scm.org/D2581

py3: unblock C extensions on Python 3 Please make sure to build C extensions before running tests with -l: $ make local PYTHON=python3

py3: make test-ancestors.py pass on Python 3 with C extensions # skip-blame just some b prefixes

py3: do not pass a memoryview to bdiff.bdiff() This doesn't look nice, but I don't know how to make a zero-copy slice of bytes which is compatible with the buffer protocol.

py3: do not pass a list of iterators to computephasesmapsets()

cext: fix computephasesmapsets() not to return without setting an exception Spotted by map() of Python 3.

py3: do not pass a float to dict_new_presized() I really don't like the division operator of Python 3 since I'm not doing a math.

cext: mark tuple_format as a constant Just for clarity.

py3: bulk-replace 'const char*' format specifier passed to PyArg_ParseTuple*() Perhaps we need this because 's' accepts a unicode string. https://docs.python.org/3/c-api/arg.html#strings-and-buffers Substituted using the following pattern with some manual fixes: '\b(PyArg_ParseTuple)\((\s*\w+,\s*)"([^"]+)"' '\b(PyArg_ParseTupleAndKeywords)\((\s*\w+,\s*\w+,\s*)"([^"]+)"'

py3: bulk-replace bytes format specifier passed to Py_BuildValue() On Python 3, "s" means a utf-8 string. We have to use "y" for bytes, sigh. https://docs.python.org/3/c-api/arg.html#c.Py_BuildValue Substituted using the following pattern with some manual fixes: '\b(Py_BuildValue)\((\s*)"([^"]+)"'

py3: add PY23() macro to switch string literal depending on python version I have no better idea to work around the bytes-unicode divergence of Py_BuildValue(). Maybe we can write a code transformer for C extensions? :)

py3: don't try to mangle C extension blob by code transformer

tests: add missing b prefixes in test-atomictempfile.py # skip-blame just some b prefixes Differential Revision: https://phab.mercurial-scm.org/D2570

wireproto: only expose "between" to version 1 of wire protocols We recently marked other legacy commands as only available to version 1 of the wire protocols. We held off marking "between" because it is used as part of the SSH handshake. Since SSH servers assume they are version 1 by default and the "between" request that is issued as part of the version 2 handshake is swallowed and not operated on, everything should "just work" if "between" is not available to version 2. Differential Revision: https://phab.mercurial-scm.org/D2513

tests: add more tests around hook output and getbundle The previous tests around hook output only tested Python hooks. Let's add some shell hooks in for additional test coverage. Differential Revision: https://phab.mercurial-scm.org/D2550

wireproto: add transport specific capabilities in the transport We add a method to the protocol handler interface that allows specific implementations to add their own capabilities. The SSH implementation is a no-op. The HTTP implementation adds the HTTP-specific capabilities. The end result is transport specific capabilities now live in the transport code instead of in some generic function in the wireproto module. Differential Revision: https://phab.mercurial-scm.org/D2512

wireproto: don't expose changegroupsubset capability if not available We just marked the changegroupsubset command as only available to version 1 of the wire transports. There is a capability of the same name of the command that indicates if the command is supported. This commit teaches the capabilities code to conditionally emit that capability depending on whether the command is available for the current transport. Most test output is reordering of capabilities. But the limited tests for version 2 of the SSH protocol do show the capability disappearing. Differential Revision: https://phab.mercurial-scm.org/D2486

wireproto: don't expose legacy commands to version 2 of wire protocol Now that we have the ability to control which transports a wire protocol command is exposed on, let's put it to use. We flag the "branches," "changegroup," and "changegroupsubset" commands as only available on version 1. "branches" was used by the legacy discovery mechanism and was replaced by the "known" and "heads" commands. "changegroup" and "changegroupsubset" were replaced by "getbundle." "between" is also legacy. However, since it is used by the SSH handshake protocol, marking it as legacy is a bit more complicated and will be done in a later commit. Another nuanced issue with this change is that the server-advertised capabilities still list "changegroupsubset" despite the command not being available. This will be addressed in a subsequent commit. Differential Revision: https://phab.mercurial-scm.org/D2485

wireprotoserver: identify requests via version 2 of SSH protocol as such The protocol handler needs to advertise itself as version 2 in order for a future feature to work. Differential Revision: https://phab.mercurial-scm.org/D2484

wireproto: allow wire protocol commands to declare transport support Currently, wire protocol commands are exposed on all transports. Some wire protocol commands are only supported or sensical on some transports. In the future, new wire protocol commands may only be available on new transports and legacy wire protocol commands may not be available to newer transports. This commit introduces a mechanism to allow @wireprotocommand to declare transports for which they should not be available. The mechanism for determining if a wire protocol command is available for a given transport instance has been taught to take this knowledge into account. To help implement this feature, we add a dict to wireprototypes declaring all wire transports and their metadata. There's probably room to refactor the constants used to identify the wire protocols. But that can be in another commit. Differential Revision: https://phab.mercurial-scm.org/D2483

sshpeer: don't read from stderr when that behavior is disabled We previously prevented the creation of doublepipe instances when we're not supposed to automatically read from stderr. However, there were other automatic calls to read from stderr that were undermining this effort. This commit prevents all automatic reads from stderr from occurring when they are supposed to be disabled. Because stderr is no longer being read, we need to call "readavailable" from tests so stderr is read from. Test output changes because stderr is now always (manually) read after stdout. And, since sshpeer no longer automatically tends to stderr, no "remote: " messages are printed. This should fix non-deterministic test output. FWIW, doublepipe automatically reads from stderr when reading from stdout, so I'm not sure some of these calls to self._readerr() are even needed. Differential Revision: https://phab.mercurial-scm.org/D2571

util: extract all date-related utils in utils/dateutil module With this commit, util.py lose 262 lines Note for extensions author, if this commit breaks your extension, you can pull the step-by-step split here to help you more easily pinpoint the renaming that broke your extension: hg pull https://bitbucket.org/octobus/mercurial-devel/ -r ac1f6453010d Differential Revision: https://phab.mercurial-scm.org/D2282

clone: updates the help text for hg clone -{r,b} (issue5654) Differential Revision: https://phab.mercurial-scm.org/D2095

py3: whitelist more passing tests Differential Revision: https://phab.mercurial-scm.org/D2569

lfs: convert hexdigest to bytes using sysbytes Differential Revision: https://phab.mercurial-scm.org/D2568