extdiff: add --per-file and --confirm options The new options lets the user control how the external program is run. By default, Mercurial passes the 2 snapshot directories as usual, but it can also run the program repeatedly on each file's snapshot pair, and optionally prompt the user each time.

run-tests: sort the skip, failure and error lists in the final output This will help keep the lists consistent, for comparison across runs.

tests: add Python 3 output for test-remotefilelog-gc.t Python 3 raises a slightly different error on invalid paths. Differential Revision: https://phab.mercurial-scm.org/D5773

hg: raise Abort on invalid path Currently, some os.path functions when opening repositories may raise an uncaught TypeError or ValueError if the path is invalid. Let's catch these exceptions and turn them into an Abort for convenience. Differential Revision: https://phab.mercurial-scm.org/D5772

subrepo: bytes/str cleanups on Git support Git subrepo tests now pass on Python 3. Differential Revision: https://phab.mercurial-scm.org/D5768

tests: compare against a bytes in test-lock.py skip-blame: just b'' prefixes Differential Revision: https://phab.mercurial-scm.org/D5771

tests: perform a shallow copy instead of a deep copy Python 3 can't perform a deep copy because it looks like symbols in the locals() namespace are not deep-copyable. For the curious, somehow the deepcopy() is attempting to copy objects attached to the unittest.* functions for the running test! We don't use deepcopy() anywhere in the code base and a shallow object copy should be sufficient to test lock copying. Actually, I'm not sure why we even test this, as I couldn't find copy.copy() being used for lock copying either. Who knows. Differential Revision: https://phab.mercurial-scm.org/D5770

tests: make and use a new `svnurlof.py` helper for constructing svn urls The previous trick of a Python oneliner and some subshells is too hard to make portable, and this lets us consolidate some Windows-specific logic down to a single place. Differential Revision: https://phab.mercurial-scm.org/D5766

server: skip logging of ECONNRESET I believe this was exposed by 5492dc20, because the sending of the 500 would have already failed and prevented this logging. On Python 3, this will be a ConnectionResetError, which is a subtype of OSError, which is why we check for both OSError and socket.error. Bonus: this fixes a race in test-hgweb.t where sometimes the ECONNRESET wouldn't happen, because now we just don't log those errors. Differential Revision: https://phab.mercurial-scm.org/D5764

git: a little pycompat.bytestring() love to make this code work in py3 Differential Revision: https://phab.mercurial-scm.org/D5765

py3: have test-revset2.t write test scripts in a more portable way Fixes the test on Python 3. Differential Revision: https://phab.mercurial-scm.org/D5763

py3: fix up test-remotefilelog-cacheprocess.t to not depend on a repr It looks like the repr() of Exceptions is different from Python 2 to Python 3.7 (but not 3.5?), but the str() is still stable. Sigh. Differential Revision: https://phab.mercurial-scm.org/D5761

remotefilelog: cast division result to an int Otherwise mid is a float and this confuses __slice__ on Python 3. Differential Revision: https://phab.mercurial-scm.org/D5760

tests: cast division result to int Otherwise it is a float on Python 3 and code later compares about casting a float to an int. Differential Revision: https://phab.mercurial-scm.org/D5759

tests: various Python 3 ports for test-remotefilelog-datapack.py Use bytes I/O. Use byteschr(). Convert temporary path to bytes. Differential Revision: https://phab.mercurial-scm.org/D5758

tests: use items() in test-remotefilelog-datapack.py Performance doesn't matter in tests. iteritems() doesn't exist in Python 3. Differential Revision: https://phab.mercurial-scm.org/D5757

tests: use bytes and %d formatting in test-remotefilelog-datapack.py There were numerous failures on Python 3 due to str/bytes mismatch and '%s' not working for ints. Differential Revision: https://phab.mercurial-scm.org/D5756

perf: add a --[no-]clear-caches option to `perfnodemap` The option is useful to look at pure lookup performance on a warm data structure.

perf: add a perfnodemap command The command focus on timing of the nodemap object itself.

wireprotov1server: use binascii.unhexlify The "hex" codec doesn't exist in Python 3. We could use `codecs.decode(h, 'hex_codec')`. But `binascii.unhexlify()` exists and should work the same on Python 2 and 3. Differential Revision: https://phab.mercurial-scm.org/D5755

tests: conditionalize test-http-bad-server.t for Python 3.5 It appears that Python 3 introduced output buffering in the HTTP response stack. And Python 3.6 switched from sock.makefile().write() to sock.sendall(). So, we need to conditionalize test-http-bad-server.t to account for the difference in behavior between Python 3.5 and 3.6. Differential Revision: https://phab.mercurial-scm.org/D5754

tests: log sendall() operations and port test-http-bad-server.t Python 3's HTTP server layer buffers output and uses sendall() instead of write(). In order to make test-http-bad-server.t pass on Python 3, we needed to teach our socket proxy to log sendall() events and to abort future sends if we reached our send limit. The tests using `tail` were difficult to port with inline output conditionals since the number of lines varied. So we now use `#if py3` for these tests. test-http-bad-server.t now passes on Python 3.6 and 3.7 on at least Linux. However, it does not yet pass on Python 3.5 because of low-level differences to how the HTTP server is implemented. Differential Revision: https://phab.mercurial-scm.org/D5753

tests: glob away readline(-1) Most of these are readline(65537) on Python 3. I don't think it is worth the readability hit to use (re), as it would require escaping parenthesis. Differential Revision: https://phab.mercurial-scm.org/D5752

tests: change how sockets are closed Python 3 uses a different type to represent a socket file object than Python 2. We need to conditionalize how the socket is closed accordingly. While we're here, we switch to use socket.shutdown() to close the socket. This is because socket.close() may not actually close the socket until it is GCd. socket.shutdown() forces an immediate shutdown. I suspect Python 3 changed semantic behavior here, as I can't get test-http-bad-server.t to work with socket.close(). socket.shutdown() does appear to work, however. Differential Revision: https://phab.mercurial-scm.org/D5751

tests: add b'' prefixes to badserverext.py This avoids a handful of failures due to missing str and bytes. # skip-blame: just a bunch of b'' prefixes Differential Revision: https://phab.mercurial-scm.org/D5750

hgweb: log error before attempting I/O Previously, an uncaught exception during HTTP request serving would attempt to send an error response then log the exception. If an exception occurred during I/O, this exception would be raised and the original exception wouldn't be logged. This commit changes behavior so the original exception is logged first, before we attempt to do anything else. This ensures the exception is logged. This change resulted in new tracebacks appearing in various tests. Because tracebacks can vary between Python versions, we added a simple script to filter the stack part of traceback lines. This makes testing much simpler, as we don't need to glob over lines and make lines conditional. Differential Revision: https://phab.mercurial-scm.org/D5749

tests: write commit message using file I/O Python 2.7 will print() \x94\x5c\x0a whereas Python 3 will print() \xc2\x94\x5c\x0a. Why, I'm not sure. It probably has to do with print() being Unicode aware on Python 3 and Python attempting some kind of encoding before emitting the output. This difference results in a different bytes making it to the commit message and the JSON output varying. We work around this by writing bytes to a commit message file. Differential Revision: https://phab.mercurial-scm.org/D5741

patch: handle 0 context lines (diff.unified=0) when parsing patches Previously, if there were no context lines, we would just keep updating the ranges and the hunk, but not actually storing the hunk (just overwriting it each time). Thus a diff like this: $ hg diff --config diff.unified=0 diff --git a/bar b/bar --- a/bar +++ b/bar @@ -1,0 +2,1 @@ 1 +change1 @@ -3,0 +5,1 @@ 3 +change2 would come out of the parser like this (change1 is lost): bar: @@ -3,0 +5,1 @@ 3 +change2 This had some really weird side effects for things like commit --interactive, split, etc. Differential Revision: https://phab.mercurial-scm.org/D5743

py3: pass str into RuntimeError() to prevent b'' in output # skip-blame as just r'' prefixes Differential Revision: https://phab.mercurial-scm.org/D5740

py3: use '%d' instead of '%s' for integers Differential Revision: https://phab.mercurial-scm.org/D5738