tests: various Python 3 ports for test-remotefilelog-datapack.py Use bytes I/O. Use byteschr(). Convert temporary path to bytes. Differential Revision: https://phab.mercurial-scm.org/D5758

tests: use items() in test-remotefilelog-datapack.py Performance doesn't matter in tests. iteritems() doesn't exist in Python 3. Differential Revision: https://phab.mercurial-scm.org/D5757

tests: use bytes and %d formatting in test-remotefilelog-datapack.py There were numerous failures on Python 3 due to str/bytes mismatch and '%s' not working for ints. Differential Revision: https://phab.mercurial-scm.org/D5756

perf: add a --[no-]clear-caches option to `perfnodemap` The option is useful to look at pure lookup performance on a warm data structure.

perf: add a perfnodemap command The command focus on timing of the nodemap object itself.

wireprotov1server: use binascii.unhexlify The "hex" codec doesn't exist in Python 3. We could use `codecs.decode(h, 'hex_codec')`. But `binascii.unhexlify()` exists and should work the same on Python 2 and 3. Differential Revision: https://phab.mercurial-scm.org/D5755

tests: conditionalize test-http-bad-server.t for Python 3.5 It appears that Python 3 introduced output buffering in the HTTP response stack. And Python 3.6 switched from sock.makefile().write() to sock.sendall(). So, we need to conditionalize test-http-bad-server.t to account for the difference in behavior between Python 3.5 and 3.6. Differential Revision: https://phab.mercurial-scm.org/D5754

tests: log sendall() operations and port test-http-bad-server.t Python 3's HTTP server layer buffers output and uses sendall() instead of write(). In order to make test-http-bad-server.t pass on Python 3, we needed to teach our socket proxy to log sendall() events and to abort future sends if we reached our send limit. The tests using `tail` were difficult to port with inline output conditionals since the number of lines varied. So we now use `#if py3` for these tests. test-http-bad-server.t now passes on Python 3.6 and 3.7 on at least Linux. However, it does not yet pass on Python 3.5 because of low-level differences to how the HTTP server is implemented. Differential Revision: https://phab.mercurial-scm.org/D5753

tests: glob away readline(-1) Most of these are readline(65537) on Python 3. I don't think it is worth the readability hit to use (re), as it would require escaping parenthesis. Differential Revision: https://phab.mercurial-scm.org/D5752

tests: change how sockets are closed Python 3 uses a different type to represent a socket file object than Python 2. We need to conditionalize how the socket is closed accordingly. While we're here, we switch to use socket.shutdown() to close the socket. This is because socket.close() may not actually close the socket until it is GCd. socket.shutdown() forces an immediate shutdown. I suspect Python 3 changed semantic behavior here, as I can't get test-http-bad-server.t to work with socket.close(). socket.shutdown() does appear to work, however. Differential Revision: https://phab.mercurial-scm.org/D5751

tests: add b'' prefixes to badserverext.py This avoids a handful of failures due to missing str and bytes. # skip-blame: just a bunch of b'' prefixes Differential Revision: https://phab.mercurial-scm.org/D5750

hgweb: log error before attempting I/O Previously, an uncaught exception during HTTP request serving would attempt to send an error response then log the exception. If an exception occurred during I/O, this exception would be raised and the original exception wouldn't be logged. This commit changes behavior so the original exception is logged first, before we attempt to do anything else. This ensures the exception is logged. This change resulted in new tracebacks appearing in various tests. Because tracebacks can vary between Python versions, we added a simple script to filter the stack part of traceback lines. This makes testing much simpler, as we don't need to glob over lines and make lines conditional. Differential Revision: https://phab.mercurial-scm.org/D5749

tests: write commit message using file I/O Python 2.7 will print() \x94\x5c\x0a whereas Python 3 will print() \xc2\x94\x5c\x0a. Why, I'm not sure. It probably has to do with print() being Unicode aware on Python 3 and Python attempting some kind of encoding before emitting the output. This difference results in a different bytes making it to the commit message and the JSON output varying. We work around this by writing bytes to a commit message file. Differential Revision: https://phab.mercurial-scm.org/D5741

Added signature for changeset 83377b4b4ae0

Added tag 4.9 for changeset 83377b4b4ae0

subrepo: reject potentially unsafe subrepo paths (BC) (SEC) In addition to the previous patch, this prohibits '~', '$nonexistent', etc. for any subrepo types. I think this is safer, and real-world subrepos wouldn't use such (local) paths.

subrepo: prohibit variable expansion on creation of hg subrepo (SEC) It's probably wrong to expand path at localrepo.*repository() layer, but fixing the layering issue would require careful inspection of call paths. So, this patch adds add a validation to the subrepo constructor. os.path.realpath(util.expandpath(root)) is what vfsmod.vfs() would do.

subrepo: extend path auditing test to include more weird patterns (SEC) While reviewing patches for the issue 5739, "$foo in repository path expanded", I realized that subrepo paths can also be cheated. This patch includes various subrepo paths which are potentially unsafe. Since an expanded subrepo path isn't audited, this bug allows symlink check bypass. As a result, a malicious subrepository could be checked out to a sub tree of e.g. $HOME directory. The good news is that the destination directory must be empty or nonexistent, so the existing ~/.bashrc wouldn't be overwritten. See the last part of the tests for details.

copyright: update to 2019 Differential Revision: https://phab.mercurial-scm.org/D5779

patch: handle 0 context lines (diff.unified=0) when parsing patches Previously, if there were no context lines, we would just keep updating the ranges and the hunk, but not actually storing the hunk (just overwriting it each time). Thus a diff like this: $ hg diff --config diff.unified=0 diff --git a/bar b/bar --- a/bar +++ b/bar @@ -1,0 +2,1 @@ 1 +change1 @@ -3,0 +5,1 @@ 3 +change2 would come out of the parser like this (change1 is lost): bar: @@ -3,0 +5,1 @@ 3 +change2 This had some really weird side effects for things like commit --interactive, split, etc. Differential Revision: https://phab.mercurial-scm.org/D5743

py3: pass str into RuntimeError() to prevent b'' in output # skip-blame as just r'' prefixes Differential Revision: https://phab.mercurial-scm.org/D5740

py3: use '%d' instead of '%s' for integers Differential Revision: https://phab.mercurial-scm.org/D5738

tests: port test-hgweb-auth.py to Python 3 Differential Revision: https://phab.mercurial-scm.org/D5736

url: convert some variables back to bytes Differential Revision: https://phab.mercurial-scm.org/D5735

url: add some defensive asserts on expected incoming types Our type handling is a nightmare here, and we're loading passwords to do network IO, so we can afford to be potentially-slow but pedantic here. Differential Revision: https://phab.mercurial-scm.org/D5734

changegroup: don't try to build changelog chunks if not required When we extend a narrow clone without ellipsis, we don't download changelog information because that's already present with the client. However we still try to build that chunk stream. Building that chunk stream involves calling a lookup function and store.emitrevisions() API. The lookup function is called len(cl) number of times. On large repositories, where len(cl) is in millions, calling that lookup function is not a good idea. Also it's not required to use the store.emitrevisons() API because we already have nodes present which we can use. This patch short-circuits state building logic if we are processing a non-ellipsis case and changelog is not required. This saves up ~20 seconds on our internal repo for a single extend call. Differential Revision: https://phab.mercurial-scm.org/D5733

revlog: make sure we never use sparserevlog without general delta (issue6056) We are getting user report where the delta code tries to use `sparse-revlog` logic on repository where `generaldelta` is disabled. This can't work so we ensure the two booleans have a consistent value. Creating this kind of repository is not expected to be possible the current bug report point at a clonebundle related bug that is still to be properly isolated (Yuya Nishihara seems to a have done it). Corrupting a repository to reproduce the issue is possible. A test using this method is included in this fix.

sparserevlog: document the config option This was overlooked when this graduated from experimental.

changegroup: initialize the state variable a bit earlier This will make the next patch much easier. Differential Revision: https://phab.mercurial-scm.org/D5732

tests: conditionalize test output on Python 3.7 Python 3.7 changed behavior of urllib.parse.quote() from RFC 2396 to RFC 3986 and ~ is now in the set of reserved characters and isn't escaped. We conditioanlize test output accordingly. Differential Revision: https://phab.mercurial-scm.org/D5717