py3: avoid `b'%s' % type(...)` in a ProgrammingError Differential Revision: https://phab.mercurial-scm.org/D7363

py3: fix sorting of obsolete markers in obsutil (issue6217) This is similar to 01e8eefd9434 and others. We move the sortedmarkers() function from exchange module to obsutil.

py3: fix handling of ctrl keys in crecord (issue6213) The "keypressed" value in handlekeypressed() is a key name obtained by curses's getkey(); this can be a multibyte string for special keys like CTRL keys. Calling curses.unctrl() with such a value fails on Python 3 with a TypeError as described in issue6213. (On Python 2, this does not crash, but I'm not sure the result is correct, though it does no matter here.) So instead of calling unctrl(), we compare "keypressed" with the expected "^L" obtained by curses.ascii.ctrl("L").

py3: keep "keypressed" a native str in crecord This will help in the next changeset by avoiding a decode step. Also, the actual bytes conversion seems superfluous since values coming from curses's getkey() will be a native string. As a consequence, we open the "testcommands" file (used in test-interactive-curses.t) in text mode.

py3: compare response of crecord's confirmationwindow with str confirmationwindow() returns a native string, as a result of calling chr() on getch(). On Python 3, response.lower().startswith(b"y") leads to a TypeError. This fixes a crash when typing "r" in the curses interface of interactive commit.

py3: compare http server's command with a native string The "command" attribute is an str, so comparing with a bytes would not work on Python 3. This might solve issues in test-lfs-serve-access.t that happens sometimes (especially in CI): --- /hgwork/src/tests/test-lfs-serve-access.t +++ /hgwork/src/tests/test-lfs-serve-access.t.err @@ -163,11 +163,13 @@ $ cat $TESTTMP/access.log $TESTTMP/errors.log $LOCALIP - - [$LOGDATE$] "POST /missing/objects/batch HTTP/1.1" 404 - (glob) + $LOCALIP - - [05/Nov/2019 16:32:34] "{"objects": [{"oid": "f03217a32529a28a42d03b1244fe09b6e0f9fd06d7b966d4d50567be2abe6c0e", "size": 20}], "operation": "download"}" HTTPStatus.BAD_REQUEST - $LOCALIP - - [$LOGDATE$] "GET /subdir/mount/point?cmd=capabilities HTTP/1.1" 200 - (glob) $LOCALIP - - [$LOGDATE$] "GET /subdir/mount/point?cmd=batch HTTP/1.1" 200 - x-hgarg-1:cmds=heads+%3Bknown+nodes%3D x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$ partial-pull (glob) $LOCALIP - - [$LOGDATE$] "GET /subdir/mount/point?cmd=getbundle HTTP/1.1" 200 - x-hgarg-1:bookmarks=1&bundlecaps=HG20%2Cbundle2%3DHG20%250Abookmarks%250Achangegroup%253D01%252C02%252C03%250Adigests%253Dmd5%252Csha1%252Csha512%250Aerror%253Dabort%252Cunsupportedcontent%252Cpushraced%252Cpushkey%250Ahgtagsfnodes%250Alistkeys%250Aphases%253Dheads%250Apushkey%250Aremote-changegroup%253Dhttp%252Chttps%250Arev-branch-cache%250Astream%253Dv2&cg=1&common=0000000000000000000000000000000000000000&heads=525251863cad618e55d483555f3d00a2ca99597e&listkeys=bookmarks&phases=1 x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$ partial-pull (glob) $LOCALIP - - [$LOGDATE$] "POST /subdir/mount/point/.git/info/lfs/objects/batch HTTP/1.1" 200 - (glob) $LOCALIP - - [$LOGDATE$] "GET /subdir/mount/point/.hg/lfs/objects/f03217a32529a28a42d03b1244fe09b6e0f9fd06d7b966d4d50567be2abe6c0e HTTP/1.1" 200 - (glob) + $LOCALIP - - [05/Nov/2019 16:32:34] code 400, message Bad request version ('"download"}') Blobs that already exist in the usercache are linked into the repo store, even though the client doesn't send the blob. @@ -195,6 +197,7 @@ server2/.hg/store/lfs/objects/f0/3217a32529a28a42d03b1244fe09b6e0f9fd06d7b966d4d50567be2abe6c0e $ "$PYTHON" $RUNTESTDIR/killdaemons.py $DAEMON_PIDS $ cat $TESTTMP/errors.log + $LOCALIP - - [05/Nov/2019 16:32:34] code 400, message Bad request version ('"download"}') $ cat >> $TESTTMP/lfsstoreerror.py <<EOF > import errno (from https://ci.hg.gregoryszorc.com/job-info/hg-committed-ca3dca416f8d5863ca6f5a4a6a6bb835dcd5feeb-debian10-cpython-3.7-0)

py3: don't use bytes with vars() or __dict__ Inspired by D7227. These were all the remaining instances I could find. Differential Revision: https://phab.mercurial-scm.org/D7230

Added signature for changeset ca3dca416f8d

Added tag 5.2 for changeset ca3dca416f8d

py3: add inline comment about encoding issue of str(Abort())

py3: do not reimplement Abort.__str__() on Python 2 It isn't necessary on Python 2, and the default implementation should be better than our BaseException_str() clone.

tests: write out file using bytes I/O The encoding of sys.stdout varies between Python versions. So using a one-liner to write a file from a Unicode string is not deterministic. This commit writes out the file using bytes I/O to ensure we have exactly the bytes we want in the file. This change fixes a test failure in Python 3.5/3.6. Differential Revision: https://phab.mercurial-scm.org/D7226

import-checker: open all source files as utf-8 Before, we opened in text mode and used the default encoding to interpret the bytes within. This caused problems interpreting some byte sequences in some files. This commit changes things to always open files as UTF-8, which makes the error go away. test-check-module-imports.t now passes on Python 3.5 and 3.6 with this change. Differential Revision: https://phab.mercurial-scm.org/D7225

localrepo: use str for lookup in vars() vars() returns a dict of str. So always use a native str for the key lookup. Differential Revision: https://phab.mercurial-scm.org/D7227

automation: install python3-venv Debian package Debian's python install has a crippled venv by default, as it is lacking ensurepip. When you try to run `python3 -m venv` it tells you to install `python3-venv`. So this commit does that in our automation environment so we can fully test installing Mercurial using venv+pip with the system Python. Differential Revision: https://phab.mercurial-scm.org/D7229

tests: look for ensurepip before using venv Debian appears to cripple the venv module by default by removing the associated ensurepip functionality. (The module isn't present at all.) This caused test-install.t to fail when using the Debian python3 unless the python3-venv package was installed. This commit introduces a new hghave requirement for detecting ensurepip and makes the Python 3 install variant conditional on its presence. This should make test-install.t pass when using an incomplete Debian Python. Differential Revision: https://phab.mercurial-scm.org/D7228

automation: avoid '~' in the temp directory on Windows If a long-ish username is used, the environment variable ends up with a '~' to be 8.3 path compatible. That in turn causes a handful of tests (mostly ssh related) to add quotes around $TESTMP. I have no AWS experience, so I have no idea if this is the proper way to do it. But I've hit this problem locally, and redirecting the directory is a workaround. I don't recall if the directory is created on demand by the test harness, but presumably if this is configured before the machine boots, Windows will do it for us. Differential Revision: https://phab.mercurial-scm.org/D7130

tests: use venv on Python 3 This test was failing in some Python 3 environments because `$PYTHON -m virtualenv` was somehow resulting in Python 2 being used. Why, I'm not sure. Python 3 includes virtualenv in the standard library as the `venv` module. This commit changes test-install.t to use `$PYTHON -m venv` on Python 3 and `$PYTHON -m virtualenv` on Python 2 (if available). I chose to make some test output duplicated because we can't have nested conditionals and there is no easy way to express ORing of hghave checks. Differential Revision: https://phab.mercurial-scm.org/D7224

tests: remove HGALLOWPYTHON3 reference This variable was removed from setup.py in c3e10f705a6c. Differential Revision: https://phab.mercurial-scm.org/D7223

run-tests: use byte strings for inserted output We were inserting str on Python 3 which resulted in mixed str/bytes types on the list. This would later blow up when trying to write str to the .err file opened in bytes mode. Differential Revision: https://phab.mercurial-scm.org/D7222

contrib: require Python 3.7 for byteify-strings.py bb509f39d387 made an error, it's actually 3.7 that introduced token.COMMENT. Differential Revision: https://phab.mercurial-scm.org/D7220

hghave: fix bytes/string issue on Python 3

packaging: add support for CentOS 8 The resulting executable has not been tested in detail yet. I ran 'hg version' and 'hg clone', which worked fine (except for extensions acting up due to Python 3). Differential Revision: https://phab.mercurial-scm.org/D7216

packaging: allow choosing python version depending on centos version Differential Revision: https://phab.mercurial-scm.org/D7217

fsmonitor: use stringutil.forcebytestr() instead of str() on an exception Similar to 5fa8ac91190e / D7206, should get test-install.t passing on py3. Differential Revision: https://phab.mercurial-scm.org/D7218

py3: add a __str__ method to Abort This improves the rendering of some exceptions by avoiding raw bytestrings, especially when using --traceback option.

py3: add Python 3 exception output to test-lfs-serve-access.t Similar to a973a75e92bf or 3e9c6cef949b.

automation: install black This should unblock us from running the code formatting test in our automated environment. Differential Revision: https://phab.mercurial-scm.org/D7197

py3: use %d to format an int Avoids a TypeError under py3. Fortunately this is very much an edge case since it requires the user to have deliberately created a local tag of the form 'D\d+' that isn't truthful. Differential Revision: https://phab.mercurial-scm.org/D7215

fsmonitor: normalize exception types to bytes Unavailable.msg should now always be bytes. We also rename Unavailable.__str__ to __bytes__ as it always returns bytes. We make __str__ a simple wrapper that decodes that result to str. There's probably some excessive strutil.forcebytestr() in fsmonitor/__init__.py now. But at least the exceptions around type coercion should now be gone. Differential Revision: https://phab.mercurial-scm.org/D7214

fsmonitor: normalize clock value to bytes We normalize the value returned by watchman because we perform a number of compares with this value in code. So the easiest path forward is to normalize to bytes so we don't have to update many call sites. With this commit, the fsmonitor extension appears to be working with Python 3! Although there are still some failures in edge cases... Differential Revision: https://phab.mercurial-scm.org/D7213

fsmonitor: use next() instead of .next() This is needed for Python 3 compatibility. Differential Revision: https://phab.mercurial-scm.org/D7212