changeset 44651:00e0c5c06ed5

pycompat: change argv conversion semantics Use of os.fsencode() to convert Python's sys.argv back to bytes was not correct because it isn't the logically inverse operation from what CPython was doing under the hood. This commit changes the logic for doing the str -> bytes conversion. This required a separate implementation for POSIX and Windows. The Windows behavior is arguably not ideal. The previous behavior on Windows was leading to failing tests, such as test-http-branchmap.t, which defines a utf-8 branch name via a command argument. Previously, Mercurial's argument parser looked to be receiving wchar_t bytes in some cases. After this commit, behavior on Windows is compatible with Python 2, where CPython did not implement `int wmain()` and Windows was performing a Unicode to ANSI conversion on the wchar_t native command line. Arguably better behavior on Windows would be for Mercurial to preserve the original Unicode sequence coming from Python and to wrap this in a bytes-like type so we can round trip safely. But, this would be new, backwards incompatible behavior. My goal for this commit was to converge Mercurial behavior on Python 3 on Windows to fix busted tests. And I believe I was successful, as this commit fixes 9 tests on my Windows machine and 14 tests in the AWS CI environment! Differential Revision: https://phab.mercurial-scm.org/D8337
author Gregory Szorc <gregory.szorc@gmail.com>
date Sat, 28 Mar 2020 12:18:58 -0700
parents 949a87145336
children 3cbbfd0bfc17
files mercurial/pycompat.py
diffstat 1 files changed, 30 insertions(+), 8 deletions(-) [+]
line wrap: on
line diff
--- a/mercurial/pycompat.py	Wed Mar 18 14:53:53 2020 -0400
+++ b/mercurial/pycompat.py	Sat Mar 28 12:18:58 2020 -0700
@@ -98,6 +98,7 @@
     import codecs
     import functools
     import io
+    import locale
     import struct
 
     if os.name == r'nt' and sys.version_info >= (3, 6):
@@ -148,15 +149,36 @@
     stdout = sys.stdout.buffer
     stderr = sys.stderr.buffer
 
-    # Since Python 3 converts argv to wchar_t type by Py_DecodeLocale() on Unix,
-    # we can use os.fsencode() to get back bytes argv.
-    #
-    # https://hg.python.org/cpython/file/v3.5.1/Programs/python.c#l55
-    #
-    # On Windows, the native argv is unicode and is converted to MBCS bytes
-    # since we do enable the legacy filesystem encoding.
     if getattr(sys, 'argv', None) is not None:
-        sysargv = list(map(os.fsencode, sys.argv))
+        # On POSIX, the char** argv array is converted to Python str using
+        # Py_DecodeLocale(). The inverse of this is Py_EncodeLocale(), which isn't
+        # directly callable from Python code. So, we need to emulate it.
+        # Py_DecodeLocale() calls mbstowcs() and falls back to mbrtowc() with
+        # surrogateescape error handling on failure. These functions take the
+        # current system locale into account. So, the inverse operation is to
+        # .encode() using the system locale's encoding and using the
+        # surrogateescape error handler. The only tricky part here is getting
+        # the system encoding correct, since `locale.getlocale()` can return
+        # None. We fall back to the filesystem encoding if lookups via `locale`
+        # fail, as this seems like a reasonable thing to do.
+        #
+        # On Windows, the wchar_t **argv is passed into the interpreter as-is.
+        # Like POSIX, we need to emulate what Py_EncodeLocale() would do. But
+        # there's an additional wrinkle. What we really want to access is the
+        # ANSI codepage representation of the arguments, as this is what
+        # `int main()` would receive if Python 3 didn't define `int wmain()`
+        # (this is how Python 2 worked). To get that, we encode with the mbcs
+        # encoding, which will pass CP_ACP to the underlying Windows API to
+        # produce bytes.
+        if os.name == r'nt':
+            sysargv = [a.encode("mbcs", "ignore") for a in sys.argv]
+        else:
+            encoding = (
+                locale.getlocale()[1]
+                or locale.getdefaultlocale()[1]
+                or sys.getfilesystemencoding()
+            )
+            sysargv = [a.encode(encoding, "surrogateescape") for a in sys.argv]
 
     bytechr = struct.Struct('>B').pack
     byterepr = b'%r'.__mod__