comparison mercurial/pycompat.py @ 44998:f2de8f31cb59

pycompat: use os.fsencode() to re-encode sys.argv

Historically, the previous code made sense, as Py_EncodeLocale() and
fs.fsencode() could possibly use different encodings. However, this is not the
case anymore for Python 3.2, which uses the locale encoding as the filesystem
encoding (this is not true for later Python versions, but see below). See
https://vstinner.github.io/painful-history-python-filesystem-encoding.html for
a source and more background information.

Using os.fsencode() is safer, as the documentation for sys.argv says that it can
be used to get the original bytes. When doing further changes, the Python
developers will take care that this continues to work.

One concrete case where os.fsencode() is more correct is when enabling Python's
UTF-8 mode. Py_DecodeLocale() will use UTF-8 in this case. Our previous code
would have encoded it using the locale encoding (which might be different),
whereas os.fsencode() will encode it with UTF-8.

Since we don’t claim to support the UTF-8 mode, this is not really a bug and the
patch can go to the default branch. It might be a good idea to not commit this
to the stable branch, as it could in theory introduce regressions.

44997:93aa152d4295

if ispy3:
    import builtins
    import codecs
    import functools
    import io
    import locale
    import struct

    if os.name == r'nt' and sys.version_info >= (3, 6):
        # MBCS (or ANSI) filesystem encoding must be used as before.
        # Otherwise non-ASCII filenames in existing repositories would be

    stdout = sys.stdout.buffer
    stderr = sys.stderr.buffer

    if getattr(sys, 'argv', None) is not None:
        # On POSIX, the char** argv array is converted to Python str using
        # Py_DecodeLocale(). The inverse of this is Py_EncodeLocale(), which isn't
        # directly callable from Python code. So, we need to emulate it.
        # Py_DecodeLocale() calls mbstowcs() and falls back to mbrtowc() with
        # surrogateescape error handling on failure. These functions take the
        # current system locale into account. So, the inverse operation is to
        # .encode() using the system locale's encoding and using the
        # surrogateescape error handler. The only tricky part here is getting
        # the system encoding correct, since `locale.getlocale()` can return
        # None. We fall back to the filesystem encoding if lookups via `locale`
        # fail, as this seems like a reasonable thing to do.
        #
        # On Windows, the wchar_t **argv is passed into the interpreter as-is.
        # Like POSIX, we need to emulate what Py_EncodeLocale() would do. But
        # there's an additional wrinkle. What we really want to access is the
        # ANSI codepage representation of the arguments, as this is what

        # encoding, which will pass CP_ACP to the underlying Windows API to
        # produce bytes.
        if os.name == r'nt':
            sysargv = [a.encode("mbcs", "ignore") for a in sys.argv]
        else:

            def getdefaultlocale_if_known():
                try:
                    return locale.getdefaultlocale()
                except ValueError:
                    return None, None

            encoding = (
                locale.getlocale()[1]
                or getdefaultlocale_if_known()[1]
                or sys.getfilesystemencoding()
            )
            sysargv = [a.encode(encoding, "surrogateescape") for a in sys.argv]

    bytechr = struct.Struct('>B').pack
    byterepr = b'%r'.__mod__

    class bytestr(bytes):

44998:f2de8f31cb59

if ispy3:
    import builtins
    import codecs
    import functools
    import io
    import struct

    if os.name == r'nt' and sys.version_info >= (3, 6):
        # MBCS (or ANSI) filesystem encoding must be used as before.
        # Otherwise non-ASCII filenames in existing repositories would be

    stdout = sys.stdout.buffer
    stderr = sys.stderr.buffer

    if getattr(sys, 'argv', None) is not None:
        # On POSIX, the char** argv array is converted to Python str using
        # Py_DecodeLocale(). The inverse of this is Py_EncodeLocale(), which
        # isn't directly callable from Python code. In practice, os.fsencode()
        # can be used instead (this is recommended by Python's documentation
        # for sys.argv).
        #
        # On Windows, the wchar_t **argv is passed into the interpreter as-is.
        # Like POSIX, we need to emulate what Py_EncodeLocale() would do. But
        # there's an additional wrinkle. What we really want to access is the
        # ANSI codepage representation of the arguments, as this is what

        # encoding, which will pass CP_ACP to the underlying Windows API to
        # produce bytes.
        if os.name == r'nt':
            sysargv = [a.encode("mbcs", "ignore") for a in sys.argv]
        else:
            sysargv = [fsencode(a) for a in sys.argv]

    bytechr = struct.Struct('>B').pack
    byterepr = b'%r'.__mod__

    class bytestr(bytes):