comparison mercurial/encoding.py @ 37946:57b0c7221dba

encoding: fix toutf8b() to resurrect lossy characters even if "\xed" in it

If 's' is a localstr, 's._utf8' must be returned to get the original UTF-8
sequence back. Because of this, it was totally wrong to test if '"\xed" not
in s', which should be either '"\xed" not in s._utf8' or just omitted.

This patch moves the localstr handling to top as the validity of 's._utf8'
should be pre-checked by encoding.tolocal().

37945:bfe8ef6e370e

    arbitrary bytes into an internal Unicode format that can be
    re-encoded back into the original. Here we are exposing the
    internal surrogate encoding as a UTF-8 string.)
    '''

    if not isinstance(s, localstr) and isasciistr(s):
        return s
    if "\xed" not in s:
        if isinstance(s, localstr):
            return s._utf8
        try:
            s.decode('utf-8', _utf8strict)
            return s
        except UnicodeDecodeError:
            pass

37946:57b0c7221dba

    arbitrary bytes into an internal Unicode format that can be
    re-encoded back into the original. Here we are exposing the
    internal surrogate encoding as a UTF-8 string.)
    '''

    if isinstance(s, localstr):
        # assume that the original UTF-8 sequence would never contain
        # invalid characters in U+DCxx range
        return s._utf8
    elif isasciistr(s):
        return s
    if "\xed" not in s:
        try:
            s.decode('utf-8', _utf8strict)
            return s
        except UnicodeDecodeError:
            pass