view doc/docchecker @ 45022:e3b19004087a stable

convert: correctly convert paths to UTF-8 for Subversion

The previous code using encoding.tolocal() only worked by chance in these
situations:

* The string is ASCII: The fast path was triggered and the string was returned
  unmodified.
* The local encoding is UTF-8: The source and target encoding is the same.
* The string is not valid UTF-8 and the native encoding is ISO-8859-1: If the
  string doesn’t decode using UTF-8, ISO-8859-1 is tried as a fallback. During
  `hg convert`, the local encoding is always UTF-8. The irony is that in this
  case, encoding.tolocal() behaves like what someone would expect the reverse
  function, encoding.fromlocal(), to do.

When the locale encoding is ISO-8859-15, trying to convert a SVN repo `/tmp/a€`
failed before like this:

file:///tmp/a%C2%A4 does not look like a Subversion repository to libsvn version 1.14.0

The correct URL is `file:///tmp/a%E2%82%AC`.

Unlike previously (with the ISO-8859-1 fallback), decoding the path using the
locale encoding can fail. In this case, we have to bail out, as Subversion
won’t be able to do anything useful with the path.

#!/usr/bin/env python
#
# docchecker - look for problematic markup
#
# Copyright 2016 timeless <[e-mail]> and others
#
# This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version.

from __future__ import absolute_import, print_function

import os
import re
import sys

try:
    import msvcrt

    msvcrt.setmode(sys.stdout.fileno(), os.O_BINARY)
    msvcrt.setmode(sys.stderr.fileno(), os.O_BINARY)
except ImportError:
    pass

stdout = getattr(sys.stdout, 'buffer', sys.stdout)

leadingline = re.compile(br'(^\s*)(\S.*)$')

checks = [
    (
        br""":hg:`[^`]*'[^`]*`""",
        b"""warning: please avoid nesting ' in :hg:`...`""",
    ),
    (br'\w:hg:`', b'warning: please have a space before :hg:'),
    (
        br"""(?:[^a-z][^'.])hg ([^,;"`]*'(?!hg)){2}""",
        b'''warning: please use " instead of ' for hg ... "..."''',
    ),
]


def check(line):
    messages = []
    for match, msg in checks:
        if re.search(match, line):
            messages.append(msg)
    if messages:
        stdout.write(b'%s\n' % line)
        for msg in messages:
            stdout.write(b'%s\n' % msg)


def work(file):
    (llead, lline) = (b'', b'')

    for line in file:
        # this section unwraps lines
        match = leadingline.match(line)
        if not match:
            check(lline)
            (llead, lline) = (b'', b'')
            continue

        lead, line = match.group(1), match.group(2)
        if lead == llead:
            if lline != b'':
                lline += b' ' + line
            else:
                lline = line
        else:
            check(lline)
            (llead, lline) = (lead, line)
    check(lline)


def main():
    for f in sys.argv[1:]:
        try:
            with open(f, 'rb') as file:
                work(file)
        except BaseException as e:
            sys.stdout.write(r"failed to process %s: %s\n" % (f, e))


main()