tests/test-highlight

highlight: fixes garbled text in non-UTF-8 environment

This patch treats all files inside repository as encoded by
locale's encoding when pygmentize.

We can assume that most files are written in locale's encoding,
but current implementation treats them as UTF-8.
So there's no way to specify the encoding of files.

Current implementation, db7557359636 (issue1341):
1. Convert original `text`, which is treated as UTF-8, to locale's encoding.
   `encoding.tolocal()` is the method to convert from internal UTF-8 to local.
   If original `text` is not UTF-8, e.g. Japanese EUC-JP, some characters
   become garbled here.
2. pygmentize, with no UnicodeDecodeError.

This patch:
1. Convert original `text`, which is treated as locale's encoding, to unicode.
   Pygments prefers unicode object than raw str. [1]_
   If original `text` is not encoded by locale's encoding, some characters
   become garbled here.
2. pygmentize, also with no UnicodeDecodeError :)
3. Convert unicode back to raw str, which is encoded by locale's.

.. [1] http://pygments.org/docs/unicode/

#!/bin/sh

"$TESTDIR/hghave" pygments || exit 80

cat <<EOF >> $HGRCPATH
[extensions]
hgext.highlight =
[web]
pygments_style = friendly
EOF

hg init test
cd test
# create random Python file to exercise Pygments
cat <<EOF > primes.py
#!/usr/bin/env python

"""Fun with generators. Corresponding Haskell implementation:

primes = 2 : sieve [3, 5..]
    where sieve (p:ns) = p : sieve [n | n <- ns, mod n p /= 0]
"""

from itertools import dropwhile, ifilter, islice, count, chain

def primes():
    """Generate all primes."""
    def sieve(ns):
        p = ns.next()
        # It is important to yield *here* in order to stop the
        # infinite recursion.
        yield p
        ns = ifilter(lambda n: n % p != 0, ns)
        for n in sieve(ns):
            yield n

    odds = ifilter(lambda i: i % 2 == 1, count())
    return chain([2], sieve(dropwhile(lambda n: n < 3, odds)))

if __name__ == "__main__":
    import sys
    try:
        n = int(sys.argv[1])
    except (ValueError, IndexError):
        n = 10
    p = primes()
    print "The first %d primes: %s" % (n, list(islice(p, n)))
EOF

# check for UnicodeDecodeError with iso-8859-1 file contents
python -c 'fp = open("isolatin.txt", "w"); fp.write("h\xFCbsch\n"); fp.close();'

hg ci -Ama

echo % hg serve
hg serve -p $HGPORT -d -n test --pid-file=hg.pid -A access.log -E errors.log
cat hg.pid >> $DAEMON_PIDS

echo % hgweb filerevision, html
("$TESTDIR/get-with-headers.py" localhost:$HGPORT '/file/tip/primes.py') \
    | sed "s/class=\"k\"/class=\"kn\"/g" | sed "s/class=\"mf\"/class=\"mi\"/g"

echo % hgweb filerevision, html
("$TESTDIR/get-with-headers.py" localhost:$HGPORT '/file/tip/isolatin.txt') \
    | sed "s/class=\"k\"/class=\"kn\"/g"

echo % hgweb fileannotate, html
("$TESTDIR/get-with-headers.py" localhost:$HGPORT '/annotate/tip/primes.py') \
    | sed "s/class=\"k\"/class=\"kn\"/g" | sed "s/class=\"mi\"/class=\"mf\"/g"

echo % hgweb fileannotate, raw
("$TESTDIR/get-with-headers.py" localhost:$HGPORT '/annotate/tip/primes.py?style=raw') \
    | sed "s/test@//" > a

echo "200 Script output follows" > b
echo "" >> b
echo "" >> b
hg annotate "primes.py" >> b
echo "" >> b
echo "" >> b
echo "" >> b
echo "" >> b

diff -u b a

echo
echo % hgweb filerevision, raw
("$TESTDIR/get-with-headers.py" localhost:$HGPORT '/file/tip/primes.py?style=raw') \
    > a

echo "200 Script output follows" > b
echo "" >> b
hg cat primes.py >> b

diff -u b a

echo
echo % hgweb highlightcss friendly
"$TESTDIR/get-with-headers.py" localhost:$HGPORT '/highlightcss' > out
head -n 4 out
rm out

echo % errors encountered
cat errors.log
"$TESTDIR/killdaemons.py"

# Change the pygments style
cat > .hg/hgrc <<EOF
[web]
pygments_style = fruity
EOF

echo % hg serve again
hg serve -p $HGPORT -d -n test --pid-file=hg.pid -A access.log -E errors.log
cat hg.pid >> $DAEMON_PIDS

echo % hgweb highlightcss fruity
"$TESTDIR/get-with-headers.py" localhost:$HGPORT '/highlightcss' > out
head -n 4 out
rm out

echo % errors encountered
cat errors.log

cd ..
hg init eucjp
cd eucjp

printf '\265\376\n' >> eucjp.txt  # Japanese kanji "Kyo"

hg ci -Ama

hgserveget () {
    "$TESTDIR/killdaemons.py"
    echo % HGENCODING="$1" hg serve
    HGENCODING="$1" hg serve -p $HGPORT -d -n test --pid-file=hg.pid -E errors.log
    cat hg.pid >> $DAEMON_PIDS

    echo % hgweb filerevision, html
    "$TESTDIR/get-with-headers.py" localhost:$HGPORT "/file/tip/$2" \
        | grep '<div class="parity0 source">' | $TESTDIR/printrepr.py
    echo % errors encountered
    cat errors.log
}

hgserveget euc-jp eucjp.txt
hgserveget utf-8 eucjp.txt
hgserveget us-ascii eucjp.txt