comparison tests/test-highlight.t @ 26680:7a3f6490ef97

highlight: add option to prevent content-only based fallback

When Mozilla enabled Pygments on hg.mozilla.org, we got a lot of weirdly
colorized files. Upon further investigation, the hightlight extension
is first attempting a filename+content based match then falling back to a
purely content-driven detection mode in Pygments. Sounds good in theory.

Unfortunately, Pygments' content-driven detection establishes no minimum
threshold for returning a lexer. Furthermore, the detection code for
a number of languages is very liberal. For example, ActionScript 3 will
return a confidence of 0.3 (out of 1.0) if the first 1k of the file
we pass in matches the regex "\w+\s*:\s*\w"! Python matches on
"import ". It's no coincidence that a number of our extension-less files
were getting highlighted improperly.

This patch adds an option to have the highlighter not fall back to
purely content-based detection when filename+content detection failed.
This can be enabled to render unlighted text instead of taking the risk
that unknown file types are highlighted incorrectly. The old behavior is
still the default.

26679:0d93df4d1e44

  $ hgserveget us-ascii eucjp.txt
  % HGENCODING=us-ascii hg serve
  % hgweb filerevision, html
  % errors encountered

  $ cd ..

26680:7a3f6490ef97

  $ hgserveget us-ascii eucjp.txt
  % HGENCODING=us-ascii hg serve
  % hgweb filerevision, html
  % errors encountered

We attempt to highlight unknown files by default

  $ killdaemons.py

  $ cat > .hg/hgrc << EOF
  > [web]
  > highlightfiles = **
  > EOF

  $ cat > unknownfile << EOF
  > #!/usr/bin/python
  > def foo():
  >    pass
  > EOF

  $ hg add unknownfile
  $ hg commit -m unknown unknownfile

  $ hg serve -p $HGPORT -d -n test --pid-file=hg.pid
  $ cat hg.pid >> $DAEMON_PIDS

  $ get-with-headers.py localhost:$HGPORT 'file/tip/unknownfile' | grep l2
  <span id="l2"><span class="k">def</span> <span class="nf">foo</span><span class="p">():</span></span><a href="#l2"></a>

We can prevent Pygments from falling back to a non filename-based
detection mode

  $ cat > .hg/hgrc << EOF
  > [web]
  > highlightfiles = **
  > highlightonlymatchfilename = true
  > EOF

  $ killdaemons.py
  $ hg serve -p $HGPORT -d -n test --pid-file=hg.pid
  $ cat hg.pid >> $DAEMON_PIDS
  $ get-with-headers.py localhost:$HGPORT 'file/tip/unknownfile' | grep l2
  <span id="l2">def foo():</span><a href="#l2"></a>

  $ cd ..