comparison mercurial/templater.py @ 25676:ec9c258e666d stable

templater: parse \"...\" as string for 2.9.2-3.4 compatibility (issue4733)

As of Mercurial 3.4, there were several syntax rules to process nested
template strings. Unfortunately, they were inconsistent and conflicted
each other.

 a. buildmap() rule
    - template string is _parsed_ as string, and parsed as template
    - <\"> is not allowed in nested template:
      {xs % "{f(\"{x}\")}"} -> parse error
    - template escaping <\{> is handled consistently:
      {xs % "\{x}"} -> escaped
 b. _evalifliteral() rule
    - template string is _interpreted_ as string, and parsed as template
      in crafted environment to avoid double processing of escape sequences
    - <\"> is allowed in nested template:
      {if(x, "{f(\"{x}\")}")}
    - <\{> and escape sequences in string literal in nested template are not
      handled well
 c. pad() rule
    - template string is first interpreted as string, and parsed as template,
      which means escape sequences are processed twice
    - <\"> is allowed in nested template:
      {pad("{xs % \"{x}\"}', 10)}

Because of the issue of template escaping, issue4714, 7298da81f5a9 (in stable)
unified the rule (b) to (a). Then, 576d6c74784b (in default) unified the rule
(c) to (b) = (a). But they disabled the following syntax that was somewhat
considered valid.

  {if(rev, "{if(rev, \"{rev}\")}")}
  {pad("{files % \"{file}\"}", 10)}

So, this patch introduces \"...\" literal to work around the escaped-quoted
nested template strings. Because this parsing rule exists only for the backward
compatibility, it is designed to copy the behavior of old _evalifliteral() as
possible.

Future patches will introduce a better parsing rule similar to a command
substitution of POSIX shells or a string interpolation of Ruby, where extra
escapes won't be necessary at all.

  {pad("{files % "{file}"}", 10)}
        ~~~~~~~~~~~~~~~~~~
        parsed as a template, not as a string

Because <\> character wasn't allowed in a template fragment, this patch won't
introduce more breakages. But the syntax of nested templates are interpreted
differently by people, there might be unknown issues. So if we want, we could
instead remove db7463aa080f, 890845af1ac2 and 7298da81f5a9 from the stable
branch as the bug fixed by these patches existed for longer periods.

554d6fcc3c8, "strip single backslash before quotation mark in quoted template",
should be superseded by this patch. I'll remove it later.

25665:dc05a10e1e45

                    yield ('string', program[s:pos], s)
                    break
                pos += 1
            else:
                raise error.ParseError(_("unterminated string"), s)
        elif c.isalnum() or c in '_':
            s = pos
            pos += 1
            while pos < end: # find end of symbol
                d = program[pos]

25676:ec9c258e666d

                    yield ('string', program[s:pos], s)
                    break
                pos += 1
            else:
                raise error.ParseError(_("unterminated string"), s)
        elif (c == '\\' and program[pos:pos + 2] in (r"\'", r'\"')
              or c == 'r' and program[pos:pos + 3] in (r"r\'", r'r\"')):
            # handle escaped quoted strings for compatibility with 2.9.2-3.4,
            # where some of nested templates were preprocessed as strings and
            # then compiled. therefore, \"...\" was allowed. (issue4733)
            #
            # processing flow of _evalifliteral() at 5ab28a2e9962:
            # outer template string    -> stringify()  -> compiletemplate()
            # ------------------------    ------------    ------------------
            # {f("\\\\ {g(\"\\\"\")}"}    \\ {g("\"")}    [r'\\', {g("\"")}]
            #             ~~~~~~~~
            #             escaped quoted string
            if c == 'r':
                pos += 1
                token = 'rawstring'
            else:
                token = 'string'
            quote = program[pos:pos + 2]
            s = pos = pos + 2
            while pos < end: # find closing escaped quote
                if program.startswith('\\\\\\', pos, end):
                    pos += 4 # skip over double escaped characters
                    continue
                if program.startswith(quote, pos, end):
                    try:
                        # interpret as if it were a part of an outer string
                        data = program[s:pos].decode('string-escape')
                    except ValueError: # unbalanced escapes
                        raise error.ParseError(_("syntax error"), s)
                    yield (token, data, s)
                    pos += 1
                    break
                pos += 1
            else:
                raise error.ParseError(_("unterminated string"), s)
        elif c.isalnum() or c in '_':
            s = pos
            pos += 1
            while pos < end: # find end of symbol
                d = program[pos]