view tests/test-ui-config.py.out @ 25783:1f6878c87c25

templater: introduce one-pass parsing of nested template strings

Instead of re-parsing quoted strings as templates, the tokenizer can delegate
the parsing of nested template strings to the parser. It has two benefits:

 1. syntax errors can be reported with absolute positions
 2. nested template can use quotes just like shell: "{"{rev}"}"

It doesn't sound nice that the tokenizer recurses into the parser. We could
instead make the tokenize itself recursive, but it would be much more
complicated because we would have to adjust binding strengths carefully and
put dummy infix operators to concatenate template fragments.

Now "string" token without r"" never appears. It will be removed by the next
patch.

[('string', 'string value'), ('bool1', 'true'), ('bool2', 'false'), ('boolinvalid', 'foo'), ('int1', '42'), ('int2', '-42'), ('intinvalid', 'foo')]
[('list1', 'foo'), ('list2', 'foo bar baz'), ('list3', 'alice, bob'), ('list4', 'foo bar baz alice, bob'), ('list5', 'abc d"ef"g "hij def"'), ('list6', '"hello world", "how are you?"'), ('list7', 'Do"Not"Separate'), ('list8', '"Do"Separate'), ('list9', '"Do\\"NotSeparate"'), ('list10', 'string "with extraneous" quotation mark"'), ('list11', 'x, y'), ('list12', '"x", "y"'), ('list13', '""" key = "x", "y" """'), ('list14', ',,,,     '), ('list15', '" just with starting quotation'), ('list16', '"longer quotation" with "no ending quotation'), ('list17', 'this is \\" "not a quotation mark"'), ('list18', '\n \n\nding\ndong')]
---
'string value'
'true'
'false'
None
---
values.string is not a boolean ('string value')
True
False
False
False
True
---
42
-42
---
['foo']
['foo', 'bar', 'baz']
['alice', 'bob']
['foo', 'bar', 'baz', 'alice', 'bob']
['foo', 'bar', 'baz', 'alice', 'bob']
['abc', 'd"ef"g', 'hij def']
['hello world', 'how are you?']
['Do"Not"Separate']
['Do', 'Separate']
['Do"NotSeparate']
['string', 'with extraneous', 'quotation', 'mark"']
['x', 'y']
['x', 'y']
['', ' key = ', 'x"', 'y', '', '"']
[]
['"', 'just', 'with', 'starting', 'quotation']
['longer quotation', 'with', '"no', 'ending', 'quotation']
['this', 'is', '"', 'not a quotation mark']
['ding', 'dong']
[]
[]
['foo']
['foo']
['foo', 'bar']
['foo', 'bar']
['foo bar']
['foo', 'bar']
None
True
boolinvalid
intinvalid