Mercurial > hg
annotate contrib/hgfixes/fix_bytesmod.py @ 25783:1f6878c87c25
templater: introduce one-pass parsing of nested template strings
Instead of re-parsing quoted strings as templates, the tokenizer can delegate
the parsing of nested template strings to the parser. It has two benefits:
1. syntax errors can be reported with absolute positions
2. nested template can use quotes just like shell: "{"{rev}"}"
It doesn't sound nice that the tokenizer recurses into the parser. We could
instead make the tokenize itself recursive, but it would be much more
complicated because we would have to adjust binding strengths carefully and
put dummy infix operators to concatenate template fragments.
Now "string" token without r"" never appears. It will be removed by the next
patch.
author | Yuya Nishihara <yuya@tcha.org> |
---|---|
date | Mon, 15 Jun 2015 23:11:35 +0900 |
parents | d20817ac628a |
children |
rev | line source |
---|---|
11749
e627fef94604
hgfixes: added a fixer that makes bytes to be formatted correctly
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
1 """Fixer that changes bytes % whatever to a function that actually formats |
e627fef94604
hgfixes: added a fixer that makes bytes to be formatted correctly
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
2 it.""" |
e627fef94604
hgfixes: added a fixer that makes bytes to be formatted correctly
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
3 |
e627fef94604
hgfixes: added a fixer that makes bytes to be formatted correctly
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
4 from lib2to3 import fixer_base |
e627fef94604
hgfixes: added a fixer that makes bytes to be formatted correctly
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
5 from lib2to3.fixer_util import is_tuple, Call, Comma, Name, touch_import |
e627fef94604
hgfixes: added a fixer that makes bytes to be formatted correctly
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
6 |
e627fef94604
hgfixes: added a fixer that makes bytes to be formatted correctly
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
7 # XXX: Implementing a blacklist in 2to3 turned out to be more troublesome than |
e627fef94604
hgfixes: added a fixer that makes bytes to be formatted correctly
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
8 # blacklisting some modules inside the fixers. So, this is what I came with. |
e627fef94604
hgfixes: added a fixer that makes bytes to be formatted correctly
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
9 |
e627fef94604
hgfixes: added a fixer that makes bytes to be formatted correctly
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
10 blacklist = ['mercurial/demandimport.py', |
e627fef94604
hgfixes: added a fixer that makes bytes to be formatted correctly
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
11 'mercurial/py3kcompat.py', |
e627fef94604
hgfixes: added a fixer that makes bytes to be formatted correctly
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
12 'mercurial/i18n.py', |
e627fef94604
hgfixes: added a fixer that makes bytes to be formatted correctly
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
13 ] |
e627fef94604
hgfixes: added a fixer that makes bytes to be formatted correctly
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
14 |
e627fef94604
hgfixes: added a fixer that makes bytes to be formatted correctly
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
15 def isnumberremainder(formatstr, data): |
e627fef94604
hgfixes: added a fixer that makes bytes to be formatted correctly
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
16 try: |
e627fef94604
hgfixes: added a fixer that makes bytes to be formatted correctly
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
17 if data.value.isdigit(): |
e627fef94604
hgfixes: added a fixer that makes bytes to be formatted correctly
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
18 return True |
e627fef94604
hgfixes: added a fixer that makes bytes to be formatted correctly
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
19 except AttributeError: |
e627fef94604
hgfixes: added a fixer that makes bytes to be formatted correctly
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
20 return False |
e627fef94604
hgfixes: added a fixer that makes bytes to be formatted correctly
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
21 |
e627fef94604
hgfixes: added a fixer that makes bytes to be formatted correctly
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
22 class FixBytesmod(fixer_base.BaseFix): |
e627fef94604
hgfixes: added a fixer that makes bytes to be formatted correctly
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
23 # XXX: There's one case (I suppose) I can't handle: when a remainder |
e627fef94604
hgfixes: added a fixer that makes bytes to be formatted correctly
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
24 # operation like foo % bar is performed, I can't really know what the |
e627fef94604
hgfixes: added a fixer that makes bytes to be formatted correctly
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
25 # contents of foo and bar are. I believe the best approach is to "correct" |
e627fef94604
hgfixes: added a fixer that makes bytes to be formatted correctly
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
26 # the to-be-converted code and let bytesformatter handle that case in |
e627fef94604
hgfixes: added a fixer that makes bytes to be formatted correctly
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
27 # runtime. |
e627fef94604
hgfixes: added a fixer that makes bytes to be formatted correctly
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
28 PATTERN = ''' |
e627fef94604
hgfixes: added a fixer that makes bytes to be formatted correctly
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
29 term< formatstr=STRING '%' data=STRING > | |
e627fef94604
hgfixes: added a fixer that makes bytes to be formatted correctly
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
30 term< formatstr=STRING '%' data=atom > | |
e627fef94604
hgfixes: added a fixer that makes bytes to be formatted correctly
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
31 term< formatstr=NAME '%' data=any > | |
e627fef94604
hgfixes: added a fixer that makes bytes to be formatted correctly
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
32 term< formatstr=any '%' data=any > |
e627fef94604
hgfixes: added a fixer that makes bytes to be formatted correctly
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
33 ''' |
e627fef94604
hgfixes: added a fixer that makes bytes to be formatted correctly
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
34 |
e627fef94604
hgfixes: added a fixer that makes bytes to be formatted correctly
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
35 def transform(self, node, results): |
20399
74daabdf5ab5
fix_bytesmod: fix defects in fix_bytesmod so it produces working code
Augie Fackler <raf@durin42.com>
parents:
19872
diff
changeset
|
36 for bfn in blacklist: |
74daabdf5ab5
fix_bytesmod: fix defects in fix_bytesmod so it produces working code
Augie Fackler <raf@durin42.com>
parents:
19872
diff
changeset
|
37 if self.filename.endswith(bfn): |
74daabdf5ab5
fix_bytesmod: fix defects in fix_bytesmod so it produces working code
Augie Fackler <raf@durin42.com>
parents:
19872
diff
changeset
|
38 return |
74daabdf5ab5
fix_bytesmod: fix defects in fix_bytesmod so it produces working code
Augie Fackler <raf@durin42.com>
parents:
19872
diff
changeset
|
39 if not self.filename.endswith('mercurial/py3kcompat.py'): |
20701
d20817ac628a
fix_bytesmod: use the "from mercurial" form of the import to avoid breaking httpclient
Augie Fackler <raf@durin42.com>
parents:
20399
diff
changeset
|
40 touch_import('mercurial', 'py3kcompat', node=node) |
11749
e627fef94604
hgfixes: added a fixer that makes bytes to be formatted correctly
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
41 |
e627fef94604
hgfixes: added a fixer that makes bytes to be formatted correctly
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
42 formatstr = results['formatstr'].clone() |
e627fef94604
hgfixes: added a fixer that makes bytes to be formatted correctly
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
43 data = results['data'].clone() |
e627fef94604
hgfixes: added a fixer that makes bytes to be formatted correctly
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
44 formatstr.prefix = '' # remove spaces from start |
e627fef94604
hgfixes: added a fixer that makes bytes to be formatted correctly
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
45 |
e627fef94604
hgfixes: added a fixer that makes bytes to be formatted correctly
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
46 if isnumberremainder(formatstr, data): |
e627fef94604
hgfixes: added a fixer that makes bytes to be formatted correctly
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
47 return |
e627fef94604
hgfixes: added a fixer that makes bytes to be formatted correctly
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
48 |
e627fef94604
hgfixes: added a fixer that makes bytes to be formatted correctly
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
49 # We have two possibilities: |
e627fef94604
hgfixes: added a fixer that makes bytes to be formatted correctly
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
50 # 1- An identifier or name is passed, it is going to be a leaf, thus, we |
e627fef94604
hgfixes: added a fixer that makes bytes to be formatted correctly
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
51 # just need to copy its value as an argument to the formatter; |
e627fef94604
hgfixes: added a fixer that makes bytes to be formatted correctly
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
52 # 2- A tuple is explicitly passed. In this case, we're gonna explode it |
e627fef94604
hgfixes: added a fixer that makes bytes to be formatted correctly
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
53 # to pass to the formatter |
e627fef94604
hgfixes: added a fixer that makes bytes to be formatted correctly
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
54 # TODO: Check for normal strings. They don't need to be translated |
e627fef94604
hgfixes: added a fixer that makes bytes to be formatted correctly
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
55 |
e627fef94604
hgfixes: added a fixer that makes bytes to be formatted correctly
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
56 if is_tuple(data): |
e627fef94604
hgfixes: added a fixer that makes bytes to be formatted correctly
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
57 args = [formatstr, Comma().clone()] + \ |
e627fef94604
hgfixes: added a fixer that makes bytes to be formatted correctly
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
58 [c.clone() for c in data.children[:]] |
e627fef94604
hgfixes: added a fixer that makes bytes to be formatted correctly
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
59 else: |
e627fef94604
hgfixes: added a fixer that makes bytes to be formatted correctly
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
60 args = [formatstr, Comma().clone(), data] |
e627fef94604
hgfixes: added a fixer that makes bytes to be formatted correctly
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
61 |
19872
681f7b9213a4
check-code: check for spaces around = for named parameters
Mads Kiilerich <madski@unity3d.com>
parents:
11749
diff
changeset
|
62 call = Call(Name('bytesformatter', prefix=' '), args) |
11749
e627fef94604
hgfixes: added a fixer that makes bytes to be formatted correctly
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
63 return call |