Mercurial > hg
annotate contrib/hgfixes/fix_leftover_imports.py @ 25708:d3d32643c060
wireproto: correctly escape batched args and responses (issue4739)
This issue appears to be as old as wireproto batching itself: I can
reproduce the failure as far back as 08ef6b5f3715 trivially by
rebasing the test changes in this patch, which was back in the 1.9
era. I didn't test before that change, because prior to that the
testfile has a different name and I'm lazy.
Note that the test thought it was checking this case, but it actually
wasn't: it put a literal ; in the arg and response for its greet
command, but the mangle/unmangle step defined in the test meant that
instead of "Fo, =;o" going over the wire, "Gp-!><p" went instead,
which doesn't contain any special characters (those being [.=;]) and
thus not exercising the escaping. The test has been updated to use
pre-unmangled special characters, so the request is now "Fo+<:o",
which mangles to "Gp,=;p". I have confirmed that the test fails
without the adjustment to the escaping rules in wireproto.py.
No existing clients of RPC batching were depending on the old behavior
in any way. The only *actual* users of batchable RPCs in core were:
1) largefiles, wherein it batches up many statlfile calls. It sends
hexlified hashes over the wire and gets a 0, 1, or 2 back as a
response. No risk of special characters.
2) setdiscovery, which was using heads() and known(), both of which
communicate via hexlified nodes. Again, no risk of special characters.
Since the escaping functionality has been completely broken since it
was introduced, we know that it has no users. As such, we can change
the escaping mechanism without having to worry about backwards
compatibility issues.
For the curious, this was detected by chance: it happens that the
lz4-compressed text of a test file for remotefilelog compressed to
something containing a ;, which then caused the failure when I moved
remotefilelog to using batching for file content fetching.
author | Augie Fackler <augie@google.com> |
---|---|
date | Tue, 30 Jun 2015 19:19:17 -0400 |
parents | 9de689d20230 |
children |
rev | line source |
---|---|
11949
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
1 "Fixer that translates some APIs ignored by the default 2to3 fixers." |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
2 |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
3 # FIXME: This fixer has some ugly hacks. Its main design is based on that of |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
4 # fix_imports, from lib2to3. Unfortunately, the fix_imports framework only |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
5 # changes module names "without dots", meaning it won't work for some changes |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
6 # in the email module/package. Thus this fixer was born. I believe that with a |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
7 # bit more thinking, a more generic fixer can be implemented, but I'll leave |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
8 # that as future work. |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
9 |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
10 from lib2to3.fixer_util import Name |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
11 from lib2to3.fixes import fix_imports |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
12 |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
13 # This maps the old names to the new names. Note that a drawback of the current |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
14 # design is that the dictionary keys MUST have EXACTLY one dot (.) in them, |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
15 # otherwise things will break. (If you don't need a module hierarchy, you're |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
16 # better of just inherit from fix_imports and overriding the MAPPING dict.) |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
17 |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
18 MAPPING = {'email.Utils': 'email.utils', |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
19 'email.Errors': 'email.errors', |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
20 'email.Header': 'email.header', |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
21 'email.Parser': 'email.parser', |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
22 'email.Encoders': 'email.encoders', |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
23 'email.MIMEText': 'email.mime.text', |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
24 'email.MIMEBase': 'email.mime.base', |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
25 'email.Generator': 'email.generator', |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
26 'email.MIMEMultipart': 'email.mime.multipart', |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
27 } |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
28 |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
29 def alternates(members): |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
30 return "(" + "|".join(map(repr, members)) + ")" |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
31 |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
32 def build_pattern(mapping=MAPPING): |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
33 packages = {} |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
34 for key in mapping: |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
35 # What we are doing here is the following: with dotted names, we'll |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
36 # have something like package_name <trailer '.' module>. Then, we are |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
37 # making a dictionary to copy this structure. For example, if |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
38 # mapping={'A.B': 'a.b', 'A.C': 'a.c'}, it will generate the dictionary |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
39 # {'A': ['b', 'c']} to, then, generate something like "A <trailer '.' |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
40 # ('b' | 'c')". |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
41 name = key.split('.') |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
42 prefix = name[0] |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
43 if prefix in packages: |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
44 packages[prefix].append(name[1:][0]) |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
45 else: |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
46 packages[prefix] = name[1:] |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
47 |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
48 mod_list = ' | '.join(["'%s' '.' ('%s')" % |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
49 (key, "' | '".join(packages[key])) for key in packages]) |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
50 mod_list = '(' + mod_list + ' )' |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
51 |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
52 yield """name_import=import_name< 'import' module_name=dotted_name< %s > > |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
53 """ % mod_list |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
54 |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
55 yield """name_import=import_name< 'import' |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
56 multiple_imports=dotted_as_names< any* |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
57 module_name=dotted_name< %s > |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
58 any* > |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
59 >""" % mod_list |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
60 |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
61 packs = ' | '.join(["'%s' trailer<'.' ('%s')>" % (key, |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
62 "' | '".join(packages[key])) for key in packages]) |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
63 |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
64 yield "power< package=(%s) trailer<'.' any > any* >" % packs |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
65 |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
66 class FixLeftoverImports(fix_imports.FixImports): |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
67 # We want to run this fixer after fix_import has run (this shouldn't matter |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
68 # for hg, though, as setup3k prefers to run the default fixers first) |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
69 mapping = MAPPING |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
70 |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
71 def build_pattern(self): |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
72 return "|".join(build_pattern(self.mapping)) |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
73 |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
74 def transform(self, node, results): |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
75 # Mostly copied from fix_imports.py |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
76 import_mod = results.get("module_name") |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
77 if import_mod: |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
78 try: |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
79 mod_name = import_mod.value |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
80 except AttributeError: |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
81 # XXX: A hack to remove whitespace prefixes and suffixes |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
82 mod_name = str(import_mod).strip() |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
83 new_name = self.mapping[mod_name] |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
84 import_mod.replace(Name(new_name, prefix=import_mod.prefix)) |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
85 if "name_import" in results: |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
86 # If it's not a "from x import x, y" or "import x as y" import, |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
87 # marked its usage to be replaced. |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
88 self.replace[mod_name] = new_name |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
89 if "multiple_imports" in results: |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
90 # This is a nasty hack to fix multiple imports on a line (e.g., |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
91 # "import StringIO, urlparse"). The problem is that I can't |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
92 # figure out an easy way to make a pattern recognize the keys of |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
93 # MAPPING randomly sprinkled in an import statement. |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
94 results = self.match(node) |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
95 if results: |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
96 self.transform(node, results) |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
97 else: |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
98 # Replace usage of the module. |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
99 # Now this is, mostly, a hack |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
100 bare_name = results["package"][0] |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
101 bare_name_text = ''.join(map(str, results['package'])).strip() |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
102 new_name = self.replace.get(bare_name_text) |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
103 prefix = results['package'][0].prefix |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
104 if new_name: |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
105 bare_name.replace(Name(new_name, prefix=prefix)) |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
106 results["package"][1].replace(Name('')) |
626fe5c99231
hgfixes: added a fixer to convert changes in the email package
Renato Cunha <renatoc@gmail.com>
parents:
diff
changeset
|
107 |