changeset 30118:c19266edd93e

py3: a second argument to open can't be bytes

This fixes open(filename, 'r'), open(filename, 'w'), etc. calls. In Python
3, that second argument *must* be a string, you can't use bytes.

The fix is the same as used with getattr() (where the second argument must
also always be a string); in the tokenizer, where we detect calls, if there
is something that looks like a call to open (and is not an attribute, so
the previous token is not a "." dot) then make sure that that second
argument is not converted to a `bytes` object instead.

There is some remaining issue where the current transformer will also rewrite

  open(f('foo')).

However this also affect function for which we perform similar rewrite
('getattr', 'setattr', 'hasattr', 'safehasattr') and will be dealt with in a
follow up.

--- a/mercurial/__init__.py	Sun Oct 09 17:02:34 2016 +0200
+++ b/mercurial/__init__.py	Sun Oct 09 14:10:01 2016 +0200
@@ -305,6 +305,24 @@
                     except IndexError:
                         pass
 
+                # Bare open call (not an attribute on something else)
+                if (fn == 'open' and not (prevtoken.type == token.OP and
+                                          prevtoken.string == '.')):
+                    try:
+                        # (NAME, 'open')
+                        # (OP, '(')
+                        # (NAME|STRING, 'filename')
+                        # (OP, ',')
+                        # (NAME|STRING, mode)
+                        st = tokens[i + 4]
+                        if (st.type == token.STRING and
+                                st.string[0] in ("'", '"')):
+                            rt = tokenize.TokenInfo(st.type, 'u%s' % st.string,
+                                                    st.start, st.end, st.line)
+                            tokens[i + 4] = rt
+                    except IndexError:
+                        pass
+
                 # It changes iteritems to items as iteritems is not
                 # present in Python 3 world.
                 if fn == 'iteritems':
@@ -319,7 +337,7 @@
     # ``replacetoken`` or any mechanism that changes semantics of module
     # loading is changed. Otherwise cached bytecode may get loaded without
     # the new transformation mechanisms applied.
-    BYTECODEHEADER = b'HG\x00\x04'
+    BYTECODEHEADER = b'HG\x00\x05'
 
     class hgloader(importlib.machinery.SourceFileLoader):
         """Custom module loader that transforms source code.