py3: add an os.fsencode backport to ease path handling

py3: a second argument to open can't be bytes This fixes open(filename, 'r'), open(filename, 'w'), etc. calls. In Python 3, that second argument *must* be a string, you can't use bytes. The fix is the same as used with getattr() (where the second argument must also always be a string); in the tokenizer, where we detect calls, if there is something that looks like a call to open (and is not an attribute, so the previous token is not a "." dot) then make sure that that second argument is not converted to a `bytes` object instead. There is some remaining issue where the current transformer will also rewrite open(f('foo')). However this also affect function for which we perform similar rewrite ('getattr', 'setattr', 'hasattr', 'safehasattr') and will be dealt with in a follow up.

py3: make check-py3-compat.py import importlib only if necessary importlib isn't available on Python 2.6, and it isn't necessary for Py2 checks.

templater: handle division by zero in arithmetic For now, just turn it to an abort.

templater: provide arithmetic operations on integers The termwidth template keyword is of limited use without some way to ensure that margins are respected. Provide a full set of arithmetic operators (four basic operations plus the mod function, defined to match Python's // for division), so that you can create termwidth based layouts that match the user's terminal size

eol: store and reuse pattern matchers instead of creating in tight loop More "right" and more efficient.

eol: fix variable naming - call it _eolmatch instead of _eolfile It is not the file but a match object based on it.

parsers: move PyInt aliasing out of util.h The PyInt aliasing is only used by parsers.c. Since we don't want to encourage the use of PyInt parsing, move the aliasing to parsers.c.

osutil: use PyLongObject on Python 3 for listdir_slot This code looks performance sensitive. So let's retain PyIntObject on Python 2 and use PyLongObject explicitly on Python 3.

osutil: use PyLongObject in recvfds PyIntObject doesn't exist in Python 3. While PyIntObject is preferred on Python 2 because it is a fixed capacity and faster, the difference between PyIntObject and PyLongObject for scenarios where performance isn't critical or the caller isn't performing type checking shouldn't be relevant. So change recvfds to return a list of longs instead of ints on Python 2.

py3: use encoding.environ instead of os.environ This complains while running hg version on Python 3.5

store: py26 compat, don't use a dict comprehension

dirs: document performance reasons for bypassing Python C API So someone isn't tempted to change it.

dirs: port PyInt code to work on Python 3 PyIntObject no longer exists in Python 3. Instead, there is PyLongObject. Furthermore, PyInt_AS_LONG is a macro referencing a struct member. PyInt_AS_LONG doesn't exist in Python 3 and PyLong_AS_LONG is a #define for PyLong_AsLong, which is a function. So assigning to the return value of PyLong_AS_LONG doesn't work. This patch introduces a macro for obtaining the value of an integer-like type that works on Python 2 and Python 3. On Python 3, we access the struct field of the underlying PyLongObjet directly, without overflow checking. This is essentially the same as what Python 2 was doing except using a PyLong instead of a PyInt.

dirs: convert PyString to PyBytes PyStringObject was renamed to PyBytes in Python 3 along with the corresponding PyString* functions and macros. PyString* doesn't exist in Python 3. But PyBytes* is an alias to PyString in Python 2. So rewrite PyString* to PyBytes* for Python 2/3 dual compatibility.

dirs: inline string macros The old code happened to work because of how the macro was defined. This no longer works in Python 3. Furthermore, assigning to a macro just feels weird. So just inline the macro.

parsers: use PyVarObject_HEAD_INIT The macro changed slightly in Python 3, introducing curly brackets that somehow confuse Clang into issuing a ton of compiler warnings. Using PyVarObject_HEAD_INIT makes these go away. It's worth noting that the code is identical: the 2nd argument to PyVarObject_HEAD_INIT is assigned to the ob_size field and is inserted immediately after "PyObject_HEAD_INIT(type)" is generated. Compilers are weird.

pathencode: use Py_SIZE directly On Python 2, PyBytes_GET_SIZE is the same as PyString_GET_SIZE which is the same as Py_SIZE which resolves to a struct member. On Python 3, PyBytes_GET_SIZE is "(assert(PyBytes_Check(op)),Py_SIZE(op))". The compiler barfs when assigning to this version. This patch simply changes PyBytes_GET_SIZE to Py_SIZE. On Python 2, there is no effective change in behavior. On Python 3, we drop the PyBytes_Check(). However, in all cases we have explicitly created a PyBytesObject in the same function, so the PyBytes_Check() is guaranteed to be true. Despite this, code changes over time, so I've added added assert() in all callers so we can catch this in debug builds. With this patch, all mercurial.* C extensions now compile on Python 3 on my OS X machine. There are several compiler warnings and I'm sure there are incompatibilities with Python 3, including possibly segfaults. But it is a milestone.

util: remove PyString* aliases on Python 3 We no longer have any users of the legacy PyString* functions. We no longer need these redefinitions. After this change, the only reference to "PyString" in the repo is in watchman's C extension. That isn't our code and porting Mercurial extensions to Python 3 is not a high priority at the moment. watchman's C extension will be dealt with later.

parsers: convert PyString* to PyBytes* With this change, we no longer have any occurrences of "PyString" in our C extensions.

pathencode: convert PyString* to PyBytes*

osutil: convert PyString* to PyBytes* Continuing the conversion from PyString* to PyBytes*.

manifest: convert PyString* to PyBytes* Python 2.6 introduced PyBytesObject and PyBytes* as aliases for PyStringObject and PyString*. So on Python 2.6+, PyBytes* and PyString* are identical and this patch should be a no-op. On Python 3, PyStringObject is effectively renamed to PyUnicodeObject and PyBytesObject becomes the main type for byte strings. This patch begins the process of mass converting PyString* to PyBytes* so the C extensions use the correct type on Python 3.

merge: update doc of manifestmerge() per 18c2184c27dc p1 was renamed to wctx by 18c2184c27dc.