perf: add functions to get vfs-like object for Mercurial earlier than 2.3 Before this patch, using svfs prevents perf.py from measuring performance of Mercurial earlier than 2.3 (or 7034365089bf), because svfs isn't available in such Mercurial, even though there are some code paths for Mercurial earlier than 2.3 in perf.py. For example, setting "_prereadsize" attribute in perfindex() and perfnodelookup() is effective only with hg earlier than 1.8 (or 61c9bc3da402). To get appropriate vfs-like object to access files under .hg/store, this patch adds getsvfs() (and also getvfs(), for future use). To avoid examining existence of attribute at each repetition while measuring performance, getsvfs() is invoked outside the function to be called repeatedly. This patch also adds check-perf-code.py an extra check entry to detect direct usage of repo.(vfs|svfs|opener|sopener) in perf.py.

perf: avoid actual writing branch cache out correctly Mercurial 2.5 (or 9b6ae29d4801) introduced "perfbranchmap" command, and tried to avoid actual writing branch cache out by replacing write() of branchcache class in branchmap.py with no-op function (probably, for elimination of noisy and heavy file I/O factor). But its implementation isn't correct, because 9b6ae29d4801 replaced not branchmap.branchcache.write() but branchmap.write(). The latter doesn't exist, even at that change. To avoid actual writing branch cache out correctly, this patch replaces branchmap.branchcache.write() with no-op function. To detect mistake of replacement or change of API in the future quickly, this patch uses safeattrsetter() instead of direct attribute assignment. For similarity between replacements, this patch also changes replacement of branchmap.read(). In this patch, replacement of read()/write() can run safely outside "try" block, because two safeattrsetter() invocations ensure that replacement doesn't cause exception. FYI, the table below compares "base" filter wall time of perfbranchmap on recent mozilla-central repo with each Mercurial version between before and after this patch. ==== ========= ========= ver before after ==== ========= ========= 2.5 18.492334 18.232455 2.6 18.733858 18.156702 2.7 18.245598 18.349210 2.8 18.289070 18.528422 2.9 17.572742 16.989655 3.0 17.406953 17.615012 3.1 17.228419 17.689805 3.2 17.862961 17.718367 3.3 2.632110 2.707960 3.4 3.285683 3.272060 3.5 3.370141 3.352176 3.6 3.366939 3.242455 3.7 3.300778 3.367328 3.8 3.300132 3.267298 3.9 3.418996 3.370265 ==== ========= ========= IMHO, there is no serious overlooking performance regression.

perf: get subsettable from appropriate module for Mercurial earlier than 2.9 Before this patch, using branchmap.subsettable prevents perfbranchmap from measuring performance of Mercurial earlier than 2.9 (or 175c6fd8cacc), because 175c6fd8cacc moved subsettable from repoview.py to branchmap.py, even though there are some code paths for Mercurial earlier than 2.9 in perf.py. For example, setting "_prereadsize" attribute in perfindex() and perfnodelookup() is effective only with hg earlier than 1.8 (or 61c9bc3da402). To get subsettable from appropriate module, this patch examines existence of subsettable in branchmap and repoview. This patch also adds check-perf-code.py an extra check entry to detect direct usage of subsettable attribute in perf.py.

perf: introduce safeattrsetter to replace direct attribute assignment Referring not-existing attribute immediately causes failure, but assigning a value to such attribute doesn't. For example, perf.py has code paths below, which assign a value to not-existing attribute. This causes incorrect performance measurement, but these code paths are executed successfully. - "repo._tags = None" in perftags() recent Mercurial has tags cache information in repo._tagscache - "branchmap.write = lambda repo: None" in perfbranchmap() branchmap cache is written out by branchcache.write() in branchmap.py "util.safehasattr() before assignment" can avoid this issue, but might increase mistake at "copy & paste" attribute name or so. To centralize (1) examining existence of, (2) assigning a value to, and (3) restoring an old value to the attribute, this patch introduces safeattrsetter(). This is used to replace direct attribute assignment in subsequent patches. Encapsulation of restoring is needed to completely remove direct attribute assignment from perf.py, even though restoring isn't needed so often.

largefiles: use context for file closing Make the code slightly smaller and safer (and more deeply indented).

largefiles: when setting/clearing x bit on largefiles, don't change other bits It is only the X bit that it matters to copy from the standin to the largefile in the working directory. While it generally doesn't do any harm to copy the whole mode, it is also "wrong" to copy more than the X bit we care about. It can make a difference if someone should try to handle largefiles differently, such as marking them read-only. Thus, do similar to what utils.setflags does and set the X bit where there are R bits and obey umask.

eol: on update, only re-check files if filtering changed Before, update would mark all files as 'normallookup' in dirstate if .hgeol changed so all files would get the new filtering applied. That takes some time ... and is pointless if the filtering for that file didn't change. Instead, keep track of the old filtering and only check files where the filtering is changed. To keep the old filtering, change to write the applied .hgeol content to .hg/eol.cache instead of just touching it. That change is backwards/forwards compatible. In a real world test, this takes an update that is changing .hgeol and 30000 files from 12s to 4s - where the remaining eol overhead is 1-2s.

dirs: add comment about _PyBytes_Resize So readers have a canonical function to compare this code to.

checkcopies: extract the '_related' closure There is not need for it to be a closure.

checkcopies: add an inline comment about the '_related' call This helps understanding the flow of the function.

checkcopies: minor change to comment This helped me understand the refactoring so this must be helpful.

checkcopies: rename 'ca' to 'base' This variable was named after the common ancestor. It is actually the merge base that might differ from the common ancestor in the graft case. We rename the variable before a larger refactoring to clarify the situation.

bisect: extra a small initialisation outside of a loop Having initialisation done during the first iteration is cute, but can be avoided.

pycompat: only accept a bytestring filepath in Python 2

py3: use raw strings in line continuation (convert ext) Our py2 to py3 string translations marks those as bytestrings.

py3: namedtuple takes unicode (journal ext) namedtuple usage consistent with changelog.py:141

debuginstall: use %d instead of %s for formatting an int % formatting on bytes on Python 3 is pickier about which % character we specify.

py3: test to check which commands run This test helps us to keep track on the commands which runs to Python 3. The full traceback is hidden. Thanks to Augie and Martijn to wrap it up in four lines.

bisect: build a displayer only once There is multiple spot using this, building it early will help to extract more of the logic into the bisect module.

bisect: factor commonly update sequence For now, This remains a closure in the module to avoid circular import with used module.

bisect: move check_state into the bisect module Now that the function is simpler, we resume our quest to move the logic into the bisect module. In the process, we add basic documentation.

bisect: simplify conditional in 'check_state' Now that extra code about "updating" flag have been removed, we can simplify the condition flow and remove a level.

bisect: remove code about "update-flag" in check_state Now that the flag dedicated to updating the flag are handled earlier, we do not need to handle them in the 'check_state' function.

bisect: rename 'check_code' to match our naming scheme We need to do it early, otherwise 'check-commit' will complain every time we touch it.

bisect: minor movement of code handle flag updating state The code flag handling is quite complicated, we are moving code around to prepare further simplification.

test-clone: discard lock-related messages We can't predict where those will show up and they're not super-important for the contents of this particular test, so just drop them. Further reduces the flakiness of the test to zero.

test-clone: fix some instability in pooled clone race condition test Healthy output (one log file mentioning "existing pooled" and one mentioning "new pooled") will now print in a stable order, but unhealthy output will print some sort of error. This reduces the flakiness of the test from 55% to 38%. My next patch makes it completely stable.

py3: add an os.fsencode backport to ease path handling

py3: a second argument to open can't be bytes This fixes open(filename, 'r'), open(filename, 'w'), etc. calls. In Python 3, that second argument *must* be a string, you can't use bytes. The fix is the same as used with getattr() (where the second argument must also always be a string); in the tokenizer, where we detect calls, if there is something that looks like a call to open (and is not an attribute, so the previous token is not a "." dot) then make sure that that second argument is not converted to a `bytes` object instead. There is some remaining issue where the current transformer will also rewrite open(f('foo')). However this also affect function for which we perform similar rewrite ('getattr', 'setattr', 'hasattr', 'safehasattr') and will be dealt with in a follow up.

py3: make check-py3-compat.py import importlib only if necessary importlib isn't available on Python 2.6, and it isn't necessary for Py2 checks.

templater: handle division by zero in arithmetic For now, just turn it to an abort.

templater: provide arithmetic operations on integers The termwidth template keyword is of limited use without some way to ensure that margins are respected. Provide a full set of arithmetic operators (four basic operations plus the mod function, defined to match Python's // for division), so that you can create termwidth based layouts that match the user's terminal size

eol: store and reuse pattern matchers instead of creating in tight loop More "right" and more efficient.

eol: fix variable naming - call it _eolmatch instead of _eolfile It is not the file but a match object based on it.

parsers: move PyInt aliasing out of util.h The PyInt aliasing is only used by parsers.c. Since we don't want to encourage the use of PyInt parsing, move the aliasing to parsers.c.

osutil: use PyLongObject on Python 3 for listdir_slot This code looks performance sensitive. So let's retain PyIntObject on Python 2 and use PyLongObject explicitly on Python 3.

osutil: use PyLongObject in recvfds PyIntObject doesn't exist in Python 3. While PyIntObject is preferred on Python 2 because it is a fixed capacity and faster, the difference between PyIntObject and PyLongObject for scenarios where performance isn't critical or the caller isn't performing type checking shouldn't be relevant. So change recvfds to return a list of longs instead of ints on Python 2.

py3: use encoding.environ instead of os.environ This complains while running hg version on Python 3.5

store: py26 compat, don't use a dict comprehension

dirs: document performance reasons for bypassing Python C API So someone isn't tempted to change it.

dirs: port PyInt code to work on Python 3 PyIntObject no longer exists in Python 3. Instead, there is PyLongObject. Furthermore, PyInt_AS_LONG is a macro referencing a struct member. PyInt_AS_LONG doesn't exist in Python 3 and PyLong_AS_LONG is a #define for PyLong_AsLong, which is a function. So assigning to the return value of PyLong_AS_LONG doesn't work. This patch introduces a macro for obtaining the value of an integer-like type that works on Python 2 and Python 3. On Python 3, we access the struct field of the underlying PyLongObjet directly, without overflow checking. This is essentially the same as what Python 2 was doing except using a PyLong instead of a PyInt.

dirs: convert PyString to PyBytes PyStringObject was renamed to PyBytes in Python 3 along with the corresponding PyString* functions and macros. PyString* doesn't exist in Python 3. But PyBytes* is an alias to PyString in Python 2. So rewrite PyString* to PyBytes* for Python 2/3 dual compatibility.

dirs: inline string macros The old code happened to work because of how the macro was defined. This no longer works in Python 3. Furthermore, assigning to a macro just feels weird. So just inline the macro.

parsers: use PyVarObject_HEAD_INIT The macro changed slightly in Python 3, introducing curly brackets that somehow confuse Clang into issuing a ton of compiler warnings. Using PyVarObject_HEAD_INIT makes these go away. It's worth noting that the code is identical: the 2nd argument to PyVarObject_HEAD_INIT is assigned to the ob_size field and is inserted immediately after "PyObject_HEAD_INIT(type)" is generated. Compilers are weird.

pathencode: use Py_SIZE directly On Python 2, PyBytes_GET_SIZE is the same as PyString_GET_SIZE which is the same as Py_SIZE which resolves to a struct member. On Python 3, PyBytes_GET_SIZE is "(assert(PyBytes_Check(op)),Py_SIZE(op))". The compiler barfs when assigning to this version. This patch simply changes PyBytes_GET_SIZE to Py_SIZE. On Python 2, there is no effective change in behavior. On Python 3, we drop the PyBytes_Check(). However, in all cases we have explicitly created a PyBytesObject in the same function, so the PyBytes_Check() is guaranteed to be true. Despite this, code changes over time, so I've added added assert() in all callers so we can catch this in debug builds. With this patch, all mercurial.* C extensions now compile on Python 3 on my OS X machine. There are several compiler warnings and I'm sure there are incompatibilities with Python 3, including possibly segfaults. But it is a milestone.

util: remove PyString* aliases on Python 3 We no longer have any users of the legacy PyString* functions. We no longer need these redefinitions. After this change, the only reference to "PyString" in the repo is in watchman's C extension. That isn't our code and porting Mercurial extensions to Python 3 is not a high priority at the moment. watchman's C extension will be dealt with later.

parsers: convert PyString* to PyBytes* With this change, we no longer have any occurrences of "PyString" in our C extensions.

pathencode: convert PyString* to PyBytes*