hg: add support for HGUNICODEPEDANTRY environment variable This lets us easily verify that there are no implicit conversions between unicodes and bytes in Mercurial's codebase. Based on something mpm did by hand periodically, but it kept regressing, so just open the door to running it in a buildbot.

dirstate.status: assign members one by one instead of unpacking the tuple With this patch, hg status and hg diff regain their previous speed. The following tests are run against a working copy with over 270,000 files. Here, 'before' means without this or the previous patch applied. Note that in this case `hg perfstatus` isn't representative since it doesn't take dirstate parsing time into account. $ time hg status # best of 5 before: 2.03s user 1.25s system 99% cpu 3.290 total after: 2.01s user 1.25s system 99% cpu 3.261 total $ time hg diff # best of 5 before: 1.32s user 0.78s system 99% cpu 2.105 total after: 1.27s user 0.79s system 99% cpu 2.066 total

dirstate.status: assign members one by one instead of unpacking the tuple With this patch, hg status and hg diff regain their previous speed. The following tests are run against a working copy with over 270,000 files. Here, 'before' means without this or the previous patch applied. Note that in this case `hg perfstatus` isn't representative since it doesn't take dirstate parsing time into account. $ time hg status # best of 5 before: 2.03s user 1.25s system 99% cpu 3.290 total after: 2.01s user 1.25s system 99% cpu 3.261 total $ time hg diff # best of 5 before: 1.32s user 0.78s system 99% cpu 2.105 total after: 1.27s user 0.79s system 99% cpu 2.066 total

parsers: inline fields of dirstate values in C version Previously, while unpacking the dirstate we'd create 3-4 new CPython objects for most dirstate values: - the state is a single character string, which is pooled by CPython - the mode is a new object if it isn't 0 due to being in the lookup set - the size is a new object if it is greater than 255 - the mtime is a new object if it isn't -1 due to being in the lookup set - the tuple to contain them all In some cases such as regular hg status, we actually look at all the objects. In other cases like hg add, hg status for a subdirectory, or hg status with the third-party hgwatchman enabled, we look at almost none of the objects. This patch eliminates most object creation in these cases by defining a custom C struct that is exposed to Python with an interface similar to a tuple. Only when tuple elements are actually requested are the respective objects created. The gains, where they're expected, are significant. The following tests are run against a working copy with over 270,000 files. parse_dirstate becomes significantly faster: $ hg perfdirstate before: wall 0.186437 comb 0.180000 user 0.160000 sys 0.020000 (best of 35) after: wall 0.093158 comb 0.100000 user 0.090000 sys 0.010000 (best of 95) and as a result, several commands benefit: $ time hg status # with hgwatchman enabled before: 0.42s user 0.14s system 99% cpu 0.563 total after: 0.34s user 0.12s system 99% cpu 0.471 total $ time hg add new-file before: 0.85s user 0.18s system 99% cpu 1.033 total after: 0.76s user 0.17s system 99% cpu 0.931 total There is a slight regression in regular status performance, but this is fixed in an upcoming patch.

dirstate: add dirstatetuple to create dirstate values Upcoming patches will switch away from using Python tuples for dirstate values in compiled builds. Make that easier by introducing a variable called dirstatetuple, currently set to tuple. In upcoming patches, this will be set to an object from the parsers module.

parsers: remove no longer used dirstate_unset

pack_dirstate: in C version, for invalidation set dict to what we write to disk For files written out in the last second, Mercurial used to invalidate all the stat data (state, size, mode, mtime) while persisting to disk. This included invalidating the data in the dirstate dict as well. In commit 187bf2dde7c1, this was found to be unnecessary, and Mercurial switched to invalidating only the mtime. However, in the C version of pack_dirstate the value set in the dict was still the fully invalidated one. Switch to invalidating just the mtime in the dict as well.

record: update comment to match code Commit 205599e31870 changed how newfiles get passed to commitfunc, but did not change the corresponding comment that explains this. This commit also updates this comment.