parsers: inline fields of dirstate values in C version
Previously, while unpacking the dirstate we'd create 3-4 new CPython objects
for most dirstate values:
- the state is a single character string, which is pooled by CPython
- the mode is a new object if it isn't 0 due to being in the lookup set
- the size is a new object if it is greater than 255
- the mtime is a new object if it isn't -1 due to being in the lookup set
- the tuple to contain them all
In some cases such as regular hg status, we actually look at all the objects.
In other cases like hg add, hg status for a subdirectory, or hg status with the
third-party hgwatchman enabled, we look at almost none of the objects.
This patch eliminates most object creation in these cases by defining a custom
C struct that is exposed to Python with an interface similar to a tuple. Only
when tuple elements are actually requested are the respective objects created.
The gains, where they're expected, are significant. The following tests are run
against a working copy with over 270,000 files.
parse_dirstate becomes significantly faster:
$ hg perfdirstate
before: wall 0.186437 comb 0.180000 user 0.160000 sys 0.020000 (best of 35)
after: wall 0.093158 comb 0.100000 user 0.090000 sys 0.010000 (best of 95)
and as a result, several commands benefit:
$ time hg status # with hgwatchman enabled
before: 0.42s user 0.14s system 99% cpu 0.563 total
after: 0.34s user 0.12s system 99% cpu 0.471 total
$ time hg add new-file
before: 0.85s user 0.18s system 99% cpu 1.033 total
after: 0.76s user 0.17s system 99% cpu 0.931 total
There is a slight regression in regular status performance, but this is fixed
in an upcoming patch.
dirstate: add dirstatetuple to create dirstate values
Upcoming patches will switch away from using Python tuples for dirstate values
in compiled builds. Make that easier by introducing a variable called
dirstatetuple, currently set to tuple. In upcoming patches, this will be set to
an object from the parsers module.
pack_dirstate: in C version, for invalidation set dict to what we write to disk
For files written out in the last second, Mercurial used to invalidate all the
stat data (state, size, mode, mtime) while persisting to disk. This included
invalidating the data in the dirstate dict as well.
In commit
187bf2dde7c1, this was found to be unnecessary, and Mercurial
switched to invalidating only the mtime. However, in the C version of
pack_dirstate the value set in the dict was still the fully invalidated one.
Switch to invalidating just the mtime in the dict as well.
record: update comment to match code
Commit
205599e31870 changed how newfiles get passed to commitfunc, but
did not change the corresponding comment that explains this. This
commit also updates this comment.
hg: update newly added listdir function of vfs in clone
This change invokes os.listdir() via newly added vfs function in clone.
hg: use vfs functions in clone
This change applies vfs functions in clone. destvfs.exists will invoke
os.path.lexists via lexists of vfs to check for broken symbolic links.