hgtagsfnodescache: inherit fnode from parent when possible
If a changeset does not update the content of `.hgtags`, it means it will use
the same file-node (for `.hgtags`) as its parents. In this case we can
directly reuse the parent's file-node.
We use this property when updating the `hgtagsfnodescache` taking a faster path
if we already have a cached value for the parents of the node we are looking
at.
Doing so provides a large performance boost when looking at a lot of fnodes,
especially on repository with very large manifest:
timing for `tagsmod.fnoderevs(ui, repo, repo.changelog.revs())`
mercurial: (41907 revisions, 1923 files)
before: 6.9 seconds
after: 2.7 seconds (-54%)
pypy: (96266 revisions, 5198 files)
before: 80 seconds
after: 20 seconds (-75%)
mozilla-central: (463411 revisions, 272080 files)
before: 7166.4 seconds
after: 47.8 seconds (-99%, x150 speedup)
On a copy of mozilla-try with about 35K heads ans 1.7M changesets, this moves
the computation from many hours to a couple of minutes, making it more
interesting to do a full warm up of this cache before computing tags (from a
cold cache).
There seems to be other performance low hanging fruits, like avoiding the use of
changectx or a more revision centric logic. However, the new code is fast enough
for my needs right now.
--- a/mercurial/tags.py Mon Mar 11 01:09:38 2019 +0100
+++ b/mercurial/tags.py Mon Mar 11 01:10:20 2019 +0100
@@ -18,6 +18,7 @@
bin,
hex,
nullid,
+ nullrev,
short,
)
from .i18n import _
@@ -718,12 +719,33 @@
if not computemissing:
return None
- # Populate missing entry.
- try:
- fnode = ctx.filenode('.hgtags')
- except error.LookupError:
- # No .hgtags file on this revision.
- fnode = nullid
+ fnode = None
+ cl = self._repo.changelog
+ p1rev, p2rev = cl._uncheckedparentrevs(rev)
+ p1node = cl.node(p1rev)
+ p1fnode = self.getfnode(p1node, computemissing=False)
+ if p2rev != nullrev:
+ # There is some no-merge changeset where p1 is null and p2 is set
+ # Processing them as merge is just slower, but still gives a good
+ # result.
+ p2node = cl.node(p1rev)
+ p2fnode = self.getfnode(p2node, computemissing=False)
+ if p1fnode != p2fnode:
+ # we cannot rely on readfast because we don't know against what
+ # parent the readfast delta is computed
+ p1fnode = None
+ if p1fnode is not None:
+ mctx = ctx.manifestctx()
+ fnode = mctx.readfast().get('.hgtags')
+ if fnode is None:
+ fnode = p1fnode
+ if fnode is None:
+ # Populate missing entry.
+ try:
+ fnode = ctx.filenode('.hgtags')
+ except error.LookupError:
+ # No .hgtags file on this revision.
+ fnode = nullid
self._writeentry(offset, properprefix, fnode)
return fnode
--- a/tests/test-tags.t Mon Mar 11 01:09:38 2019 +0100
+++ b/tests/test-tags.t Mon Mar 11 01:10:20 2019 +0100
@@ -145,7 +145,7 @@
$ hg blackbox -l 6
1970/01/01 00:00:00 bob @b9154636be938d3d431e75a7c906504a079bfe07 (5000)> identify
1970/01/01 00:00:00 bob @b9154636be938d3d431e75a7c906504a079bfe07 (5000)> writing 48 bytes to cache/hgtagsfnodes1
- 1970/01/01 00:00:00 bob @b9154636be938d3d431e75a7c906504a079bfe07 (5000)> 0/1 cache hits/lookups in * seconds (glob)
+ 1970/01/01 00:00:00 bob @b9154636be938d3d431e75a7c906504a079bfe07 (5000)> 0/2 cache hits/lookups in * seconds (glob)
1970/01/01 00:00:00 bob @b9154636be938d3d431e75a7c906504a079bfe07 (5000)> writing .hg/cache/tags2-visible with 1 tags
1970/01/01 00:00:00 bob @b9154636be938d3d431e75a7c906504a079bfe07 (5000)> identify exited 0 after * seconds (glob)
1970/01/01 00:00:00 bob @b9154636be938d3d431e75a7c906504a079bfe07 (5000)> blackbox -l 6
@@ -159,7 +159,7 @@
$ hg blackbox -l 6
1970/01/01 00:00:00 bob @b9154636be938d3d431e75a7c906504a079bfe07 (5000)> identify
1970/01/01 00:00:00 bob @b9154636be938d3d431e75a7c906504a079bfe07 (5000)> not writing .hg/cache/hgtagsfnodes1 because lock cannot be acquired
- 1970/01/01 00:00:00 bob @b9154636be938d3d431e75a7c906504a079bfe07 (5000)> 0/1 cache hits/lookups in * seconds (glob)
+ 1970/01/01 00:00:00 bob @b9154636be938d3d431e75a7c906504a079bfe07 (5000)> 0/2 cache hits/lookups in * seconds (glob)
1970/01/01 00:00:00 bob @b9154636be938d3d431e75a7c906504a079bfe07 (5000)> writing .hg/cache/tags2-visible with 1 tags
1970/01/01 00:00:00 bob @b9154636be938d3d431e75a7c906504a079bfe07 (5000)> identify exited 0 after * seconds (glob)
1970/01/01 00:00:00 bob @b9154636be938d3d431e75a7c906504a079bfe07 (5000)> blackbox -l 6
@@ -363,7 +363,7 @@
$ hg blackbox -l 6
1970/01/01 00:00:00 bob @8dbfe60eff306a54259cfe007db9e330e7ecf866 (5000)> tags
1970/01/01 00:00:00 bob @8dbfe60eff306a54259cfe007db9e330e7ecf866 (5000)> writing 24 bytes to cache/hgtagsfnodes1
- 1970/01/01 00:00:00 bob @8dbfe60eff306a54259cfe007db9e330e7ecf866 (5000)> 2/3 cache hits/lookups in * seconds (glob)
+ 1970/01/01 00:00:00 bob @8dbfe60eff306a54259cfe007db9e330e7ecf866 (5000)> 3/4 cache hits/lookups in * seconds (glob)
1970/01/01 00:00:00 bob @8dbfe60eff306a54259cfe007db9e330e7ecf866 (5000)> writing .hg/cache/tags2-visible with 1 tags
1970/01/01 00:00:00 bob @8dbfe60eff306a54259cfe007db9e330e7ecf866 (5000)> tags exited 0 after * seconds (glob)
1970/01/01 00:00:00 bob @8dbfe60eff306a54259cfe007db9e330e7ecf866 (5000)> blackbox -l 6
@@ -384,7 +384,7 @@
$ hg blackbox -l 6
1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> tags
1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> couldn't write cache/hgtagsfnodes1: [Errno *] * (glob)
- 1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> 2/3 cache hits/lookups in * seconds (glob)
+ 1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> 3/4 cache hits/lookups in * seconds (glob)
1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> writing .hg/cache/tags2-visible with 1 tags
1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> tags exited 0 after * seconds (glob)
1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> blackbox -l 6
@@ -399,7 +399,7 @@
$ hg blackbox -l 6
1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> tags
1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> writing 24 bytes to cache/hgtagsfnodes1
- 1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> 2/3 cache hits/lookups in * seconds (glob)
+ 1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> 3/4 cache hits/lookups in * seconds (glob)
1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> writing .hg/cache/tags2-visible with 1 tags
1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> tags exited 0 after * seconds (glob)
1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> blackbox -l 6
@@ -427,7 +427,7 @@
$ hg blackbox -l 5
1970/01/01 00:00:00 bob @0c192d7d5e6b78a714de54a2e9627952a877e25a (5000)> writing 24 bytes to cache/hgtagsfnodes1
- 1970/01/01 00:00:00 bob @0c192d7d5e6b78a714de54a2e9627952a877e25a (5000)> 2/3 cache hits/lookups in * seconds (glob)
+ 1970/01/01 00:00:00 bob @0c192d7d5e6b78a714de54a2e9627952a877e25a (5000)> 2/4 cache hits/lookups in * seconds (glob)
1970/01/01 00:00:00 bob @0c192d7d5e6b78a714de54a2e9627952a877e25a (5000)> writing .hg/cache/tags2-visible with 1 tags
1970/01/01 00:00:00 bob @0c192d7d5e6b78a714de54a2e9627952a877e25a (5000)> tags exited 0 after * seconds (glob)
1970/01/01 00:00:00 bob @0c192d7d5e6b78a714de54a2e9627952a877e25a (5000)> blackbox -l 5
@@ -445,7 +445,7 @@
$ hg blackbox -l 6
1970/01/01 00:00:00 bob @035f65efb448350f4772141702a81ab1df48c465 (5000)> tags
1970/01/01 00:00:00 bob @035f65efb448350f4772141702a81ab1df48c465 (5000)> writing 24 bytes to cache/hgtagsfnodes1
- 1970/01/01 00:00:00 bob @035f65efb448350f4772141702a81ab1df48c465 (5000)> 2/3 cache hits/lookups in * seconds (glob)
+ 1970/01/01 00:00:00 bob @035f65efb448350f4772141702a81ab1df48c465 (5000)> 3/4 cache hits/lookups in * seconds (glob)
1970/01/01 00:00:00 bob @035f65efb448350f4772141702a81ab1df48c465 (5000)> writing .hg/cache/tags2-visible with 1 tags
1970/01/01 00:00:00 bob @035f65efb448350f4772141702a81ab1df48c465 (5000)> tags exited 0 after * seconds (glob)
1970/01/01 00:00:00 bob @035f65efb448350f4772141702a81ab1df48c465 (5000)> blackbox -l 6