cache: avoid warming the fnodetags cache after clone
That cache can quite expensive to compute on large repository as not that `hg
clone` is warming all cache, this can introduces a significant slowdown for
clone time[1]. As a stop gap measure introduce a quick fix for
that on stable, skipping the fnodetags cache post-clone.
[1] https://www.mercurial-scm.org/pipermail/mercurial/2021-April/052679.html
Differential Revision: https://phab.mercurial-scm.org/D10695
--- a/mercurial/hg.py Wed May 05 17:47:30 2021 -0400
+++ b/mercurial/hg.py Fri May 07 10:39:58 2021 +0200
@@ -1053,7 +1053,7 @@
# as the only "bad" outcome would be some slowness. That potential
# slowness already affect reader.
with destrepo.lock():
- destrepo.updatecaches(full=True)
+ destrepo.updatecaches(full=b"post-clone")
finally:
release(srclock, destlock)
if cleandir is not None:
--- a/mercurial/localrepo.py Wed May 05 17:47:30 2021 -0400
+++ b/mercurial/localrepo.py Fri May 07 10:39:58 2021 +0200
@@ -2727,6 +2727,11 @@
If 'full' is set, make sure all caches the function knows about have
up-to-date data. Even the ones usually loaded more lazily.
+
+ The `full` argument can take a special "post-clone" value. In this case
+ the cache warming is made after a clone and of the slower cache might
+ be skipped, namely the `.fnodetags` one. This argument is 5.8 specific
+ as we plan for a cleaner way to deal with this for 5.9.
"""
if tr is not None and tr.hookargs.get(b'source') == b'strip':
# During strip, many caches are invalid but
@@ -2754,8 +2759,9 @@
for ctx in self[b'.'].parents():
ctx.manifest() # accessing the manifest is enough
- # accessing fnode cache warms the cache
- tagsmod.fnoderevs(self.ui, unfi, unfi.changelog.revs())
+ if not full == b"post-clone":
+ # accessing fnode cache warms the cache
+ tagsmod.fnoderevs(self.ui, unfi, unfi.changelog.revs())
# accessing tags warm the cache
self.tags()
self.filtered(b'served').tags()
--- a/tests/test-clone-uncompressed.t Wed May 05 17:47:30 2021 -0400
+++ b/tests/test-clone-uncompressed.t Fri May 07 10:39:58 2021 +0200
@@ -201,7 +201,6 @@
branch2-served.hidden
branch2-visible
branch2-visible-hidden
- hgtagsfnodes1
rbc-names-v1
rbc-revs-v1
tags2
--- a/tests/test-clone.t Wed May 05 17:47:30 2021 -0400
+++ b/tests/test-clone.t Fri May 07 10:39:58 2021 +0200
@@ -62,7 +62,6 @@
branch2-served.hidden
branch2-visible
branch2-visible-hidden
- hgtagsfnodes1
rbc-names-v1
rbc-revs-v1
tags2
@@ -142,7 +141,6 @@
branch2-served.hidden
branch2-visible
branch2-visible-hidden
- hgtagsfnodes1
rbc-names-v1
rbc-revs-v1
tags2
--- a/tests/test-hardlinks.t Wed May 05 17:47:30 2021 -0400
+++ b/tests/test-hardlinks.t Fri May 07 10:39:58 2021 +0200
@@ -244,7 +244,6 @@
2 r4/.hg/cache/branch2-served.hidden
2 r4/.hg/cache/branch2-visible
2 r4/.hg/cache/branch2-visible-hidden
- 2 r4/.hg/cache/hgtagsfnodes1
2 r4/.hg/cache/rbc-names-v1
2 r4/.hg/cache/rbc-revs-v1
2 r4/.hg/cache/tags2
@@ -302,7 +301,6 @@
2 r4/.hg/cache/branch2-served.hidden
2 r4/.hg/cache/branch2-visible
2 r4/.hg/cache/branch2-visible-hidden
- 2 r4/.hg/cache/hgtagsfnodes1
2 r4/.hg/cache/rbc-names-v1
2 r4/.hg/cache/rbc-revs-v1
2 r4/.hg/cache/tags2
--- a/tests/test-share.t Wed May 05 17:47:30 2021 -0400
+++ b/tests/test-share.t Fri May 07 10:39:58 2021 +0200
@@ -68,7 +68,6 @@
branch2-served.hidden
branch2-visible
branch2-visible-hidden
- hgtagsfnodes1
rbc-names-v1
rbc-revs-v1
tags2
--- a/tests/test-ssh.t Wed May 05 17:47:30 2021 -0400
+++ b/tests/test-ssh.t Fri May 07 10:39:58 2021 +0200
@@ -86,7 +86,7 @@
$ hg -R local-stream book mybook
$ hg clone -e "\"$PYTHON\" \"$TESTDIR/dummyssh\"" --stream ssh://user@dummy/local-stream stream2
streaming all changes
- 16 files to transfer, * of data (glob)
+ 15 files to transfer, * of data (glob)
transferred * in * seconds (*) (glob)
updating to branch default
2 files updated, 0 files merged, 0 files removed, 0 files unresolved
--- a/tests/test-tags.t Wed May 05 17:47:30 2021 -0400
+++ b/tests/test-tags.t Fri May 07 10:39:58 2021 +0200
@@ -807,11 +807,11 @@
$ f --size --hexdump tagsclient/.hg/cache/hgtagsfnodes1
tagsclient/.hg/cache/hgtagsfnodes1: size=96
- 0000: 96 ee 1d 73 00 00 00 00 00 00 00 00 00 00 00 00 |...s............|
- 0010: 00 00 00 00 00 00 00 00 c4 da b0 c2 94 65 e1 c6 |.............e..|
- 0020: 0d f7 f0 dd 32 04 ea 57 78 c8 97 97 79 fc d5 95 |....2..Wx...y...|
- 0030: f6 3c c8 fe 94 65 e1 c6 0d f7 f0 dd 32 04 ea 57 |.<...e......2..W|
- 0040: 78 c8 97 97 79 fc d5 95 40 f0 35 8c 19 e0 a7 d3 |x...y...@.5.....|
+ 0000: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
+ 0010: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
+ 0020: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
+ 0030: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
+ 0040: ff ff ff ff ff ff ff ff 40 f0 35 8c 19 e0 a7 d3 |........@.5.....|
0050: 8a 5c 6a 82 4d cf fb a5 87 d0 2f a3 1e 4f 2f 8a |.\j.M...../..O/.|
Running hg tags should produce tags2* file and not change cache
@@ -837,11 +837,11 @@
$ f --size --hexdump tagsclient/.hg/cache/hgtagsfnodes1
tagsclient/.hg/cache/hgtagsfnodes1: size=96
- 0000: 96 ee 1d 73 00 00 00 00 00 00 00 00 00 00 00 00 |...s............|
- 0010: 00 00 00 00 00 00 00 00 c4 da b0 c2 94 65 e1 c6 |.............e..|
- 0020: 0d f7 f0 dd 32 04 ea 57 78 c8 97 97 79 fc d5 95 |....2..Wx...y...|
- 0030: f6 3c c8 fe 94 65 e1 c6 0d f7 f0 dd 32 04 ea 57 |.<...e......2..W|
- 0040: 78 c8 97 97 79 fc d5 95 40 f0 35 8c 19 e0 a7 d3 |x...y...@.5.....|
+ 0000: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
+ 0010: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
+ 0020: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
+ 0030: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
+ 0040: ff ff ff ff ff ff ff ff 40 f0 35 8c 19 e0 a7 d3 |........@.5.....|
0050: 8a 5c 6a 82 4d cf fb a5 87 d0 2f a3 1e 4f 2f 8a |.\j.M...../..O/.|
Check that the bundle includes cache data
--- a/tests/test-treemanifest.t Wed May 05 17:47:30 2021 -0400
+++ b/tests/test-treemanifest.t Fri May 07 10:39:58 2021 +0200
@@ -792,7 +792,7 @@
$ hg clone --config experimental.changegroup3=True --stream -U \
> http://localhost:$HGPORT1 stream-clone-basicstore
streaming all changes
- 29 files to transfer, * of data (glob)
+ 28 files to transfer, * of data (glob)
transferred * in * seconds (*) (glob)
$ hg -R stream-clone-basicstore verify
checking changesets
@@ -806,7 +806,7 @@
$ hg clone --config experimental.changegroup3=True --stream -U \
> http://localhost:$HGPORT2 stream-clone-encodedstore
streaming all changes
- 29 files to transfer, * of data (glob)
+ 28 files to transfer, * of data (glob)
transferred * in * seconds (*) (glob)
$ hg -R stream-clone-encodedstore verify
checking changesets