Mercurial > hg
annotate tests/test-manifestv2.t @ 29830:92ac2baaea86
revlog: use an LRU cache for delta chain bases
Profiling using statprof revealed a hotspot during changegroup
application calculating delta chain bases on generaldelta repos.
Essentially, revlog._addrevision() was performing a lot of redundant
work tracing the delta chain as part of determining when the chain
distance was acceptable. This was most pronounced when adding
revisions to manifests, which can have delta chains thousands of
revisions long.
There was a delta chain base cache on revlogs before, but it only
captured a single revision. This was acceptable before generaldelta,
when _addrevision would build deltas from the previous revision and
thus we'd pretty much guarantee a cache hit when resolving the delta
chain base on a subsequent _addrevision call. However, it isn't
suitable for generaldelta because parent revisions aren't necessarily
the last processed revision.
This patch converts the delta chain base cache to an LRU dict cache.
The cache can hold multiple entries, so generaldelta repos have a
higher chance of getting a cache hit.
The impact of this change when processing changegroup additions is
significant. On a generaldelta conversion of the "mozilla-unified"
repo (which contains heads of the main Firefox repositories in
chronological order - this means there are lots of transitions between
heads in revlog order), this change has the following impact when
performing an `hg unbundle` of an uncompressed bundle of the repo:
before: 5:42 CPU time
after: 4:34 CPU time
Most of this time is saved when applying the changelog and manifest
revlogs:
before: 2:30 CPU time
after: 1:17 CPU time
That nearly a 50% reduction in CPU time applying changesets and
manifests!
Applying a gzipped bundle of the same repo (effectively simulating a
`hg clone` over HTTP) showed a similar speedup:
before: 5:53 CPU time
after: 4:46 CPU time
Wall time improvements were basically the same as CPU time.
I didn't measure explicitly, but it feels like most of the time
is saved when processing manifests. This makes sense, as large
manifests tend to have very long delta chains and thus benefit the
most from this cache.
So, this change effectively makes changegroup application (which is
used by `hg unbundle`, `hg clone`, `hg pull`, `hg unshelve`, and
various other commands) significantly faster when delta chains are
long (which can happen on repos with large numbers of files and thus
large manifests).
In theory, this change can result in more memory utilization. However,
we're caching a dict of ints. At most we have 200 ints + Python object
overhead per revlog. And, the cache is really only populated when
performing read-heavy operations, such as adding changegroups or
scanning an individual revlog. For memory bloat to be an issue, we'd
need to scan/read several revisions from several revlogs all while
having active references to several revlogs. I don't think there are
many operations that do this, so I don't think memory bloat from the
cache will be an issue.
author | Gregory Szorc <gregory.szorc@gmail.com> |
---|---|
date | Mon, 22 Aug 2016 21:48:50 -0700 |
parents | 2329ca3ebc7a |
children | eb586ed5d8ce |
rev | line source |
---|---|
24935
3035b75cd594
tests: test that we can convert to manifestv2 with clone
Drew Gottlieb <drgott@google.com>
parents:
24573
diff
changeset
|
1 Create repo with old manifest |
3035b75cd594
tests: test that we can convert to manifestv2 with clone
Drew Gottlieb <drgott@google.com>
parents:
24573
diff
changeset
|
2 |
26917
2329ca3ebc7a
test: use generaldelta in test-manifestv2.t
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
24935
diff
changeset
|
3 $ cat << EOF >> $HGRCPATH |
2329ca3ebc7a
test: use generaldelta in test-manifestv2.t
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
24935
diff
changeset
|
4 > [format] |
2329ca3ebc7a
test: use generaldelta in test-manifestv2.t
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
24935
diff
changeset
|
5 > usegeneraldelta=yes |
2329ca3ebc7a
test: use generaldelta in test-manifestv2.t
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
24935
diff
changeset
|
6 > EOF |
2329ca3ebc7a
test: use generaldelta in test-manifestv2.t
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
24935
diff
changeset
|
7 |
24935
3035b75cd594
tests: test that we can convert to manifestv2 with clone
Drew Gottlieb <drgott@google.com>
parents:
24573
diff
changeset
|
8 $ hg init existing |
3035b75cd594
tests: test that we can convert to manifestv2 with clone
Drew Gottlieb <drgott@google.com>
parents:
24573
diff
changeset
|
9 $ cd existing |
3035b75cd594
tests: test that we can convert to manifestv2 with clone
Drew Gottlieb <drgott@google.com>
parents:
24573
diff
changeset
|
10 $ echo footext > foo |
3035b75cd594
tests: test that we can convert to manifestv2 with clone
Drew Gottlieb <drgott@google.com>
parents:
24573
diff
changeset
|
11 $ hg add foo |
3035b75cd594
tests: test that we can convert to manifestv2 with clone
Drew Gottlieb <drgott@google.com>
parents:
24573
diff
changeset
|
12 $ hg commit -m initial |
3035b75cd594
tests: test that we can convert to manifestv2 with clone
Drew Gottlieb <drgott@google.com>
parents:
24573
diff
changeset
|
13 |
3035b75cd594
tests: test that we can convert to manifestv2 with clone
Drew Gottlieb <drgott@google.com>
parents:
24573
diff
changeset
|
14 We're using v1, so no manifestv2 entry is in requires yet. |
3035b75cd594
tests: test that we can convert to manifestv2 with clone
Drew Gottlieb <drgott@google.com>
parents:
24573
diff
changeset
|
15 |
3035b75cd594
tests: test that we can convert to manifestv2 with clone
Drew Gottlieb <drgott@google.com>
parents:
24573
diff
changeset
|
16 $ grep manifestv2 .hg/requires |
3035b75cd594
tests: test that we can convert to manifestv2 with clone
Drew Gottlieb <drgott@google.com>
parents:
24573
diff
changeset
|
17 [1] |
3035b75cd594
tests: test that we can convert to manifestv2 with clone
Drew Gottlieb <drgott@google.com>
parents:
24573
diff
changeset
|
18 |
3035b75cd594
tests: test that we can convert to manifestv2 with clone
Drew Gottlieb <drgott@google.com>
parents:
24573
diff
changeset
|
19 Let's clone this with manifestv2 enabled to switch to the new format for |
3035b75cd594
tests: test that we can convert to manifestv2 with clone
Drew Gottlieb <drgott@google.com>
parents:
24573
diff
changeset
|
20 future commits. |
3035b75cd594
tests: test that we can convert to manifestv2 with clone
Drew Gottlieb <drgott@google.com>
parents:
24573
diff
changeset
|
21 |
3035b75cd594
tests: test that we can convert to manifestv2 with clone
Drew Gottlieb <drgott@google.com>
parents:
24573
diff
changeset
|
22 $ cd .. |
3035b75cd594
tests: test that we can convert to manifestv2 with clone
Drew Gottlieb <drgott@google.com>
parents:
24573
diff
changeset
|
23 $ hg clone --pull existing new --config experimental.manifestv2=1 |
3035b75cd594
tests: test that we can convert to manifestv2 with clone
Drew Gottlieb <drgott@google.com>
parents:
24573
diff
changeset
|
24 requesting all changes |
3035b75cd594
tests: test that we can convert to manifestv2 with clone
Drew Gottlieb <drgott@google.com>
parents:
24573
diff
changeset
|
25 adding changesets |
3035b75cd594
tests: test that we can convert to manifestv2 with clone
Drew Gottlieb <drgott@google.com>
parents:
24573
diff
changeset
|
26 adding manifests |
3035b75cd594
tests: test that we can convert to manifestv2 with clone
Drew Gottlieb <drgott@google.com>
parents:
24573
diff
changeset
|
27 adding file changes |
3035b75cd594
tests: test that we can convert to manifestv2 with clone
Drew Gottlieb <drgott@google.com>
parents:
24573
diff
changeset
|
28 added 1 changesets with 1 changes to 1 files |
3035b75cd594
tests: test that we can convert to manifestv2 with clone
Drew Gottlieb <drgott@google.com>
parents:
24573
diff
changeset
|
29 updating to branch default |
3035b75cd594
tests: test that we can convert to manifestv2 with clone
Drew Gottlieb <drgott@google.com>
parents:
24573
diff
changeset
|
30 1 files updated, 0 files merged, 0 files removed, 0 files unresolved |
3035b75cd594
tests: test that we can convert to manifestv2 with clone
Drew Gottlieb <drgott@google.com>
parents:
24573
diff
changeset
|
31 $ cd new |
3035b75cd594
tests: test that we can convert to manifestv2 with clone
Drew Gottlieb <drgott@google.com>
parents:
24573
diff
changeset
|
32 |
3035b75cd594
tests: test that we can convert to manifestv2 with clone
Drew Gottlieb <drgott@google.com>
parents:
24573
diff
changeset
|
33 Check that entry was added to .hg/requires. |
3035b75cd594
tests: test that we can convert to manifestv2 with clone
Drew Gottlieb <drgott@google.com>
parents:
24573
diff
changeset
|
34 |
3035b75cd594
tests: test that we can convert to manifestv2 with clone
Drew Gottlieb <drgott@google.com>
parents:
24573
diff
changeset
|
35 $ grep manifestv2 .hg/requires |
3035b75cd594
tests: test that we can convert to manifestv2 with clone
Drew Gottlieb <drgott@google.com>
parents:
24573
diff
changeset
|
36 manifestv2 |
3035b75cd594
tests: test that we can convert to manifestv2 with clone
Drew Gottlieb <drgott@google.com>
parents:
24573
diff
changeset
|
37 |
3035b75cd594
tests: test that we can convert to manifestv2 with clone
Drew Gottlieb <drgott@google.com>
parents:
24573
diff
changeset
|
38 Make a new commit. |
3035b75cd594
tests: test that we can convert to manifestv2 with clone
Drew Gottlieb <drgott@google.com>
parents:
24573
diff
changeset
|
39 |
3035b75cd594
tests: test that we can convert to manifestv2 with clone
Drew Gottlieb <drgott@google.com>
parents:
24573
diff
changeset
|
40 $ echo newfootext > foo |
3035b75cd594
tests: test that we can convert to manifestv2 with clone
Drew Gottlieb <drgott@google.com>
parents:
24573
diff
changeset
|
41 $ hg commit -m new |
3035b75cd594
tests: test that we can convert to manifestv2 with clone
Drew Gottlieb <drgott@google.com>
parents:
24573
diff
changeset
|
42 |
3035b75cd594
tests: test that we can convert to manifestv2 with clone
Drew Gottlieb <drgott@google.com>
parents:
24573
diff
changeset
|
43 Check that the manifest actually switched to v2. |
3035b75cd594
tests: test that we can convert to manifestv2 with clone
Drew Gottlieb <drgott@google.com>
parents:
24573
diff
changeset
|
44 |
3035b75cd594
tests: test that we can convert to manifestv2 with clone
Drew Gottlieb <drgott@google.com>
parents:
24573
diff
changeset
|
45 $ hg debugdata -m 0 |
3035b75cd594
tests: test that we can convert to manifestv2 with clone
Drew Gottlieb <drgott@google.com>
parents:
24573
diff
changeset
|
46 foo\x0021e958b1dca695a60ee2e9cf151753204ee0f9e9 (esc) |
3035b75cd594
tests: test that we can convert to manifestv2 with clone
Drew Gottlieb <drgott@google.com>
parents:
24573
diff
changeset
|
47 |
3035b75cd594
tests: test that we can convert to manifestv2 with clone
Drew Gottlieb <drgott@google.com>
parents:
24573
diff
changeset
|
48 $ hg debugdata -m 1 |
3035b75cd594
tests: test that we can convert to manifestv2 with clone
Drew Gottlieb <drgott@google.com>
parents:
24573
diff
changeset
|
49 \x00 (esc) |
3035b75cd594
tests: test that we can convert to manifestv2 with clone
Drew Gottlieb <drgott@google.com>
parents:
24573
diff
changeset
|
50 \x00foo\x00 (esc) |
3035b75cd594
tests: test that we can convert to manifestv2 with clone
Drew Gottlieb <drgott@google.com>
parents:
24573
diff
changeset
|
51 I\xab\x7f\xb8(\x83\xcas\x15\x9d\xc2\xd3\xd3:5\x08\xbad5_ (esc) |
3035b75cd594
tests: test that we can convert to manifestv2 with clone
Drew Gottlieb <drgott@google.com>
parents:
24573
diff
changeset
|
52 |
3035b75cd594
tests: test that we can convert to manifestv2 with clone
Drew Gottlieb <drgott@google.com>
parents:
24573
diff
changeset
|
53 Check that manifestv2 is used if the requirement is present, even if it's |
3035b75cd594
tests: test that we can convert to manifestv2 with clone
Drew Gottlieb <drgott@google.com>
parents:
24573
diff
changeset
|
54 disabled in the config. |
3035b75cd594
tests: test that we can convert to manifestv2 with clone
Drew Gottlieb <drgott@google.com>
parents:
24573
diff
changeset
|
55 |
3035b75cd594
tests: test that we can convert to manifestv2 with clone
Drew Gottlieb <drgott@google.com>
parents:
24573
diff
changeset
|
56 $ echo newerfootext > foo |
3035b75cd594
tests: test that we can convert to manifestv2 with clone
Drew Gottlieb <drgott@google.com>
parents:
24573
diff
changeset
|
57 $ hg --config experimental.manifestv2=False commit -m newer |
3035b75cd594
tests: test that we can convert to manifestv2 with clone
Drew Gottlieb <drgott@google.com>
parents:
24573
diff
changeset
|
58 |
3035b75cd594
tests: test that we can convert to manifestv2 with clone
Drew Gottlieb <drgott@google.com>
parents:
24573
diff
changeset
|
59 $ hg debugdata -m 2 |
3035b75cd594
tests: test that we can convert to manifestv2 with clone
Drew Gottlieb <drgott@google.com>
parents:
24573
diff
changeset
|
60 \x00 (esc) |
3035b75cd594
tests: test that we can convert to manifestv2 with clone
Drew Gottlieb <drgott@google.com>
parents:
24573
diff
changeset
|
61 \x00foo\x00 (esc) |
3035b75cd594
tests: test that we can convert to manifestv2 with clone
Drew Gottlieb <drgott@google.com>
parents:
24573
diff
changeset
|
62 \xa6\xb1\xfb\xef]\x91\xa1\x19`\xf3.#\x90S\xf8\x06 \xe2\x19\x00 (esc) |
3035b75cd594
tests: test that we can convert to manifestv2 with clone
Drew Gottlieb <drgott@google.com>
parents:
24573
diff
changeset
|
63 |
3035b75cd594
tests: test that we can convert to manifestv2 with clone
Drew Gottlieb <drgott@google.com>
parents:
24573
diff
changeset
|
64 Check that we can still read v1 manifests. |
3035b75cd594
tests: test that we can convert to manifestv2 with clone
Drew Gottlieb <drgott@google.com>
parents:
24573
diff
changeset
|
65 |
3035b75cd594
tests: test that we can convert to manifestv2 with clone
Drew Gottlieb <drgott@google.com>
parents:
24573
diff
changeset
|
66 $ hg files -r 0 |
3035b75cd594
tests: test that we can convert to manifestv2 with clone
Drew Gottlieb <drgott@google.com>
parents:
24573
diff
changeset
|
67 foo |
3035b75cd594
tests: test that we can convert to manifestv2 with clone
Drew Gottlieb <drgott@google.com>
parents:
24573
diff
changeset
|
68 |
3035b75cd594
tests: test that we can convert to manifestv2 with clone
Drew Gottlieb <drgott@google.com>
parents:
24573
diff
changeset
|
69 $ cd .. |
3035b75cd594
tests: test that we can convert to manifestv2 with clone
Drew Gottlieb <drgott@google.com>
parents:
24573
diff
changeset
|
70 |
3035b75cd594
tests: test that we can convert to manifestv2 with clone
Drew Gottlieb <drgott@google.com>
parents:
24573
diff
changeset
|
71 Check that entry is added to .hg/requires on repo creation |
24571
919f8ce040be
manifestv2: set requires at repo creation time
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
72 |
919f8ce040be
manifestv2: set requires at repo creation time
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
73 $ hg --config experimental.manifestv2=True init repo |
919f8ce040be
manifestv2: set requires at repo creation time
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
74 $ cd repo |
919f8ce040be
manifestv2: set requires at repo creation time
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
75 $ grep manifestv2 .hg/requires |
919f8ce040be
manifestv2: set requires at repo creation time
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
76 manifestv2 |
919f8ce040be
manifestv2: set requires at repo creation time
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
77 |
919f8ce040be
manifestv2: set requires at repo creation time
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
78 Set up simple repo |
919f8ce040be
manifestv2: set requires at repo creation time
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
79 |
919f8ce040be
manifestv2: set requires at repo creation time
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
80 $ echo a > file1 |
919f8ce040be
manifestv2: set requires at repo creation time
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
81 $ echo b > file2 |
919f8ce040be
manifestv2: set requires at repo creation time
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
82 $ echo c > file3 |
919f8ce040be
manifestv2: set requires at repo creation time
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
83 $ hg ci -Aqm 'initial' |
919f8ce040be
manifestv2: set requires at repo creation time
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
84 $ echo d > file2 |
919f8ce040be
manifestv2: set requires at repo creation time
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
85 $ hg ci -m 'modify file2' |
919f8ce040be
manifestv2: set requires at repo creation time
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
86 |
919f8ce040be
manifestv2: set requires at repo creation time
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
87 Check that 'hg verify', which uses manifest.readdelta(), works |
919f8ce040be
manifestv2: set requires at repo creation time
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
88 |
919f8ce040be
manifestv2: set requires at repo creation time
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
89 $ hg verify |
919f8ce040be
manifestv2: set requires at repo creation time
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
90 checking changesets |
919f8ce040be
manifestv2: set requires at repo creation time
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
91 checking manifests |
919f8ce040be
manifestv2: set requires at repo creation time
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
92 crosschecking files in changesets and manifests |
919f8ce040be
manifestv2: set requires at repo creation time
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
93 checking files |
919f8ce040be
manifestv2: set requires at repo creation time
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
94 3 files, 2 changesets, 4 total revisions |
919f8ce040be
manifestv2: set requires at repo creation time
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
95 |
24573
701d3554de0e
manifestv2: add support for writing new manifest format
Martin von Zweigbergk <martinvonz@google.com>
parents:
24571
diff
changeset
|
96 Check that manifest revlog is smaller than for v1 |
24571
919f8ce040be
manifestv2: set requires at repo creation time
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
97 |
919f8ce040be
manifestv2: set requires at repo creation time
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
98 $ hg debugindex -m |
26917
2329ca3ebc7a
test: use generaldelta in test-manifestv2.t
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
24935
diff
changeset
|
99 rev offset length delta linkrev nodeid p1 p2 |
2329ca3ebc7a
test: use generaldelta in test-manifestv2.t
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
24935
diff
changeset
|
100 0 0 81 -1 0 57361477c778 000000000000 000000000000 |
24573
701d3554de0e
manifestv2: add support for writing new manifest format
Martin von Zweigbergk <martinvonz@google.com>
parents:
24571
diff
changeset
|
101 1 81 33 0 1 aeaab5a2ef74 57361477c778 000000000000 |