Mercurial > hg
annotate tests/testlib/ext-sidedata.py @ 51576:de5bf3fe0233
revset: stop serializing node when using "%ln"
Turning hundred of thousand of node from node to hex and back can be slow… what
about we stop doing it?
In many case were we are using node id we should be using revision id. However
this is not a good reason to have a stupidly slow implementation of "%ln".
This caught my attention again because the phase discovery during push make an
extensive use of "%ln" or huge set. In absolute, that phase discovery probably
should use "%ld" and need to improves its algorithmic complexity, but improving
"%ln" seems simple and long overdue. This greatly speeds up `hg push` on
repository with many drafts.
Here are some relevant poulpe benchmarks:
### data-env-vars.name = mozilla-try-2023-03-22-zstd-sparse-revlog
# benchmark.name = hg.command.push
# bin-env-vars.hg.flavor = default
# bin-env-vars.hg.py-re2-module = default
# benchmark.variants.explicit-rev = all-out-heads
# benchmark.variants.issue6528 = disabled
# benchmark.variants.protocol = ssh
# benchmark.variants.reuse-external-delta-parent = default
## benchmark.variants.revs = any-1-extra-rev
before: 44.235070
after: 20.416329 (-53.85%, -23.82)
## benchmark.variants.revs = any-100-extra-rev
before: 49.234697
after: 26.519829 (-46.14%, -22.71)
### benchmark.name = hg.command.bundle
# bin-env-vars.hg.flavor = default
# bin-env-vars.hg.py-re2-module = default
# benchmark.variants.revs = all
# benchmark.variants.type = none-streamv2
## data-env-vars.name = heptapod-public-2024-03-25-zstd-sparse-revlog
before: 10.138396
after: 7.750458 (-23.55%, -2.39)
## data-env-vars.name = mercurial-public-2024-03-22-zstd-sparse-revlog
before: 1.263859
after: 0.700229 (-44.60%, -0.56)
## data-env-vars.name = mozilla-try-2023-03-22-zstd-sparse-revlog
before: 399.484481
after: 346.5089 (-13.26%, -52.98)
## data-env-vars.name = pypy-2024-03-22-zstd-sparse-revlog
before: 4.540080
after: 3.401700 (-25.07%, -1.14)
## data-env-vars.name = tryton-public-2024-03-22-zstd-sparse-revlog
before: 2.975765
after: 1.870798 (-37.13%, -1.10)
author | Pierre-Yves David <pierre-yves.david@octobus.net> |
---|---|
date | Fri, 05 Apr 2024 11:05:54 +0200 |
parents | 7c2dc75cdc0f |
children |
rev | line source |
---|---|
43040
ba4072c0a911
sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
1 # ext-sidedata.py - small extension to test the sidedata logic |
ba4072c0a911
sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
2 # |
46718
ba8e508a8e69
sidedata-exchange: rewrite sidedata on-the-fly whenever possible
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
3 # Copyright 2019 Pierre-Yves David <pierre-yves.david@octobus.net> |
43040
ba4072c0a911
sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
4 # |
ba4072c0a911
sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
5 # This software may be used and distributed according to the terms of the |
ba4072c0a911
sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
6 # GNU General Public License version 2 or any later version. |
ba4072c0a911
sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
7 |
ba4072c0a911
sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
8 |
ba4072c0a911
sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
9 import hashlib |
ba4072c0a911
sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
10 import struct |
ba4072c0a911
sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
11 |
47012
d55b71393907
node: replace nullid and friends with nodeconstants class
Joerg Sonnenberger <joerg@bec.de>
parents:
46718
diff
changeset
|
12 from mercurial.node import nullrev |
43040
ba4072c0a911
sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
13 from mercurial import ( |
ba4072c0a911
sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
14 extensions, |
45373
d7dcc75a3eae
localrepo: move requirements constant to requirements module
Pulkit Goyal <7895pulkit@gmail.com>
parents:
43134
diff
changeset
|
15 requirements, |
43040
ba4072c0a911
sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
16 revlog, |
ba4072c0a911
sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
17 ) |
ba4072c0a911
sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
18 |
46046
f105c49e89cd
upgrade: split actual upgrade code away from the main module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
45735
diff
changeset
|
19 from mercurial.upgrade_utils import engine as upgrade_engine |
f105c49e89cd
upgrade: split actual upgrade code away from the main module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
45735
diff
changeset
|
20 |
47084
27f1191b1305
sidedata: replace sidedata upgrade mechanism with the new one
Raphaël Gomès <rgomes@octobus.net>
parents:
47012
diff
changeset
|
21 from mercurial.revlogutils import constants |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
43042
diff
changeset
|
22 from mercurial.revlogutils import sidedata |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
43042
diff
changeset
|
23 |
43040
ba4072c0a911
sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
24 |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
43042
diff
changeset
|
25 def wrapaddrevision( |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
43042
diff
changeset
|
26 orig, self, text, transaction, link, p1, p2, *args, **kwargs |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
43042
diff
changeset
|
27 ): |
43040
ba4072c0a911
sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
28 if kwargs.get('sidedata') is None: |
ba4072c0a911
sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
29 kwargs['sidedata'] = {} |
ba4072c0a911
sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
30 sd = kwargs['sidedata'] |
ba4072c0a911
sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
31 ## let's store some arbitrary data just for testing |
ba4072c0a911
sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
32 # text length |
ba4072c0a911
sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
33 sd[sidedata.SD_TEST1] = struct.pack('>I', len(text)) |
ba4072c0a911
sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
34 # and sha2 hashes |
ba4072c0a911
sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
35 sha256 = hashlib.sha256(text).digest() |
ba4072c0a911
sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
36 sd[sidedata.SD_TEST2] = struct.pack('>32s', sha256) |
ba4072c0a911
sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
37 return orig(self, text, transaction, link, p1, p2, *args, **kwargs) |
ba4072c0a911
sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
38 |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
43042
diff
changeset
|
39 |
46709
3d740058b467
sidedata: move to new sidedata storage in revlogv2
Raphaël Gomès <rgomes@octobus.net>
parents:
46113
diff
changeset
|
40 def wrap_revisiondata(orig, self, nodeorrev, *args, **kwargs): |
47385
9d9eb22b9b69
revlog: no longer return sidedata from `_revisiondata`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
47224
diff
changeset
|
41 text = orig(self, nodeorrev, *args, **kwargs) |
9d9eb22b9b69
revlog: no longer return sidedata from `_revisiondata`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
47224
diff
changeset
|
42 sd = self.sidedata(nodeorrev) |
43134
75ad8af9c95e
upgrade: allow upgrade to repository using sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43076
diff
changeset
|
43 if getattr(self, 'sidedatanocheck', False): |
47385
9d9eb22b9b69
revlog: no longer return sidedata from `_revisiondata`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
47224
diff
changeset
|
44 return text |
51061
7c2dc75cdc0f
revlog: remove legacy usage of `hassidedata`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
45 if self.feature_config.has_side_data: |
47385
9d9eb22b9b69
revlog: no longer return sidedata from `_revisiondata`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
47224
diff
changeset
|
46 return text |
47012
d55b71393907
node: replace nullid and friends with nodeconstants class
Joerg Sonnenberger <joerg@bec.de>
parents:
46718
diff
changeset
|
47 if nodeorrev != nullrev and nodeorrev != self.nullid: |
46718
ba8e508a8e69
sidedata-exchange: rewrite sidedata on-the-fly whenever possible
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
48 cat1 = sd.get(sidedata.SD_TEST1) |
ba8e508a8e69
sidedata-exchange: rewrite sidedata on-the-fly whenever possible
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
49 if cat1 is not None and len(text) != struct.unpack('>I', cat1)[0]: |
43042
03e769278ef3
sidedata: check that the sidedata safely roundtrip
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43040
diff
changeset
|
50 raise RuntimeError('text size mismatch') |
46718
ba8e508a8e69
sidedata-exchange: rewrite sidedata on-the-fly whenever possible
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
51 expected = sd.get(sidedata.SD_TEST2) |
43042
03e769278ef3
sidedata: check that the sidedata safely roundtrip
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43040
diff
changeset
|
52 got = hashlib.sha256(text).digest() |
46718
ba8e508a8e69
sidedata-exchange: rewrite sidedata on-the-fly whenever possible
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
53 if expected is not None and got != expected: |
43042
03e769278ef3
sidedata: check that the sidedata safely roundtrip
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43040
diff
changeset
|
54 raise RuntimeError('sha256 mismatch') |
47385
9d9eb22b9b69
revlog: no longer return sidedata from `_revisiondata`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
47224
diff
changeset
|
55 return text |
43042
03e769278ef3
sidedata: check that the sidedata safely roundtrip
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43040
diff
changeset
|
56 |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
43042
diff
changeset
|
57 |
47084
27f1191b1305
sidedata: replace sidedata upgrade mechanism with the new one
Raphaël Gomès <rgomes@octobus.net>
parents:
47012
diff
changeset
|
58 def wrapget_sidedata_helpers(orig, srcrepo, dstrepo): |
27f1191b1305
sidedata: replace sidedata upgrade mechanism with the new one
Raphaël Gomès <rgomes@octobus.net>
parents:
47012
diff
changeset
|
59 repo, computers, removers = orig(srcrepo, dstrepo) |
27f1191b1305
sidedata: replace sidedata upgrade mechanism with the new one
Raphaël Gomès <rgomes@octobus.net>
parents:
47012
diff
changeset
|
60 assert not computers and not removers # deal with composition later |
43134
75ad8af9c95e
upgrade: allow upgrade to repository using sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43076
diff
changeset
|
61 addedreqs = dstrepo.requirements - srcrepo.requirements |
47084
27f1191b1305
sidedata: replace sidedata upgrade mechanism with the new one
Raphaël Gomès <rgomes@octobus.net>
parents:
47012
diff
changeset
|
62 |
47224
8505d23928b1
sidedata: use revlogv2 requirement in the test helper
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
47136
diff
changeset
|
63 if requirements.REVLOGV2_REQUIREMENT in addedreqs: |
43134
75ad8af9c95e
upgrade: allow upgrade to repository using sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43076
diff
changeset
|
64 |
47084
27f1191b1305
sidedata: replace sidedata upgrade mechanism with the new one
Raphaël Gomès <rgomes@octobus.net>
parents:
47012
diff
changeset
|
65 def computer(repo, revlog, rev, old_sidedata): |
27f1191b1305
sidedata: replace sidedata upgrade mechanism with the new one
Raphaël Gomès <rgomes@octobus.net>
parents:
47012
diff
changeset
|
66 assert not old_sidedata # not supported yet |
43134
75ad8af9c95e
upgrade: allow upgrade to repository using sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43076
diff
changeset
|
67 update = {} |
75ad8af9c95e
upgrade: allow upgrade to repository using sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43076
diff
changeset
|
68 revlog.sidedatanocheck = True |
75ad8af9c95e
upgrade: allow upgrade to repository using sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43076
diff
changeset
|
69 try: |
75ad8af9c95e
upgrade: allow upgrade to repository using sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43076
diff
changeset
|
70 text = revlog.revision(rev) |
75ad8af9c95e
upgrade: allow upgrade to repository using sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43076
diff
changeset
|
71 finally: |
75ad8af9c95e
upgrade: allow upgrade to repository using sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43076
diff
changeset
|
72 del revlog.sidedatanocheck |
75ad8af9c95e
upgrade: allow upgrade to repository using sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43076
diff
changeset
|
73 ## let's store some arbitrary data just for testing |
75ad8af9c95e
upgrade: allow upgrade to repository using sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43076
diff
changeset
|
74 # text length |
75ad8af9c95e
upgrade: allow upgrade to repository using sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43076
diff
changeset
|
75 update[sidedata.SD_TEST1] = struct.pack('>I', len(text)) |
75ad8af9c95e
upgrade: allow upgrade to repository using sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43076
diff
changeset
|
76 # and sha2 hashes |
75ad8af9c95e
upgrade: allow upgrade to repository using sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43076
diff
changeset
|
77 sha256 = hashlib.sha256(text).digest() |
75ad8af9c95e
upgrade: allow upgrade to repository using sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43076
diff
changeset
|
78 update[sidedata.SD_TEST2] = struct.pack('>32s', sha256) |
47084
27f1191b1305
sidedata: replace sidedata upgrade mechanism with the new one
Raphaël Gomès <rgomes@octobus.net>
parents:
47012
diff
changeset
|
79 return update, (0, 0) |
43134
75ad8af9c95e
upgrade: allow upgrade to repository using sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43076
diff
changeset
|
80 |
47084
27f1191b1305
sidedata: replace sidedata upgrade mechanism with the new one
Raphaël Gomès <rgomes@octobus.net>
parents:
47012
diff
changeset
|
81 srcrepo.register_sidedata_computer( |
27f1191b1305
sidedata: replace sidedata upgrade mechanism with the new one
Raphaël Gomès <rgomes@octobus.net>
parents:
47012
diff
changeset
|
82 constants.KIND_CHANGELOG, |
27f1191b1305
sidedata: replace sidedata upgrade mechanism with the new one
Raphaël Gomès <rgomes@octobus.net>
parents:
47012
diff
changeset
|
83 b"whatever", |
27f1191b1305
sidedata: replace sidedata upgrade mechanism with the new one
Raphaël Gomès <rgomes@octobus.net>
parents:
47012
diff
changeset
|
84 (sidedata.SD_TEST1, sidedata.SD_TEST2), |
27f1191b1305
sidedata: replace sidedata upgrade mechanism with the new one
Raphaël Gomès <rgomes@octobus.net>
parents:
47012
diff
changeset
|
85 computer, |
27f1191b1305
sidedata: replace sidedata upgrade mechanism with the new one
Raphaël Gomès <rgomes@octobus.net>
parents:
47012
diff
changeset
|
86 0, |
27f1191b1305
sidedata: replace sidedata upgrade mechanism with the new one
Raphaël Gomès <rgomes@octobus.net>
parents:
47012
diff
changeset
|
87 ) |
27f1191b1305
sidedata: replace sidedata upgrade mechanism with the new one
Raphaël Gomès <rgomes@octobus.net>
parents:
47012
diff
changeset
|
88 dstrepo.register_wanted_sidedata(b"whatever") |
27f1191b1305
sidedata: replace sidedata upgrade mechanism with the new one
Raphaël Gomès <rgomes@octobus.net>
parents:
47012
diff
changeset
|
89 |
47085
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
47084
diff
changeset
|
90 return sidedata.get_sidedata_helpers(srcrepo, dstrepo._wanted_sidedata) |
43134
75ad8af9c95e
upgrade: allow upgrade to repository using sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43076
diff
changeset
|
91 |
75ad8af9c95e
upgrade: allow upgrade to repository using sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43076
diff
changeset
|
92 |
43040
ba4072c0a911
sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
93 def extsetup(ui): |
ba4072c0a911
sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
94 extensions.wrapfunction(revlog.revlog, 'addrevision', wrapaddrevision) |
46709
3d740058b467
sidedata: move to new sidedata storage in revlogv2
Raphaël Gomès <rgomes@octobus.net>
parents:
46113
diff
changeset
|
95 extensions.wrapfunction(revlog.revlog, '_revisiondata', wrap_revisiondata) |
43134
75ad8af9c95e
upgrade: allow upgrade to repository using sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43076
diff
changeset
|
96 extensions.wrapfunction( |
47084
27f1191b1305
sidedata: replace sidedata upgrade mechanism with the new one
Raphaël Gomès <rgomes@octobus.net>
parents:
47012
diff
changeset
|
97 upgrade_engine, 'get_sidedata_helpers', wrapget_sidedata_helpers |
43134
75ad8af9c95e
upgrade: allow upgrade to repository using sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43076
diff
changeset
|
98 ) |
46718
ba8e508a8e69
sidedata-exchange: rewrite sidedata on-the-fly whenever possible
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
99 |
ba8e508a8e69
sidedata-exchange: rewrite sidedata on-the-fly whenever possible
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
100 |
ba8e508a8e69
sidedata-exchange: rewrite sidedata on-the-fly whenever possible
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
101 def reposetup(ui, repo): |
ba8e508a8e69
sidedata-exchange: rewrite sidedata on-the-fly whenever possible
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
102 # We don't register sidedata computers because we don't care within these |
ba8e508a8e69
sidedata-exchange: rewrite sidedata on-the-fly whenever possible
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
103 # tests |
ba8e508a8e69
sidedata-exchange: rewrite sidedata on-the-fly whenever possible
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
104 repo.register_wanted_sidedata(sidedata.SD_TEST1) |
ba8e508a8e69
sidedata-exchange: rewrite sidedata on-the-fly whenever possible
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
105 repo.register_wanted_sidedata(sidedata.SD_TEST2) |