Mercurial > hg
annotate mercurial/revlogutils/deltas.py @ 45537:2f8227a12592
rhg: add `--revision` argument to `rhg files`
Add the option to list the tracked files of a revision given its number
or full node id.
Benched on a clone of moz-central
where tip is 1671467:81deaa1a68ebb28db0490954034ab38ab269409d
files -r 81deaa1a68ebb28db0490954034ab38ab269409d > out.txt
hg 0m1.633s
rhg 0m0.157s
files -r 81deaa1a68ebb28db0490954034ab38ab269409d > /dev/null
hg 0m0.415s
rhg 0m0.143s
Differential Revision: https://phab.mercurial-scm.org/D9015
author | Antoine Cezar <antoine.cezar@octobus.net> |
---|---|
date | Wed, 09 Sep 2020 14:53:15 +0200 |
parents | c59eb1560c44 |
children | d4ba4d51f85f |
rev | line source |
---|---|
39330
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
1 # revlogdeltas.py - Logic around delta computation for revlog |
8226
8b2cd04a6e97
put license and copyright info into comment blocks
Martin Geisler <mg@lazybytes.net>
parents:
8225
diff
changeset
|
2 # |
8b2cd04a6e97
put license and copyright info into comment blocks
Martin Geisler <mg@lazybytes.net>
parents:
8225
diff
changeset
|
3 # Copyright 2005-2007 Matt Mackall <mpm@selenic.com> |
39330
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
4 # Copyright 2018 Octobus <contact@octobus.net> |
8226
8b2cd04a6e97
put license and copyright info into comment blocks
Martin Geisler <mg@lazybytes.net>
parents:
8225
diff
changeset
|
5 # |
8b2cd04a6e97
put license and copyright info into comment blocks
Martin Geisler <mg@lazybytes.net>
parents:
8225
diff
changeset
|
6 # This software may be used and distributed according to the terms of the |
10263 | 7 # GNU General Public License version 2 or any later version. |
39330
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
8 """Helper class to compute deltas stored inside revlogs""" |
8227
0a9542703300
turn some comments back into module docstrings
Martin Geisler <mg@lazybytes.net>
parents:
8226
diff
changeset
|
9 |
27361
29f50344fa83
revlog: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27251
diff
changeset
|
10 from __future__ import absolute_import |
29f50344fa83
revlog: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27251
diff
changeset
|
11 |
39493
3ca144f1c8dd
snapshot: search for unrelated but reusable full-snapshot
Boris Feld <boris.feld@octobus.net>
parents:
39492
diff
changeset
|
12 import collections |
27361
29f50344fa83
revlog: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27251
diff
changeset
|
13 import struct |
29f50344fa83
revlog: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27251
diff
changeset
|
14 |
29f50344fa83
revlog: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27251
diff
changeset
|
15 # import stuff from node for others to import from revlog |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
16 from ..node import nullrev |
39330
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
17 from ..i18n import _ |
43089
c59eb1560c44
py3: manually import getattr where it is needed
Gregory Szorc <gregory.szorc@gmail.com>
parents:
43077
diff
changeset
|
18 from ..pycompat import getattr |
39330
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
19 |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
20 from .constants import ( |
39329
729082bb9938
revlog: split constants into a new `revlogutils.constants` module
Boris Feld <boris.feld@octobus.net>
parents:
39232
diff
changeset
|
21 REVIDX_ISCENSORED, |
729082bb9938
revlog: split constants into a new `revlogutils.constants` module
Boris Feld <boris.feld@octobus.net>
parents:
39232
diff
changeset
|
22 REVIDX_RAWTEXT_CHANGING_FLAGS, |
729082bb9938
revlog: split constants into a new `revlogutils.constants` module
Boris Feld <boris.feld@octobus.net>
parents:
39232
diff
changeset
|
23 ) |
39330
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
24 |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
25 from ..thirdparty import attr |
39330
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
26 |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
27 from .. import ( |
27361
29f50344fa83
revlog: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27251
diff
changeset
|
28 error, |
29f50344fa83
revlog: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27251
diff
changeset
|
29 mdiff, |
41108
38e88450138c
delta: have a native implementation of _findsnapshot
Boris Feld <boris.feld@octobus.net>
parents:
41079
diff
changeset
|
30 util, |
27361
29f50344fa83
revlog: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27251
diff
changeset
|
31 ) |
10913
f2ecc5733c89
revlog: factor out _maxinline global.
Greg Ward <greg-hg@gerg.ca>
parents:
10404
diff
changeset
|
32 |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
33 from . import flagutil |
42992
dff95420480f
flagprocessors: make `processflagsraw` a module level function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
42878
diff
changeset
|
34 |
39330
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
35 # maximum <delta-chain-data>/<revision-text-length> ratio |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
36 LIMIT_DELTA2TEXT = 2 |
1091
d62130f99a73
Move hash function back to revlog from node
mpm@selenic.com
parents:
1089
diff
changeset
|
37 |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
38 |
38637
e33f784f2a44
revlog: introduce a tiny mock of a revlog class
Boris Feld <boris.feld@octobus.net>
parents:
38636
diff
changeset
|
39 class _testrevlog(object): |
e33f784f2a44
revlog: introduce a tiny mock of a revlog class
Boris Feld <boris.feld@octobus.net>
parents:
38636
diff
changeset
|
40 """minimalist fake revlog to use in doctests""" |
e33f784f2a44
revlog: introduce a tiny mock of a revlog class
Boris Feld <boris.feld@octobus.net>
parents:
38636
diff
changeset
|
41 |
40641
85b14f0dc334
doctest: add a `issnapshot` method to _testrevlog
Boris Feld <boris.feld@octobus.net>
parents:
40608
diff
changeset
|
42 def __init__(self, data, density=0.5, mingap=0, snapshot=()): |
38637
e33f784f2a44
revlog: introduce a tiny mock of a revlog class
Boris Feld <boris.feld@octobus.net>
parents:
38636
diff
changeset
|
43 """data is an list of revision payload boundaries""" |
e33f784f2a44
revlog: introduce a tiny mock of a revlog class
Boris Feld <boris.feld@octobus.net>
parents:
38636
diff
changeset
|
44 self._data = data |
e33f784f2a44
revlog: introduce a tiny mock of a revlog class
Boris Feld <boris.feld@octobus.net>
parents:
38636
diff
changeset
|
45 self._srdensitythreshold = density |
e33f784f2a44
revlog: introduce a tiny mock of a revlog class
Boris Feld <boris.feld@octobus.net>
parents:
38636
diff
changeset
|
46 self._srmingapsize = mingap |
40641
85b14f0dc334
doctest: add a `issnapshot` method to _testrevlog
Boris Feld <boris.feld@octobus.net>
parents:
40608
diff
changeset
|
47 self._snapshot = set(snapshot) |
40709
39d29542fe40
sparse-revlog: put the native implementation of slicechunktodensity to use
Boris Feld <boris.feld@octobus.net>
parents:
40657
diff
changeset
|
48 self.index = None |
38637
e33f784f2a44
revlog: introduce a tiny mock of a revlog class
Boris Feld <boris.feld@octobus.net>
parents:
38636
diff
changeset
|
49 |
e33f784f2a44
revlog: introduce a tiny mock of a revlog class
Boris Feld <boris.feld@octobus.net>
parents:
38636
diff
changeset
|
50 def start(self, rev): |
41079
88d813cd9acd
revlog: fix pure python slicing test when chain contains nullrev
Boris Feld <boris.feld@octobus.net>
parents:
41033
diff
changeset
|
51 if rev == nullrev: |
88d813cd9acd
revlog: fix pure python slicing test when chain contains nullrev
Boris Feld <boris.feld@octobus.net>
parents:
41033
diff
changeset
|
52 return 0 |
38637
e33f784f2a44
revlog: introduce a tiny mock of a revlog class
Boris Feld <boris.feld@octobus.net>
parents:
38636
diff
changeset
|
53 if rev == 0: |
e33f784f2a44
revlog: introduce a tiny mock of a revlog class
Boris Feld <boris.feld@octobus.net>
parents:
38636
diff
changeset
|
54 return 0 |
e33f784f2a44
revlog: introduce a tiny mock of a revlog class
Boris Feld <boris.feld@octobus.net>
parents:
38636
diff
changeset
|
55 return self._data[rev - 1] |
e33f784f2a44
revlog: introduce a tiny mock of a revlog class
Boris Feld <boris.feld@octobus.net>
parents:
38636
diff
changeset
|
56 |
e33f784f2a44
revlog: introduce a tiny mock of a revlog class
Boris Feld <boris.feld@octobus.net>
parents:
38636
diff
changeset
|
57 def end(self, rev): |
41079
88d813cd9acd
revlog: fix pure python slicing test when chain contains nullrev
Boris Feld <boris.feld@octobus.net>
parents:
41033
diff
changeset
|
58 if rev == nullrev: |
88d813cd9acd
revlog: fix pure python slicing test when chain contains nullrev
Boris Feld <boris.feld@octobus.net>
parents:
41033
diff
changeset
|
59 return 0 |
38637
e33f784f2a44
revlog: introduce a tiny mock of a revlog class
Boris Feld <boris.feld@octobus.net>
parents:
38636
diff
changeset
|
60 return self._data[rev] |
e33f784f2a44
revlog: introduce a tiny mock of a revlog class
Boris Feld <boris.feld@octobus.net>
parents:
38636
diff
changeset
|
61 |
e33f784f2a44
revlog: introduce a tiny mock of a revlog class
Boris Feld <boris.feld@octobus.net>
parents:
38636
diff
changeset
|
62 def length(self, rev): |
e33f784f2a44
revlog: introduce a tiny mock of a revlog class
Boris Feld <boris.feld@octobus.net>
parents:
38636
diff
changeset
|
63 return self.end(rev) - self.start(rev) |
e33f784f2a44
revlog: introduce a tiny mock of a revlog class
Boris Feld <boris.feld@octobus.net>
parents:
38636
diff
changeset
|
64 |
38718
f8762ea73e0d
sparse-revlog: implement algorithm to write sparse delta chains (issue5480)
Paul Morelle <paul.morelle@octobus.net>
parents:
38717
diff
changeset
|
65 def __len__(self): |
f8762ea73e0d
sparse-revlog: implement algorithm to write sparse delta chains (issue5480)
Paul Morelle <paul.morelle@octobus.net>
parents:
38717
diff
changeset
|
66 return len(self._data) |
f8762ea73e0d
sparse-revlog: implement algorithm to write sparse delta chains (issue5480)
Paul Morelle <paul.morelle@octobus.net>
parents:
38717
diff
changeset
|
67 |
40641
85b14f0dc334
doctest: add a `issnapshot` method to _testrevlog
Boris Feld <boris.feld@octobus.net>
parents:
40608
diff
changeset
|
68 def issnapshot(self, rev): |
41079
88d813cd9acd
revlog: fix pure python slicing test when chain contains nullrev
Boris Feld <boris.feld@octobus.net>
parents:
41033
diff
changeset
|
69 if rev == nullrev: |
88d813cd9acd
revlog: fix pure python slicing test when chain contains nullrev
Boris Feld <boris.feld@octobus.net>
parents:
41033
diff
changeset
|
70 return True |
40641
85b14f0dc334
doctest: add a `issnapshot` method to _testrevlog
Boris Feld <boris.feld@octobus.net>
parents:
40608
diff
changeset
|
71 return rev in self._snapshot |
85b14f0dc334
doctest: add a `issnapshot` method to _testrevlog
Boris Feld <boris.feld@octobus.net>
parents:
40608
diff
changeset
|
72 |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
73 |
40604
3ac23dad6364
sparse-revlog: drop unused deltainfo parameter from _slicechunktodensity
Boris Feld <boris.feld@octobus.net>
parents:
40603
diff
changeset
|
74 def slicechunk(revlog, revs, targetsize=None): |
34824
e2ad93bcc084
revlog: introduce an experimental flag to slice chunks reads when too sparse
Paul Morelle <paul.morelle@octobus.net>
parents:
34823
diff
changeset
|
75 """slice revs to reduce the amount of unrelated data to be read from disk. |
e2ad93bcc084
revlog: introduce an experimental flag to slice chunks reads when too sparse
Paul Morelle <paul.morelle@octobus.net>
parents:
34823
diff
changeset
|
76 |
e2ad93bcc084
revlog: introduce an experimental flag to slice chunks reads when too sparse
Paul Morelle <paul.morelle@octobus.net>
parents:
34823
diff
changeset
|
77 ``revs`` is sliced into groups that should be read in one time. |
e2ad93bcc084
revlog: introduce an experimental flag to slice chunks reads when too sparse
Paul Morelle <paul.morelle@octobus.net>
parents:
34823
diff
changeset
|
78 Assume that revs are sorted. |
38640
f62b8fb0a484
revlog: document and test _slicechunk
Boris Feld <boris.feld@octobus.net>
parents:
38639
diff
changeset
|
79 |
f62b8fb0a484
revlog: document and test _slicechunk
Boris Feld <boris.feld@octobus.net>
parents:
38639
diff
changeset
|
80 The initial chunk is sliced until the overall density (payload/chunks-span |
f62b8fb0a484
revlog: document and test _slicechunk
Boris Feld <boris.feld@octobus.net>
parents:
38639
diff
changeset
|
81 ratio) is above `revlog._srdensitythreshold`. No gap smaller than |
f62b8fb0a484
revlog: document and test _slicechunk
Boris Feld <boris.feld@octobus.net>
parents:
38639
diff
changeset
|
82 `revlog._srmingapsize` is skipped. |
f62b8fb0a484
revlog: document and test _slicechunk
Boris Feld <boris.feld@octobus.net>
parents:
38639
diff
changeset
|
83 |
38643
967fee55e8d9
revlog: postprocess chunk to slice them down to a certain size
Boris Feld <boris.feld@octobus.net>
parents:
38642
diff
changeset
|
84 If `targetsize` is set, no chunk larger than `targetsize` will be yield. |
967fee55e8d9
revlog: postprocess chunk to slice them down to a certain size
Boris Feld <boris.feld@octobus.net>
parents:
38642
diff
changeset
|
85 For consistency with other slicing choice, this limit won't go lower than |
967fee55e8d9
revlog: postprocess chunk to slice them down to a certain size
Boris Feld <boris.feld@octobus.net>
parents:
38642
diff
changeset
|
86 `revlog._srmingapsize`. |
967fee55e8d9
revlog: postprocess chunk to slice them down to a certain size
Boris Feld <boris.feld@octobus.net>
parents:
38642
diff
changeset
|
87 |
967fee55e8d9
revlog: postprocess chunk to slice them down to a certain size
Boris Feld <boris.feld@octobus.net>
parents:
38642
diff
changeset
|
88 If individual revisions chunk are larger than this limit, they will still |
967fee55e8d9
revlog: postprocess chunk to slice them down to a certain size
Boris Feld <boris.feld@octobus.net>
parents:
38642
diff
changeset
|
89 be raised individually. |
967fee55e8d9
revlog: postprocess chunk to slice them down to a certain size
Boris Feld <boris.feld@octobus.net>
parents:
38642
diff
changeset
|
90 |
40642
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
91 >>> data = [ |
38640
f62b8fb0a484
revlog: document and test _slicechunk
Boris Feld <boris.feld@octobus.net>
parents:
38639
diff
changeset
|
92 ... 5, #00 (5) |
f62b8fb0a484
revlog: document and test _slicechunk
Boris Feld <boris.feld@octobus.net>
parents:
38639
diff
changeset
|
93 ... 10, #01 (5) |
f62b8fb0a484
revlog: document and test _slicechunk
Boris Feld <boris.feld@octobus.net>
parents:
38639
diff
changeset
|
94 ... 12, #02 (2) |
f62b8fb0a484
revlog: document and test _slicechunk
Boris Feld <boris.feld@octobus.net>
parents:
38639
diff
changeset
|
95 ... 12, #03 (empty) |
f62b8fb0a484
revlog: document and test _slicechunk
Boris Feld <boris.feld@octobus.net>
parents:
38639
diff
changeset
|
96 ... 27, #04 (15) |
f62b8fb0a484
revlog: document and test _slicechunk
Boris Feld <boris.feld@octobus.net>
parents:
38639
diff
changeset
|
97 ... 31, #05 (4) |
f62b8fb0a484
revlog: document and test _slicechunk
Boris Feld <boris.feld@octobus.net>
parents:
38639
diff
changeset
|
98 ... 31, #06 (empty) |
f62b8fb0a484
revlog: document and test _slicechunk
Boris Feld <boris.feld@octobus.net>
parents:
38639
diff
changeset
|
99 ... 42, #07 (11) |
f62b8fb0a484
revlog: document and test _slicechunk
Boris Feld <boris.feld@octobus.net>
parents:
38639
diff
changeset
|
100 ... 47, #08 (5) |
f62b8fb0a484
revlog: document and test _slicechunk
Boris Feld <boris.feld@octobus.net>
parents:
38639
diff
changeset
|
101 ... 47, #09 (empty) |
f62b8fb0a484
revlog: document and test _slicechunk
Boris Feld <boris.feld@octobus.net>
parents:
38639
diff
changeset
|
102 ... 48, #10 (1) |
f62b8fb0a484
revlog: document and test _slicechunk
Boris Feld <boris.feld@octobus.net>
parents:
38639
diff
changeset
|
103 ... 51, #11 (3) |
f62b8fb0a484
revlog: document and test _slicechunk
Boris Feld <boris.feld@octobus.net>
parents:
38639
diff
changeset
|
104 ... 74, #12 (23) |
f62b8fb0a484
revlog: document and test _slicechunk
Boris Feld <boris.feld@octobus.net>
parents:
38639
diff
changeset
|
105 ... 85, #13 (11) |
f62b8fb0a484
revlog: document and test _slicechunk
Boris Feld <boris.feld@octobus.net>
parents:
38639
diff
changeset
|
106 ... 86, #14 (1) |
f62b8fb0a484
revlog: document and test _slicechunk
Boris Feld <boris.feld@octobus.net>
parents:
38639
diff
changeset
|
107 ... 91, #15 (5) |
40642
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
108 ... ] |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
109 >>> revlog = _testrevlog(data, snapshot=range(16)) |
38640
f62b8fb0a484
revlog: document and test _slicechunk
Boris Feld <boris.feld@octobus.net>
parents:
38639
diff
changeset
|
110 |
39330
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
111 >>> list(slicechunk(revlog, list(range(16)))) |
38640
f62b8fb0a484
revlog: document and test _slicechunk
Boris Feld <boris.feld@octobus.net>
parents:
38639
diff
changeset
|
112 [[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]] |
39330
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
113 >>> list(slicechunk(revlog, [0, 15])) |
38640
f62b8fb0a484
revlog: document and test _slicechunk
Boris Feld <boris.feld@octobus.net>
parents:
38639
diff
changeset
|
114 [[0], [15]] |
39330
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
115 >>> list(slicechunk(revlog, [0, 11, 15])) |
38640
f62b8fb0a484
revlog: document and test _slicechunk
Boris Feld <boris.feld@octobus.net>
parents:
38639
diff
changeset
|
116 [[0], [11], [15]] |
39330
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
117 >>> list(slicechunk(revlog, [0, 11, 13, 15])) |
38640
f62b8fb0a484
revlog: document and test _slicechunk
Boris Feld <boris.feld@octobus.net>
parents:
38639
diff
changeset
|
118 [[0], [11, 13, 15]] |
39330
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
119 >>> list(slicechunk(revlog, [1, 2, 3, 5, 8, 10, 11, 14])) |
38640
f62b8fb0a484
revlog: document and test _slicechunk
Boris Feld <boris.feld@octobus.net>
parents:
38639
diff
changeset
|
120 [[1, 2], [5, 8, 10, 11], [14]] |
38643
967fee55e8d9
revlog: postprocess chunk to slice them down to a certain size
Boris Feld <boris.feld@octobus.net>
parents:
38642
diff
changeset
|
121 |
967fee55e8d9
revlog: postprocess chunk to slice them down to a certain size
Boris Feld <boris.feld@octobus.net>
parents:
38642
diff
changeset
|
122 Slicing with a maximum chunk size |
39330
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
123 >>> list(slicechunk(revlog, [0, 11, 13, 15], targetsize=15)) |
38643
967fee55e8d9
revlog: postprocess chunk to slice them down to a certain size
Boris Feld <boris.feld@octobus.net>
parents:
38642
diff
changeset
|
124 [[0], [11], [13], [15]] |
39330
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
125 >>> list(slicechunk(revlog, [0, 11, 13, 15], targetsize=20)) |
38643
967fee55e8d9
revlog: postprocess chunk to slice them down to a certain size
Boris Feld <boris.feld@octobus.net>
parents:
38642
diff
changeset
|
126 [[0], [11], [13, 15]] |
41079
88d813cd9acd
revlog: fix pure python slicing test when chain contains nullrev
Boris Feld <boris.feld@octobus.net>
parents:
41033
diff
changeset
|
127 |
88d813cd9acd
revlog: fix pure python slicing test when chain contains nullrev
Boris Feld <boris.feld@octobus.net>
parents:
41033
diff
changeset
|
128 Slicing involving nullrev |
88d813cd9acd
revlog: fix pure python slicing test when chain contains nullrev
Boris Feld <boris.feld@octobus.net>
parents:
41033
diff
changeset
|
129 >>> list(slicechunk(revlog, [-1, 0, 11, 13, 15], targetsize=20)) |
88d813cd9acd
revlog: fix pure python slicing test when chain contains nullrev
Boris Feld <boris.feld@octobus.net>
parents:
41033
diff
changeset
|
130 [[-1, 0], [11], [13, 15]] |
88d813cd9acd
revlog: fix pure python slicing test when chain contains nullrev
Boris Feld <boris.feld@octobus.net>
parents:
41033
diff
changeset
|
131 >>> list(slicechunk(revlog, [-1, 13, 15], targetsize=5)) |
88d813cd9acd
revlog: fix pure python slicing test when chain contains nullrev
Boris Feld <boris.feld@octobus.net>
parents:
41033
diff
changeset
|
132 [[-1], [13], [15]] |
34824
e2ad93bcc084
revlog: introduce an experimental flag to slice chunks reads when too sparse
Paul Morelle <paul.morelle@octobus.net>
parents:
34823
diff
changeset
|
133 """ |
38643
967fee55e8d9
revlog: postprocess chunk to slice them down to a certain size
Boris Feld <boris.feld@octobus.net>
parents:
38642
diff
changeset
|
134 if targetsize is not None: |
967fee55e8d9
revlog: postprocess chunk to slice them down to a certain size
Boris Feld <boris.feld@octobus.net>
parents:
38642
diff
changeset
|
135 targetsize = max(targetsize, revlog._srmingapsize) |
38718
f8762ea73e0d
sparse-revlog: implement algorithm to write sparse delta chains (issue5480)
Paul Morelle <paul.morelle@octobus.net>
parents:
38717
diff
changeset
|
136 # targetsize should not be specified when evaluating delta candidates: |
f8762ea73e0d
sparse-revlog: implement algorithm to write sparse delta chains (issue5480)
Paul Morelle <paul.morelle@octobus.net>
parents:
38717
diff
changeset
|
137 # * targetsize is used to ensure we stay within specification when reading, |
40709
39d29542fe40
sparse-revlog: put the native implementation of slicechunktodensity to use
Boris Feld <boris.feld@octobus.net>
parents:
40657
diff
changeset
|
138 densityslicing = getattr(revlog.index, 'slicechunktodensity', None) |
39d29542fe40
sparse-revlog: put the native implementation of slicechunktodensity to use
Boris Feld <boris.feld@octobus.net>
parents:
40657
diff
changeset
|
139 if densityslicing is None: |
39d29542fe40
sparse-revlog: put the native implementation of slicechunktodensity to use
Boris Feld <boris.feld@octobus.net>
parents:
40657
diff
changeset
|
140 densityslicing = lambda x, y, z: _slicechunktodensity(revlog, x, y, z) |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
141 for chunk in densityslicing( |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
142 revs, revlog._srdensitythreshold, revlog._srmingapsize |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
143 ): |
38643
967fee55e8d9
revlog: postprocess chunk to slice them down to a certain size
Boris Feld <boris.feld@octobus.net>
parents:
38642
diff
changeset
|
144 for subchunk in _slicechunktosize(revlog, chunk, targetsize): |
967fee55e8d9
revlog: postprocess chunk to slice them down to a certain size
Boris Feld <boris.feld@octobus.net>
parents:
38642
diff
changeset
|
145 yield subchunk |
38641
feba6be0941b
revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents:
38640
diff
changeset
|
146 |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
147 |
38718
f8762ea73e0d
sparse-revlog: implement algorithm to write sparse delta chains (issue5480)
Paul Morelle <paul.morelle@octobus.net>
parents:
38717
diff
changeset
|
148 def _slicechunktosize(revlog, revs, targetsize=None): |
38642
e59e27e52297
revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents:
38641
diff
changeset
|
149 """slice revs to match the target size |
e59e27e52297
revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents:
38641
diff
changeset
|
150 |
e59e27e52297
revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents:
38641
diff
changeset
|
151 This is intended to be used on chunk that density slicing selected by that |
e59e27e52297
revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents:
38641
diff
changeset
|
152 are still too large compared to the read garantee of revlog. This might |
e59e27e52297
revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents:
38641
diff
changeset
|
153 happens when "minimal gap size" interrupted the slicing or when chain are |
e59e27e52297
revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents:
38641
diff
changeset
|
154 built in a way that create large blocks next to each other. |
e59e27e52297
revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents:
38641
diff
changeset
|
155 |
40642
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
156 >>> data = [ |
38642
e59e27e52297
revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents:
38641
diff
changeset
|
157 ... 3, #0 (3) |
e59e27e52297
revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents:
38641
diff
changeset
|
158 ... 5, #1 (2) |
e59e27e52297
revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents:
38641
diff
changeset
|
159 ... 6, #2 (1) |
e59e27e52297
revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents:
38641
diff
changeset
|
160 ... 8, #3 (2) |
e59e27e52297
revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents:
38641
diff
changeset
|
161 ... 8, #4 (empty) |
e59e27e52297
revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents:
38641
diff
changeset
|
162 ... 11, #5 (3) |
e59e27e52297
revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents:
38641
diff
changeset
|
163 ... 12, #6 (1) |
e59e27e52297
revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents:
38641
diff
changeset
|
164 ... 13, #7 (1) |
e59e27e52297
revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents:
38641
diff
changeset
|
165 ... 14, #8 (1) |
40642
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
166 ... ] |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
167 |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
168 == All snapshots cases == |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
169 >>> revlog = _testrevlog(data, snapshot=range(9)) |
38642
e59e27e52297
revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents:
38641
diff
changeset
|
170 |
e59e27e52297
revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents:
38641
diff
changeset
|
171 Cases where chunk is already small enough |
e59e27e52297
revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents:
38641
diff
changeset
|
172 >>> list(_slicechunktosize(revlog, [0], 3)) |
e59e27e52297
revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents:
38641
diff
changeset
|
173 [[0]] |
e59e27e52297
revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents:
38641
diff
changeset
|
174 >>> list(_slicechunktosize(revlog, [6, 7], 3)) |
e59e27e52297
revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents:
38641
diff
changeset
|
175 [[6, 7]] |
e59e27e52297
revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents:
38641
diff
changeset
|
176 >>> list(_slicechunktosize(revlog, [0], None)) |
e59e27e52297
revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents:
38641
diff
changeset
|
177 [[0]] |
e59e27e52297
revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents:
38641
diff
changeset
|
178 >>> list(_slicechunktosize(revlog, [6, 7], None)) |
e59e27e52297
revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents:
38641
diff
changeset
|
179 [[6, 7]] |
e59e27e52297
revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents:
38641
diff
changeset
|
180 |
e59e27e52297
revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents:
38641
diff
changeset
|
181 cases where we need actual slicing |
e59e27e52297
revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents:
38641
diff
changeset
|
182 >>> list(_slicechunktosize(revlog, [0, 1], 3)) |
e59e27e52297
revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents:
38641
diff
changeset
|
183 [[0], [1]] |
e59e27e52297
revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents:
38641
diff
changeset
|
184 >>> list(_slicechunktosize(revlog, [1, 3], 3)) |
e59e27e52297
revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents:
38641
diff
changeset
|
185 [[1], [3]] |
e59e27e52297
revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents:
38641
diff
changeset
|
186 >>> list(_slicechunktosize(revlog, [1, 2, 3], 3)) |
e59e27e52297
revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents:
38641
diff
changeset
|
187 [[1, 2], [3]] |
e59e27e52297
revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents:
38641
diff
changeset
|
188 >>> list(_slicechunktosize(revlog, [3, 5], 3)) |
e59e27e52297
revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents:
38641
diff
changeset
|
189 [[3], [5]] |
e59e27e52297
revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents:
38641
diff
changeset
|
190 >>> list(_slicechunktosize(revlog, [3, 4, 5], 3)) |
e59e27e52297
revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents:
38641
diff
changeset
|
191 [[3], [5]] |
e59e27e52297
revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents:
38641
diff
changeset
|
192 >>> list(_slicechunktosize(revlog, [5, 6, 7, 8], 3)) |
e59e27e52297
revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents:
38641
diff
changeset
|
193 [[5], [6, 7, 8]] |
e59e27e52297
revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents:
38641
diff
changeset
|
194 >>> list(_slicechunktosize(revlog, [0, 1, 2, 3, 4, 5, 6, 7, 8], 3)) |
e59e27e52297
revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents:
38641
diff
changeset
|
195 [[0], [1, 2], [3], [5], [6, 7, 8]] |
e59e27e52297
revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents:
38641
diff
changeset
|
196 |
e59e27e52297
revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents:
38641
diff
changeset
|
197 Case with too large individual chunk (must return valid chunk) |
e59e27e52297
revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents:
38641
diff
changeset
|
198 >>> list(_slicechunktosize(revlog, [0, 1], 2)) |
e59e27e52297
revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents:
38641
diff
changeset
|
199 [[0], [1]] |
e59e27e52297
revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents:
38641
diff
changeset
|
200 >>> list(_slicechunktosize(revlog, [1, 3], 1)) |
e59e27e52297
revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents:
38641
diff
changeset
|
201 [[1], [3]] |
e59e27e52297
revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents:
38641
diff
changeset
|
202 >>> list(_slicechunktosize(revlog, [3, 4, 5], 2)) |
e59e27e52297
revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents:
38641
diff
changeset
|
203 [[3], [5]] |
40642
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
204 |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
205 == No Snapshot cases == |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
206 >>> revlog = _testrevlog(data) |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
207 |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
208 Cases where chunk is already small enough |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
209 >>> list(_slicechunktosize(revlog, [0], 3)) |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
210 [[0]] |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
211 >>> list(_slicechunktosize(revlog, [6, 7], 3)) |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
212 [[6, 7]] |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
213 >>> list(_slicechunktosize(revlog, [0], None)) |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
214 [[0]] |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
215 >>> list(_slicechunktosize(revlog, [6, 7], None)) |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
216 [[6, 7]] |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
217 |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
218 cases where we need actual slicing |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
219 >>> list(_slicechunktosize(revlog, [0, 1], 3)) |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
220 [[0], [1]] |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
221 >>> list(_slicechunktosize(revlog, [1, 3], 3)) |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
222 [[1], [3]] |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
223 >>> list(_slicechunktosize(revlog, [1, 2, 3], 3)) |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
224 [[1], [2, 3]] |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
225 >>> list(_slicechunktosize(revlog, [3, 5], 3)) |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
226 [[3], [5]] |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
227 >>> list(_slicechunktosize(revlog, [3, 4, 5], 3)) |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
228 [[3], [4, 5]] |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
229 >>> list(_slicechunktosize(revlog, [5, 6, 7, 8], 3)) |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
230 [[5], [6, 7, 8]] |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
231 >>> list(_slicechunktosize(revlog, [0, 1, 2, 3, 4, 5, 6, 7, 8], 3)) |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
232 [[0], [1, 2], [3], [5], [6, 7, 8]] |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
233 |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
234 Case with too large individual chunk (must return valid chunk) |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
235 >>> list(_slicechunktosize(revlog, [0, 1], 2)) |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
236 [[0], [1]] |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
237 >>> list(_slicechunktosize(revlog, [1, 3], 1)) |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
238 [[1], [3]] |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
239 >>> list(_slicechunktosize(revlog, [3, 4, 5], 2)) |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
240 [[3], [5]] |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
241 |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
242 == mixed case == |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
243 >>> revlog = _testrevlog(data, snapshot=[0, 1, 2]) |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
244 >>> list(_slicechunktosize(revlog, list(range(9)), 5)) |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
245 [[0, 1], [2], [3, 4, 5], [6, 7, 8]] |
38642
e59e27e52297
revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents:
38641
diff
changeset
|
246 """ |
e59e27e52297
revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents:
38641
diff
changeset
|
247 assert targetsize is None or 0 <= targetsize |
40642
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
248 startdata = revlog.start(revs[0]) |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
249 enddata = revlog.end(revs[-1]) |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
250 fullspan = enddata - startdata |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
251 if targetsize is None or fullspan <= targetsize: |
38642
e59e27e52297
revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents:
38641
diff
changeset
|
252 yield revs |
e59e27e52297
revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents:
38641
diff
changeset
|
253 return |
e59e27e52297
revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents:
38641
diff
changeset
|
254 |
e59e27e52297
revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents:
38641
diff
changeset
|
255 startrevidx = 0 |
40657
2eb48aa0acce
sparse-revlog: align endrevidx usages in the _slicechunktosize
Boris Feld <boris.feld@octobus.net>
parents:
40654
diff
changeset
|
256 endrevidx = 1 |
38642
e59e27e52297
revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents:
38641
diff
changeset
|
257 iterrevs = enumerate(revs) |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
258 next(iterrevs) # skip first rev. |
40642
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
259 # first step: get snapshots out of the way |
38642
e59e27e52297
revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents:
38641
diff
changeset
|
260 for idx, r in iterrevs: |
e59e27e52297
revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents:
38641
diff
changeset
|
261 span = revlog.end(r) - startdata |
40642
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
262 snapshot = revlog.issnapshot(r) |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
263 if span <= targetsize and snapshot: |
40657
2eb48aa0acce
sparse-revlog: align endrevidx usages in the _slicechunktosize
Boris Feld <boris.feld@octobus.net>
parents:
40654
diff
changeset
|
264 endrevidx = idx + 1 |
38642
e59e27e52297
revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents:
38641
diff
changeset
|
265 else: |
40657
2eb48aa0acce
sparse-revlog: align endrevidx usages in the _slicechunktosize
Boris Feld <boris.feld@octobus.net>
parents:
40654
diff
changeset
|
266 chunk = _trimchunk(revlog, revs, startrevidx, endrevidx) |
38642
e59e27e52297
revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents:
38641
diff
changeset
|
267 if chunk: |
e59e27e52297
revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents:
38641
diff
changeset
|
268 yield chunk |
e59e27e52297
revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents:
38641
diff
changeset
|
269 startrevidx = idx |
e59e27e52297
revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents:
38641
diff
changeset
|
270 startdata = revlog.start(r) |
40657
2eb48aa0acce
sparse-revlog: align endrevidx usages in the _slicechunktosize
Boris Feld <boris.feld@octobus.net>
parents:
40654
diff
changeset
|
271 endrevidx = idx + 1 |
40642
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
272 if not snapshot: |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
273 break |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
274 |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
275 # for the others, we use binary slicing to quickly converge toward valid |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
276 # chunks (otherwise, we might end up looking for start/end of many |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
277 # revisions). This logic is not looking for the perfect slicing point, it |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
278 # focuses on quickly converging toward valid chunks. |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
279 nbitem = len(revs) |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
280 while (enddata - startdata) > targetsize: |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
281 endrevidx = nbitem |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
282 if nbitem - startrevidx <= 1: |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
283 break # protect against individual chunk larger than limit |
40642
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
284 localenddata = revlog.end(revs[endrevidx - 1]) |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
285 span = localenddata - startdata |
40654
fd1d41ccbe38
sparse-revlog: use `span` variable as intended
Boris Feld <boris.feld@octobus.net>
parents:
40642
diff
changeset
|
286 while span > targetsize: |
40642
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
287 if endrevidx - startrevidx <= 1: |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
288 break # protect against individual chunk larger than limit |
40642
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
289 endrevidx -= (endrevidx - startrevidx) // 2 |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
290 localenddata = revlog.end(revs[endrevidx - 1]) |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
291 span = localenddata - startdata |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
292 chunk = _trimchunk(revlog, revs, startrevidx, endrevidx) |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
293 if chunk: |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
294 yield chunk |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
295 startrevidx = endrevidx |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
296 startdata = revlog.start(revs[startrevidx]) |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
297 |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
298 chunk = _trimchunk(revlog, revs, startrevidx) |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
299 if chunk: |
9c3c697267db
sparse-revlog: rework the way we enforce chunk size limit
Boris Feld <boris.feld@octobus.net>
parents:
40641
diff
changeset
|
300 yield chunk |
38642
e59e27e52297
revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents:
38641
diff
changeset
|
301 |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
302 |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
303 def _slicechunktodensity(revlog, revs, targetdensity=0.5, mingapsize=0): |
38641
feba6be0941b
revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents:
38640
diff
changeset
|
304 """slice revs to reduce the amount of unrelated data to be read from disk. |
feba6be0941b
revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents:
38640
diff
changeset
|
305 |
feba6be0941b
revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents:
38640
diff
changeset
|
306 ``revs`` is sliced into groups that should be read in one time. |
feba6be0941b
revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents:
38640
diff
changeset
|
307 Assume that revs are sorted. |
feba6be0941b
revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents:
38640
diff
changeset
|
308 |
feba6be0941b
revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents:
38640
diff
changeset
|
309 The initial chunk is sliced until the overall density (payload/chunks-span |
feba6be0941b
revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents:
38640
diff
changeset
|
310 ratio) is above `targetdensity`. No gap smaller than `mingapsize` is |
feba6be0941b
revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents:
38640
diff
changeset
|
311 skipped. |
feba6be0941b
revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents:
38640
diff
changeset
|
312 |
feba6be0941b
revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents:
38640
diff
changeset
|
313 >>> revlog = _testrevlog([ |
feba6be0941b
revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents:
38640
diff
changeset
|
314 ... 5, #00 (5) |
feba6be0941b
revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents:
38640
diff
changeset
|
315 ... 10, #01 (5) |
feba6be0941b
revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents:
38640
diff
changeset
|
316 ... 12, #02 (2) |
feba6be0941b
revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents:
38640
diff
changeset
|
317 ... 12, #03 (empty) |
feba6be0941b
revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents:
38640
diff
changeset
|
318 ... 27, #04 (15) |
feba6be0941b
revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents:
38640
diff
changeset
|
319 ... 31, #05 (4) |
feba6be0941b
revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents:
38640
diff
changeset
|
320 ... 31, #06 (empty) |
feba6be0941b
revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents:
38640
diff
changeset
|
321 ... 42, #07 (11) |
feba6be0941b
revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents:
38640
diff
changeset
|
322 ... 47, #08 (5) |
feba6be0941b
revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents:
38640
diff
changeset
|
323 ... 47, #09 (empty) |
feba6be0941b
revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents:
38640
diff
changeset
|
324 ... 48, #10 (1) |
feba6be0941b
revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents:
38640
diff
changeset
|
325 ... 51, #11 (3) |
feba6be0941b
revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents:
38640
diff
changeset
|
326 ... 74, #12 (23) |
feba6be0941b
revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents:
38640
diff
changeset
|
327 ... 85, #13 (11) |
feba6be0941b
revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents:
38640
diff
changeset
|
328 ... 86, #14 (1) |
feba6be0941b
revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents:
38640
diff
changeset
|
329 ... 91, #15 (5) |
feba6be0941b
revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents:
38640
diff
changeset
|
330 ... ]) |
feba6be0941b
revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents:
38640
diff
changeset
|
331 |
38655
cd1c484e31e8
revlog: adjust doctest examples to be portable to Python 3
Augie Fackler <augie@google.com>
parents:
38644
diff
changeset
|
332 >>> list(_slicechunktodensity(revlog, list(range(16)))) |
38641
feba6be0941b
revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents:
38640
diff
changeset
|
333 [[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]] |
feba6be0941b
revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents:
38640
diff
changeset
|
334 >>> list(_slicechunktodensity(revlog, [0, 15])) |
feba6be0941b
revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents:
38640
diff
changeset
|
335 [[0], [15]] |
feba6be0941b
revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents:
38640
diff
changeset
|
336 >>> list(_slicechunktodensity(revlog, [0, 11, 15])) |
feba6be0941b
revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents:
38640
diff
changeset
|
337 [[0], [11], [15]] |
feba6be0941b
revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents:
38640
diff
changeset
|
338 >>> list(_slicechunktodensity(revlog, [0, 11, 13, 15])) |
feba6be0941b
revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents:
38640
diff
changeset
|
339 [[0], [11, 13, 15]] |
feba6be0941b
revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents:
38640
diff
changeset
|
340 >>> list(_slicechunktodensity(revlog, [1, 2, 3, 5, 8, 10, 11, 14])) |
feba6be0941b
revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents:
38640
diff
changeset
|
341 [[1, 2], [5, 8, 10, 11], [14]] |
feba6be0941b
revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents:
38640
diff
changeset
|
342 >>> list(_slicechunktodensity(revlog, [1, 2, 3, 5, 8, 10, 11, 14], |
feba6be0941b
revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents:
38640
diff
changeset
|
343 ... mingapsize=20)) |
feba6be0941b
revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents:
38640
diff
changeset
|
344 [[1, 2, 3, 5, 8, 10, 11], [14]] |
feba6be0941b
revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents:
38640
diff
changeset
|
345 >>> list(_slicechunktodensity(revlog, [1, 2, 3, 5, 8, 10, 11, 14], |
feba6be0941b
revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents:
38640
diff
changeset
|
346 ... targetdensity=0.95)) |
feba6be0941b
revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents:
38640
diff
changeset
|
347 [[1, 2], [5], [8, 10, 11], [14]] |
feba6be0941b
revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents:
38640
diff
changeset
|
348 >>> list(_slicechunktodensity(revlog, [1, 2, 3, 5, 8, 10, 11, 14], |
feba6be0941b
revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents:
38640
diff
changeset
|
349 ... targetdensity=0.95, mingapsize=12)) |
feba6be0941b
revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents:
38640
diff
changeset
|
350 [[1, 2], [5, 8, 10, 11], [14]] |
feba6be0941b
revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents:
38640
diff
changeset
|
351 """ |
34824
e2ad93bcc084
revlog: introduce an experimental flag to slice chunks reads when too sparse
Paul Morelle <paul.morelle@octobus.net>
parents:
34823
diff
changeset
|
352 start = revlog.start |
e2ad93bcc084
revlog: introduce an experimental flag to slice chunks reads when too sparse
Paul Morelle <paul.morelle@octobus.net>
parents:
34823
diff
changeset
|
353 length = revlog.length |
e2ad93bcc084
revlog: introduce an experimental flag to slice chunks reads when too sparse
Paul Morelle <paul.morelle@octobus.net>
parents:
34823
diff
changeset
|
354 |
34880
9e18ab7f7240
sparse-read: move from a recursive-based approach to a heap-based one
Boris Feld <boris.feld@octobus.net>
parents:
34825
diff
changeset
|
355 if len(revs) <= 1: |
9e18ab7f7240
sparse-read: move from a recursive-based approach to a heap-based one
Boris Feld <boris.feld@octobus.net>
parents:
34825
diff
changeset
|
356 yield revs |
9e18ab7f7240
sparse-read: move from a recursive-based approach to a heap-based one
Boris Feld <boris.feld@octobus.net>
parents:
34825
diff
changeset
|
357 return |
34824
e2ad93bcc084
revlog: introduce an experimental flag to slice chunks reads when too sparse
Paul Morelle <paul.morelle@octobus.net>
parents:
34823
diff
changeset
|
358 |
40604
3ac23dad6364
sparse-revlog: drop unused deltainfo parameter from _slicechunktodensity
Boris Feld <boris.feld@octobus.net>
parents:
40603
diff
changeset
|
359 deltachainspan = segmentspan(revlog, revs) |
34880
9e18ab7f7240
sparse-read: move from a recursive-based approach to a heap-based one
Boris Feld <boris.feld@octobus.net>
parents:
34825
diff
changeset
|
360 |
38641
feba6be0941b
revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents:
38640
diff
changeset
|
361 if deltachainspan < mingapsize: |
38635
d083ae26c325
revlog: early return in _slicechunk when span is already small enough
Boris Feld <boris.feld@octobus.net>
parents:
38634
diff
changeset
|
362 yield revs |
d083ae26c325
revlog: early return in _slicechunk when span is already small enough
Boris Feld <boris.feld@octobus.net>
parents:
38634
diff
changeset
|
363 return |
d083ae26c325
revlog: early return in _slicechunk when span is already small enough
Boris Feld <boris.feld@octobus.net>
parents:
38634
diff
changeset
|
364 |
38718
f8762ea73e0d
sparse-revlog: implement algorithm to write sparse delta chains (issue5480)
Paul Morelle <paul.morelle@octobus.net>
parents:
38717
diff
changeset
|
365 readdata = deltachainspan |
40606
bfbfd15d65bd
sparse-revlog: fast-path before computing payload size
Boris Feld <boris.feld@octobus.net>
parents:
40605
diff
changeset
|
366 chainpayload = sum(length(r) for r in revs) |
34824
e2ad93bcc084
revlog: introduce an experimental flag to slice chunks reads when too sparse
Paul Morelle <paul.morelle@octobus.net>
parents:
34823
diff
changeset
|
367 |
34880
9e18ab7f7240
sparse-read: move from a recursive-based approach to a heap-based one
Boris Feld <boris.feld@octobus.net>
parents:
34825
diff
changeset
|
368 if deltachainspan: |
9e18ab7f7240
sparse-read: move from a recursive-based approach to a heap-based one
Boris Feld <boris.feld@octobus.net>
parents:
34825
diff
changeset
|
369 density = chainpayload / float(deltachainspan) |
9e18ab7f7240
sparse-read: move from a recursive-based approach to a heap-based one
Boris Feld <boris.feld@octobus.net>
parents:
34825
diff
changeset
|
370 else: |
9e18ab7f7240
sparse-read: move from a recursive-based approach to a heap-based one
Boris Feld <boris.feld@octobus.net>
parents:
34825
diff
changeset
|
371 density = 1.0 |
34824
e2ad93bcc084
revlog: introduce an experimental flag to slice chunks reads when too sparse
Paul Morelle <paul.morelle@octobus.net>
parents:
34823
diff
changeset
|
372 |
38641
feba6be0941b
revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents:
38640
diff
changeset
|
373 if density >= targetdensity: |
38634
f0ea8b847831
revlog: early return in _slicechunk when density is already good
Paul Morelle <paul.morelle@octobus.net>
parents:
38632
diff
changeset
|
374 yield revs |
f0ea8b847831
revlog: early return in _slicechunk when density is already good
Paul Morelle <paul.morelle@octobus.net>
parents:
38632
diff
changeset
|
375 return |
f0ea8b847831
revlog: early return in _slicechunk when density is already good
Paul Morelle <paul.morelle@octobus.net>
parents:
38632
diff
changeset
|
376 |
34880
9e18ab7f7240
sparse-read: move from a recursive-based approach to a heap-based one
Boris Feld <boris.feld@octobus.net>
parents:
34825
diff
changeset
|
377 # Store the gaps in a heap to have them sorted by decreasing size |
40607
54de23400b2a
sparse-revlog: stop using a heap to track gaps
Boris Feld <boris.feld@octobus.net>
parents:
40606
diff
changeset
|
378 gaps = [] |
34880
9e18ab7f7240
sparse-read: move from a recursive-based approach to a heap-based one
Boris Feld <boris.feld@octobus.net>
parents:
34825
diff
changeset
|
379 prevend = None |
9e18ab7f7240
sparse-read: move from a recursive-based approach to a heap-based one
Boris Feld <boris.feld@octobus.net>
parents:
34825
diff
changeset
|
380 for i, rev in enumerate(revs): |
40604
3ac23dad6364
sparse-revlog: drop unused deltainfo parameter from _slicechunktodensity
Boris Feld <boris.feld@octobus.net>
parents:
40603
diff
changeset
|
381 revstart = start(rev) |
3ac23dad6364
sparse-revlog: drop unused deltainfo parameter from _slicechunktodensity
Boris Feld <boris.feld@octobus.net>
parents:
40603
diff
changeset
|
382 revlen = length(rev) |
34824
e2ad93bcc084
revlog: introduce an experimental flag to slice chunks reads when too sparse
Paul Morelle <paul.morelle@octobus.net>
parents:
34823
diff
changeset
|
383 |
34898
1bde8e8e5de0
sparse-read: ignore trailing empty revs in each read chunk
Paul Morelle <paul.morelle@octobus.net>
parents:
34881
diff
changeset
|
384 # Skip empty revisions to form larger holes |
1bde8e8e5de0
sparse-read: ignore trailing empty revs in each read chunk
Paul Morelle <paul.morelle@octobus.net>
parents:
34881
diff
changeset
|
385 if revlen == 0: |
1bde8e8e5de0
sparse-read: ignore trailing empty revs in each read chunk
Paul Morelle <paul.morelle@octobus.net>
parents:
34881
diff
changeset
|
386 continue |
1bde8e8e5de0
sparse-read: ignore trailing empty revs in each read chunk
Paul Morelle <paul.morelle@octobus.net>
parents:
34881
diff
changeset
|
387 |
34880
9e18ab7f7240
sparse-read: move from a recursive-based approach to a heap-based one
Boris Feld <boris.feld@octobus.net>
parents:
34825
diff
changeset
|
388 if prevend is not None: |
9e18ab7f7240
sparse-read: move from a recursive-based approach to a heap-based one
Boris Feld <boris.feld@octobus.net>
parents:
34825
diff
changeset
|
389 gapsize = revstart - prevend |
34881
8c9b08a0c48c
sparse-read: skip gaps too small to be worth splitting
Paul Morelle <paul.morelle@octobus.net>
parents:
34880
diff
changeset
|
390 # only consider holes that are large enough |
38641
feba6be0941b
revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents:
38640
diff
changeset
|
391 if gapsize > mingapsize: |
40607
54de23400b2a
sparse-revlog: stop using a heap to track gaps
Boris Feld <boris.feld@octobus.net>
parents:
40606
diff
changeset
|
392 gaps.append((gapsize, i)) |
34880
9e18ab7f7240
sparse-read: move from a recursive-based approach to a heap-based one
Boris Feld <boris.feld@octobus.net>
parents:
34825
diff
changeset
|
393 |
9e18ab7f7240
sparse-read: move from a recursive-based approach to a heap-based one
Boris Feld <boris.feld@octobus.net>
parents:
34825
diff
changeset
|
394 prevend = revstart + revlen |
40607
54de23400b2a
sparse-revlog: stop using a heap to track gaps
Boris Feld <boris.feld@octobus.net>
parents:
40606
diff
changeset
|
395 # sort the gaps to pop them from largest to small |
54de23400b2a
sparse-revlog: stop using a heap to track gaps
Boris Feld <boris.feld@octobus.net>
parents:
40606
diff
changeset
|
396 gaps.sort() |
34880
9e18ab7f7240
sparse-read: move from a recursive-based approach to a heap-based one
Boris Feld <boris.feld@octobus.net>
parents:
34825
diff
changeset
|
397 |
9e18ab7f7240
sparse-read: move from a recursive-based approach to a heap-based one
Boris Feld <boris.feld@octobus.net>
parents:
34825
diff
changeset
|
398 # Collect the indices of the largest holes until the density is acceptable |
40608
526ee887c4d5
sparse-revlog: stop using a heap to track selected gap
Boris Feld <boris.feld@octobus.net>
parents:
40607
diff
changeset
|
399 selected = [] |
40607
54de23400b2a
sparse-revlog: stop using a heap to track gaps
Boris Feld <boris.feld@octobus.net>
parents:
40606
diff
changeset
|
400 while gaps and density < targetdensity: |
54de23400b2a
sparse-revlog: stop using a heap to track gaps
Boris Feld <boris.feld@octobus.net>
parents:
40606
diff
changeset
|
401 gapsize, gapidx = gaps.pop() |
34880
9e18ab7f7240
sparse-read: move from a recursive-based approach to a heap-based one
Boris Feld <boris.feld@octobus.net>
parents:
34825
diff
changeset
|
402 |
40608
526ee887c4d5
sparse-revlog: stop using a heap to track selected gap
Boris Feld <boris.feld@octobus.net>
parents:
40607
diff
changeset
|
403 selected.append(gapidx) |
34824
e2ad93bcc084
revlog: introduce an experimental flag to slice chunks reads when too sparse
Paul Morelle <paul.morelle@octobus.net>
parents:
34823
diff
changeset
|
404 |
34880
9e18ab7f7240
sparse-read: move from a recursive-based approach to a heap-based one
Boris Feld <boris.feld@octobus.net>
parents:
34825
diff
changeset
|
405 # the gap sizes are stored as negatives to be sorted decreasingly |
9e18ab7f7240
sparse-read: move from a recursive-based approach to a heap-based one
Boris Feld <boris.feld@octobus.net>
parents:
34825
diff
changeset
|
406 # by the heap |
40607
54de23400b2a
sparse-revlog: stop using a heap to track gaps
Boris Feld <boris.feld@octobus.net>
parents:
40606
diff
changeset
|
407 readdata -= gapsize |
34880
9e18ab7f7240
sparse-read: move from a recursive-based approach to a heap-based one
Boris Feld <boris.feld@octobus.net>
parents:
34825
diff
changeset
|
408 if readdata > 0: |
9e18ab7f7240
sparse-read: move from a recursive-based approach to a heap-based one
Boris Feld <boris.feld@octobus.net>
parents:
34825
diff
changeset
|
409 density = chainpayload / float(readdata) |
9e18ab7f7240
sparse-read: move from a recursive-based approach to a heap-based one
Boris Feld <boris.feld@octobus.net>
parents:
34825
diff
changeset
|
410 else: |
9e18ab7f7240
sparse-read: move from a recursive-based approach to a heap-based one
Boris Feld <boris.feld@octobus.net>
parents:
34825
diff
changeset
|
411 density = 1.0 |
40608
526ee887c4d5
sparse-revlog: stop using a heap to track selected gap
Boris Feld <boris.feld@octobus.net>
parents:
40607
diff
changeset
|
412 selected.sort() |
34824
e2ad93bcc084
revlog: introduce an experimental flag to slice chunks reads when too sparse
Paul Morelle <paul.morelle@octobus.net>
parents:
34823
diff
changeset
|
413 |
34880
9e18ab7f7240
sparse-read: move from a recursive-based approach to a heap-based one
Boris Feld <boris.feld@octobus.net>
parents:
34825
diff
changeset
|
414 # Cut the revs at collected indices |
9e18ab7f7240
sparse-read: move from a recursive-based approach to a heap-based one
Boris Feld <boris.feld@octobus.net>
parents:
34825
diff
changeset
|
415 previdx = 0 |
40608
526ee887c4d5
sparse-revlog: stop using a heap to track selected gap
Boris Feld <boris.feld@octobus.net>
parents:
40607
diff
changeset
|
416 for idx in selected: |
34898
1bde8e8e5de0
sparse-read: ignore trailing empty revs in each read chunk
Paul Morelle <paul.morelle@octobus.net>
parents:
34881
diff
changeset
|
417 |
1bde8e8e5de0
sparse-read: ignore trailing empty revs in each read chunk
Paul Morelle <paul.morelle@octobus.net>
parents:
34881
diff
changeset
|
418 chunk = _trimchunk(revlog, revs, previdx, idx) |
1bde8e8e5de0
sparse-read: ignore trailing empty revs in each read chunk
Paul Morelle <paul.morelle@octobus.net>
parents:
34881
diff
changeset
|
419 if chunk: |
1bde8e8e5de0
sparse-read: ignore trailing empty revs in each read chunk
Paul Morelle <paul.morelle@octobus.net>
parents:
34881
diff
changeset
|
420 yield chunk |
1bde8e8e5de0
sparse-read: ignore trailing empty revs in each read chunk
Paul Morelle <paul.morelle@octobus.net>
parents:
34881
diff
changeset
|
421 |
34880
9e18ab7f7240
sparse-read: move from a recursive-based approach to a heap-based one
Boris Feld <boris.feld@octobus.net>
parents:
34825
diff
changeset
|
422 previdx = idx |
34898
1bde8e8e5de0
sparse-read: ignore trailing empty revs in each read chunk
Paul Morelle <paul.morelle@octobus.net>
parents:
34881
diff
changeset
|
423 |
1bde8e8e5de0
sparse-read: ignore trailing empty revs in each read chunk
Paul Morelle <paul.morelle@octobus.net>
parents:
34881
diff
changeset
|
424 chunk = _trimchunk(revlog, revs, previdx) |
1bde8e8e5de0
sparse-read: ignore trailing empty revs in each read chunk
Paul Morelle <paul.morelle@octobus.net>
parents:
34881
diff
changeset
|
425 if chunk: |
1bde8e8e5de0
sparse-read: ignore trailing empty revs in each read chunk
Paul Morelle <paul.morelle@octobus.net>
parents:
34881
diff
changeset
|
426 yield chunk |
34824
e2ad93bcc084
revlog: introduce an experimental flag to slice chunks reads when too sparse
Paul Morelle <paul.morelle@octobus.net>
parents:
34823
diff
changeset
|
427 |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
428 |
39330
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
429 def _trimchunk(revlog, revs, startidx, endidx=None): |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
430 """returns revs[startidx:endidx] without empty trailing revs |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
431 |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
432 Doctest Setup |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
433 >>> revlog = _testrevlog([ |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
434 ... 5, #0 |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
435 ... 10, #1 |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
436 ... 12, #2 |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
437 ... 12, #3 (empty) |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
438 ... 17, #4 |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
439 ... 21, #5 |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
440 ... 21, #6 (empty) |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
441 ... ]) |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
442 |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
443 Contiguous cases: |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
444 >>> _trimchunk(revlog, [0, 1, 2, 3, 4, 5, 6], 0) |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
445 [0, 1, 2, 3, 4, 5] |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
446 >>> _trimchunk(revlog, [0, 1, 2, 3, 4, 5, 6], 0, 5) |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
447 [0, 1, 2, 3, 4] |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
448 >>> _trimchunk(revlog, [0, 1, 2, 3, 4, 5, 6], 0, 4) |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
449 [0, 1, 2] |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
450 >>> _trimchunk(revlog, [0, 1, 2, 3, 4, 5, 6], 2, 4) |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
451 [2] |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
452 >>> _trimchunk(revlog, [0, 1, 2, 3, 4, 5, 6], 3) |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
453 [3, 4, 5] |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
454 >>> _trimchunk(revlog, [0, 1, 2, 3, 4, 5, 6], 3, 5) |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
455 [3, 4] |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
456 |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
457 Discontiguous cases: |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
458 >>> _trimchunk(revlog, [1, 3, 5, 6], 0) |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
459 [1, 3, 5] |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
460 >>> _trimchunk(revlog, [1, 3, 5, 6], 0, 2) |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
461 [1] |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
462 >>> _trimchunk(revlog, [1, 3, 5, 6], 1, 3) |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
463 [3, 5] |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
464 >>> _trimchunk(revlog, [1, 3, 5, 6], 1) |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
465 [3, 5] |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
466 """ |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
467 length = revlog.length |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
468 |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
469 if endidx is None: |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
470 endidx = len(revs) |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
471 |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
472 # If we have a non-emtpy delta candidate, there are nothing to trim |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
473 if revs[endidx - 1] < len(revlog): |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
474 # Trim empty revs at the end, except the very first revision of a chain |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
475 while ( |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
476 endidx > 1 and endidx > startidx and length(revs[endidx - 1]) == 0 |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
477 ): |
39330
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
478 endidx -= 1 |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
479 |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
480 return revs[startidx:endidx] |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
481 |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
482 |
40605
a32ccd32982b
sparse-revlog: drop unused deltainfo parameter from segmentspan
Boris Feld <boris.feld@octobus.net>
parents:
40604
diff
changeset
|
483 def segmentspan(revlog, revs): |
39330
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
484 """Get the byte span of a segment of revisions |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
485 |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
486 revs is a sorted array of revision numbers |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
487 |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
488 >>> revlog = _testrevlog([ |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
489 ... 5, #0 |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
490 ... 10, #1 |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
491 ... 12, #2 |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
492 ... 12, #3 (empty) |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
493 ... 17, #4 |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
494 ... ]) |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
495 |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
496 >>> segmentspan(revlog, [0, 1, 2, 3, 4]) |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
497 17 |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
498 >>> segmentspan(revlog, [0, 4]) |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
499 17 |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
500 >>> segmentspan(revlog, [3, 4]) |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
501 5 |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
502 >>> segmentspan(revlog, [1, 2, 3,]) |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
503 7 |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
504 >>> segmentspan(revlog, [1, 3]) |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
505 7 |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
506 """ |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
507 if not revs: |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
508 return 0 |
40605
a32ccd32982b
sparse-revlog: drop unused deltainfo parameter from segmentspan
Boris Feld <boris.feld@octobus.net>
parents:
40604
diff
changeset
|
509 end = revlog.end(revs[-1]) |
39330
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
510 return end - revlog.start(revs[0]) |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
511 |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
512 |
39331
fd0150a3c2fe
revlogdeltas: extra fulltext building in its own function
Boris Feld <boris.feld@octobus.net>
parents:
39330
diff
changeset
|
513 def _textfromdelta(fh, revlog, baserev, delta, p1, p2, flags, expectednode): |
fd0150a3c2fe
revlogdeltas: extra fulltext building in its own function
Boris Feld <boris.feld@octobus.net>
parents:
39330
diff
changeset
|
514 """build full text from a (base, delta) pair and other metadata""" |
fd0150a3c2fe
revlogdeltas: extra fulltext building in its own function
Boris Feld <boris.feld@octobus.net>
parents:
39330
diff
changeset
|
515 # special case deltas which replace entire base; no need to decode |
fd0150a3c2fe
revlogdeltas: extra fulltext building in its own function
Boris Feld <boris.feld@octobus.net>
parents:
39330
diff
changeset
|
516 # base revision. this neatly avoids censored bases, which throw when |
fd0150a3c2fe
revlogdeltas: extra fulltext building in its own function
Boris Feld <boris.feld@octobus.net>
parents:
39330
diff
changeset
|
517 # they're decoded. |
43077
687b865b95ad
formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents:
43076
diff
changeset
|
518 hlen = struct.calcsize(b">lll") |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
519 if delta[:hlen] == mdiff.replacediffheader( |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
520 revlog.rawsize(baserev), len(delta) - hlen |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
521 ): |
39331
fd0150a3c2fe
revlogdeltas: extra fulltext building in its own function
Boris Feld <boris.feld@octobus.net>
parents:
39330
diff
changeset
|
522 fulltext = delta[hlen:] |
fd0150a3c2fe
revlogdeltas: extra fulltext building in its own function
Boris Feld <boris.feld@octobus.net>
parents:
39330
diff
changeset
|
523 else: |
fd0150a3c2fe
revlogdeltas: extra fulltext building in its own function
Boris Feld <boris.feld@octobus.net>
parents:
39330
diff
changeset
|
524 # deltabase is rawtext before changed by flag processors, which is |
fd0150a3c2fe
revlogdeltas: extra fulltext building in its own function
Boris Feld <boris.feld@octobus.net>
parents:
39330
diff
changeset
|
525 # equivalent to non-raw text |
fd0150a3c2fe
revlogdeltas: extra fulltext building in its own function
Boris Feld <boris.feld@octobus.net>
parents:
39330
diff
changeset
|
526 basetext = revlog.revision(baserev, _df=fh, raw=False) |
fd0150a3c2fe
revlogdeltas: extra fulltext building in its own function
Boris Feld <boris.feld@octobus.net>
parents:
39330
diff
changeset
|
527 fulltext = mdiff.patch(basetext, delta) |
fd0150a3c2fe
revlogdeltas: extra fulltext building in its own function
Boris Feld <boris.feld@octobus.net>
parents:
39330
diff
changeset
|
528 |
fd0150a3c2fe
revlogdeltas: extra fulltext building in its own function
Boris Feld <boris.feld@octobus.net>
parents:
39330
diff
changeset
|
529 try: |
42992
dff95420480f
flagprocessors: make `processflagsraw` a module level function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
42878
diff
changeset
|
530 validatehash = flagutil.processflagsraw(revlog, fulltext, flags) |
39331
fd0150a3c2fe
revlogdeltas: extra fulltext building in its own function
Boris Feld <boris.feld@octobus.net>
parents:
39330
diff
changeset
|
531 if validatehash: |
fd0150a3c2fe
revlogdeltas: extra fulltext building in its own function
Boris Feld <boris.feld@octobus.net>
parents:
39330
diff
changeset
|
532 revlog.checkhash(fulltext, expectednode, p1=p1, p2=p2) |
fd0150a3c2fe
revlogdeltas: extra fulltext building in its own function
Boris Feld <boris.feld@octobus.net>
parents:
39330
diff
changeset
|
533 if flags & REVIDX_ISCENSORED: |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
534 raise error.StorageError( |
43077
687b865b95ad
formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents:
43076
diff
changeset
|
535 _(b'node %s is not censored') % expectednode |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
536 ) |
39774
4a2466b2a434
revlog: drop some more error aliases (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39773
diff
changeset
|
537 except error.CensoredNodeError: |
39331
fd0150a3c2fe
revlogdeltas: extra fulltext building in its own function
Boris Feld <boris.feld@octobus.net>
parents:
39330
diff
changeset
|
538 # must pass the censored index flag to add censored revisions |
fd0150a3c2fe
revlogdeltas: extra fulltext building in its own function
Boris Feld <boris.feld@octobus.net>
parents:
39330
diff
changeset
|
539 if not flags & REVIDX_ISCENSORED: |
fd0150a3c2fe
revlogdeltas: extra fulltext building in its own function
Boris Feld <boris.feld@octobus.net>
parents:
39330
diff
changeset
|
540 raise |
fd0150a3c2fe
revlogdeltas: extra fulltext building in its own function
Boris Feld <boris.feld@octobus.net>
parents:
39330
diff
changeset
|
541 return fulltext |
fd0150a3c2fe
revlogdeltas: extra fulltext building in its own function
Boris Feld <boris.feld@octobus.net>
parents:
39330
diff
changeset
|
542 |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
543 |
35638
edc9330acac1
revlog: introduce 'deltainfo' to distinguish from 'delta'
Paul Morelle <paul.morelle@octobus.net>
parents:
35637
diff
changeset
|
544 @attr.s(slots=True, frozen=True) |
edc9330acac1
revlog: introduce 'deltainfo' to distinguish from 'delta'
Paul Morelle <paul.morelle@octobus.net>
parents:
35637
diff
changeset
|
545 class _deltainfo(object): |
edc9330acac1
revlog: introduce 'deltainfo' to distinguish from 'delta'
Paul Morelle <paul.morelle@octobus.net>
parents:
35637
diff
changeset
|
546 distance = attr.ib() |
edc9330acac1
revlog: introduce 'deltainfo' to distinguish from 'delta'
Paul Morelle <paul.morelle@octobus.net>
parents:
35637
diff
changeset
|
547 deltalen = attr.ib() |
edc9330acac1
revlog: introduce 'deltainfo' to distinguish from 'delta'
Paul Morelle <paul.morelle@octobus.net>
parents:
35637
diff
changeset
|
548 data = attr.ib() |
edc9330acac1
revlog: introduce 'deltainfo' to distinguish from 'delta'
Paul Morelle <paul.morelle@octobus.net>
parents:
35637
diff
changeset
|
549 base = attr.ib() |
edc9330acac1
revlog: introduce 'deltainfo' to distinguish from 'delta'
Paul Morelle <paul.morelle@octobus.net>
parents:
35637
diff
changeset
|
550 chainbase = attr.ib() |
edc9330acac1
revlog: introduce 'deltainfo' to distinguish from 'delta'
Paul Morelle <paul.morelle@octobus.net>
parents:
35637
diff
changeset
|
551 chainlen = attr.ib() |
edc9330acac1
revlog: introduce 'deltainfo' to distinguish from 'delta'
Paul Morelle <paul.morelle@octobus.net>
parents:
35637
diff
changeset
|
552 compresseddeltalen = attr.ib() |
39154
e0da43e2f71f
revlog: compute snapshot depth on delta info
Boris Feld <boris.feld@octobus.net>
parents:
39152
diff
changeset
|
553 snapshotdepth = attr.ib() |
35638
edc9330acac1
revlog: introduce 'deltainfo' to distinguish from 'delta'
Paul Morelle <paul.morelle@octobus.net>
parents:
35637
diff
changeset
|
554 |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
555 |
39330
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
556 def isgooddeltainfo(revlog, deltainfo, revinfo): |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
557 """Returns True if the given delta is good. Good means that it is within |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
558 the disk span, disk size, and chain length bounds that we know to be |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
559 performant.""" |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
560 if deltainfo is None: |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
561 return False |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
562 |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
563 # - 'deltainfo.distance' is the distance from the base revision -- |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
564 # bounding it limits the amount of I/O we need to do. |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
565 # - 'deltainfo.compresseddeltalen' is the sum of the total size of |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
566 # deltas we need to apply -- bounding it limits the amount of CPU |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
567 # we consume. |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
568 |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
569 textlen = revinfo.textlen |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
570 defaultmax = textlen * 4 |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
571 maxdist = revlog._maxdeltachainspan |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
572 if not maxdist: |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
573 maxdist = deltainfo.distance # ensure the conditional pass |
39330
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
574 maxdist = max(maxdist, defaultmax) |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
575 |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
576 # Bad delta from read span: |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
577 # |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
578 # If the span of data read is larger than the maximum allowed. |
40603
2f7e531ef3e7
sparse-revlog: skip the span check in the sparse-revlog case
Boris Feld <boris.feld@octobus.net>
parents:
40451
diff
changeset
|
579 # |
2f7e531ef3e7
sparse-revlog: skip the span check in the sparse-revlog case
Boris Feld <boris.feld@octobus.net>
parents:
40451
diff
changeset
|
580 # In the sparse-revlog case, we rely on the associated "sparse reading" |
2f7e531ef3e7
sparse-revlog: skip the span check in the sparse-revlog case
Boris Feld <boris.feld@octobus.net>
parents:
40451
diff
changeset
|
581 # to avoid issue related to the span of data. In theory, it would be |
2f7e531ef3e7
sparse-revlog: skip the span check in the sparse-revlog case
Boris Feld <boris.feld@octobus.net>
parents:
40451
diff
changeset
|
582 # possible to build pathological revlog where delta pattern would lead |
2f7e531ef3e7
sparse-revlog: skip the span check in the sparse-revlog case
Boris Feld <boris.feld@octobus.net>
parents:
40451
diff
changeset
|
583 # to too many reads. However, they do not happen in practice at all. So |
2f7e531ef3e7
sparse-revlog: skip the span check in the sparse-revlog case
Boris Feld <boris.feld@octobus.net>
parents:
40451
diff
changeset
|
584 # we skip the span check entirely. |
2f7e531ef3e7
sparse-revlog: skip the span check in the sparse-revlog case
Boris Feld <boris.feld@octobus.net>
parents:
40451
diff
changeset
|
585 if not revlog._sparserevlog and maxdist < deltainfo.distance: |
39330
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
586 return False |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
587 |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
588 # Bad delta from new delta size: |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
589 # |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
590 # If the delta size is larger than the target text, storing the |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
591 # delta will be inefficient. |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
592 if textlen < deltainfo.deltalen: |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
593 return False |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
594 |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
595 # Bad delta from cumulated payload size: |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
596 # |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
597 # If the sum of delta get larger than K * target text length. |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
598 if textlen * LIMIT_DELTA2TEXT < deltainfo.compresseddeltalen: |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
599 return False |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
600 |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
601 # Bad delta from chain length: |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
602 # |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
603 # If the number of delta in the chain gets too high. |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
604 if revlog._maxchainlen and revlog._maxchainlen < deltainfo.chainlen: |
39330
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
605 return False |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
606 |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
607 # bad delta from intermediate snapshot size limit |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
608 # |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
609 # If an intermediate snapshot size is higher than the limit. The |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
610 # limit exist to prevent endless chain of intermediate delta to be |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
611 # created. |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
612 if ( |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
613 deltainfo.snapshotdepth is not None |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
614 and (textlen >> deltainfo.snapshotdepth) < deltainfo.deltalen |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
615 ): |
39330
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
616 return False |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
617 |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
618 # bad delta if new intermediate snapshot is larger than the previous |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
619 # snapshot |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
620 if ( |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
621 deltainfo.snapshotdepth |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
622 and revlog.length(deltainfo.base) < deltainfo.deltalen |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
623 ): |
39330
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
624 return False |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
625 |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
626 return True |
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
627 |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
628 |
40978
42f59d3f714d
delta: exclude base candidate much smaller than the target
Boris Feld <boris.feld@octobus.net>
parents:
40957
diff
changeset
|
629 # If a revision's full text is that much bigger than a base candidate full |
42f59d3f714d
delta: exclude base candidate much smaller than the target
Boris Feld <boris.feld@octobus.net>
parents:
40957
diff
changeset
|
630 # text's, it is very unlikely that it will produce a valid delta. We no longer |
42f59d3f714d
delta: exclude base candidate much smaller than the target
Boris Feld <boris.feld@octobus.net>
parents:
40957
diff
changeset
|
631 # consider these candidates. |
41033
b373477948df
revlog: limit base to rev size ratio to 500 instead of 50
Boris Feld <boris.feld@octobus.net>
parents:
40979
diff
changeset
|
632 LIMIT_BASE2TEXT = 500 |
40978
42f59d3f714d
delta: exclude base candidate much smaller than the target
Boris Feld <boris.feld@octobus.net>
parents:
40957
diff
changeset
|
633 |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
634 |
39337
37957e07138c
revlogdeltas: move finddeltainfo filtering inside _candidategroups
Boris Feld <boris.feld@octobus.net>
parents:
39336
diff
changeset
|
635 def _candidategroups(revlog, textlen, p1, p2, cachedelta): |
39336
1c6ff52fe9cf
revlogdeltas: split candidate groups selection from the filtering logic
Boris Feld <boris.feld@octobus.net>
parents:
39335
diff
changeset
|
636 """Provides group of revision to be tested as delta base |
1c6ff52fe9cf
revlogdeltas: split candidate groups selection from the filtering logic
Boris Feld <boris.feld@octobus.net>
parents:
39335
diff
changeset
|
637 |
1c6ff52fe9cf
revlogdeltas: split candidate groups selection from the filtering logic
Boris Feld <boris.feld@octobus.net>
parents:
39335
diff
changeset
|
638 This top level function focus on emitting groups with unique and worthwhile |
1c6ff52fe9cf
revlogdeltas: split candidate groups selection from the filtering logic
Boris Feld <boris.feld@octobus.net>
parents:
39335
diff
changeset
|
639 content. See _raw_candidate_groups for details about the group order. |
39334
507f5b1dd7c8
revlogdeltas: extract _getcandidaterevs in a function
Boris Feld <boris.feld@octobus.net>
parents:
39333
diff
changeset
|
640 """ |
39336
1c6ff52fe9cf
revlogdeltas: split candidate groups selection from the filtering logic
Boris Feld <boris.feld@octobus.net>
parents:
39335
diff
changeset
|
641 # should we try to build a delta? |
1c6ff52fe9cf
revlogdeltas: split candidate groups selection from the filtering logic
Boris Feld <boris.feld@octobus.net>
parents:
39335
diff
changeset
|
642 if not (len(revlog) and revlog._storedeltachains): |
39497
5b308a4e6d03
snapshot: use None as a stop value when looking for a good delta
Boris Feld <boris.feld@octobus.net>
parents:
39496
diff
changeset
|
643 yield None |
39336
1c6ff52fe9cf
revlogdeltas: split candidate groups selection from the filtering logic
Boris Feld <boris.feld@octobus.net>
parents:
39335
diff
changeset
|
644 return |
1c6ff52fe9cf
revlogdeltas: split candidate groups selection from the filtering logic
Boris Feld <boris.feld@octobus.net>
parents:
39335
diff
changeset
|
645 |
39337
37957e07138c
revlogdeltas: move finddeltainfo filtering inside _candidategroups
Boris Feld <boris.feld@octobus.net>
parents:
39336
diff
changeset
|
646 deltalength = revlog.length |
37957e07138c
revlogdeltas: move finddeltainfo filtering inside _candidategroups
Boris Feld <boris.feld@octobus.net>
parents:
39336
diff
changeset
|
647 deltaparent = revlog.deltaparent |
40978
42f59d3f714d
delta: exclude base candidate much smaller than the target
Boris Feld <boris.feld@octobus.net>
parents:
40957
diff
changeset
|
648 sparse = revlog._sparserevlog |
39498
04b75f3a3f2a
snapshot: add refining logic at the findeltainfo level
Boris Feld <boris.feld@octobus.net>
parents:
39497
diff
changeset
|
649 good = None |
39337
37957e07138c
revlogdeltas: move finddeltainfo filtering inside _candidategroups
Boris Feld <boris.feld@octobus.net>
parents:
39336
diff
changeset
|
650 |
37957e07138c
revlogdeltas: move finddeltainfo filtering inside _candidategroups
Boris Feld <boris.feld@octobus.net>
parents:
39336
diff
changeset
|
651 deltas_limit = textlen * LIMIT_DELTA2TEXT |
37957e07138c
revlogdeltas: move finddeltainfo filtering inside _candidategroups
Boris Feld <boris.feld@octobus.net>
parents:
39336
diff
changeset
|
652 |
42057
566daffc607d
cleanup: use set literals where possible
Martin von Zweigbergk <martinvonz@google.com>
parents:
41819
diff
changeset
|
653 tested = {nullrev} |
39499
51cec7fb672e
snapshot: also use None as a stop value for `_refinegroup`
Boris Feld <boris.feld@octobus.net>
parents:
39498
diff
changeset
|
654 candidates = _refinedgroups(revlog, p1, p2, cachedelta) |
51cec7fb672e
snapshot: also use None as a stop value for `_refinegroup`
Boris Feld <boris.feld@octobus.net>
parents:
39498
diff
changeset
|
655 while True: |
39500
cc85ebb68ff9
snapshot: turn _refinedgroups into a coroutine
Boris Feld <boris.feld@octobus.net>
parents:
39499
diff
changeset
|
656 temptative = candidates.send(good) |
39499
51cec7fb672e
snapshot: also use None as a stop value for `_refinegroup`
Boris Feld <boris.feld@octobus.net>
parents:
39498
diff
changeset
|
657 if temptative is None: |
51cec7fb672e
snapshot: also use None as a stop value for `_refinegroup`
Boris Feld <boris.feld@octobus.net>
parents:
39498
diff
changeset
|
658 break |
39337
37957e07138c
revlogdeltas: move finddeltainfo filtering inside _candidategroups
Boris Feld <boris.feld@octobus.net>
parents:
39336
diff
changeset
|
659 group = [] |
37957e07138c
revlogdeltas: move finddeltainfo filtering inside _candidategroups
Boris Feld <boris.feld@octobus.net>
parents:
39336
diff
changeset
|
660 for rev in temptative: |
37957e07138c
revlogdeltas: move finddeltainfo filtering inside _candidategroups
Boris Feld <boris.feld@octobus.net>
parents:
39336
diff
changeset
|
661 # skip over empty delta (no need to include them in a chain) |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
662 while revlog._generaldelta and not ( |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
663 rev == nullrev or rev in tested or deltalength(rev) |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
664 ): |
39594
bdb41eaa8b59
snapshot: fix line order when skipping over empty deltas
Boris Feld <boris.feld@octobus.net>
parents:
39505
diff
changeset
|
665 tested.add(rev) |
39337
37957e07138c
revlogdeltas: move finddeltainfo filtering inside _candidategroups
Boris Feld <boris.feld@octobus.net>
parents:
39336
diff
changeset
|
666 rev = deltaparent(rev) |
40957
f960c51eebf3
delta: filter nullrev out first
Boris Feld <boris.feld@octobus.net>
parents:
40709
diff
changeset
|
667 # no need to try a delta against nullrev, this will be done as a |
f960c51eebf3
delta: filter nullrev out first
Boris Feld <boris.feld@octobus.net>
parents:
40709
diff
changeset
|
668 # last resort. |
f960c51eebf3
delta: filter nullrev out first
Boris Feld <boris.feld@octobus.net>
parents:
40709
diff
changeset
|
669 if rev == nullrev: |
f960c51eebf3
delta: filter nullrev out first
Boris Feld <boris.feld@octobus.net>
parents:
40709
diff
changeset
|
670 continue |
39337
37957e07138c
revlogdeltas: move finddeltainfo filtering inside _candidategroups
Boris Feld <boris.feld@octobus.net>
parents:
39336
diff
changeset
|
671 # filter out revision we tested already |
37957e07138c
revlogdeltas: move finddeltainfo filtering inside _candidategroups
Boris Feld <boris.feld@octobus.net>
parents:
39336
diff
changeset
|
672 if rev in tested: |
37957e07138c
revlogdeltas: move finddeltainfo filtering inside _candidategroups
Boris Feld <boris.feld@octobus.net>
parents:
39336
diff
changeset
|
673 continue |
37957e07138c
revlogdeltas: move finddeltainfo filtering inside _candidategroups
Boris Feld <boris.feld@octobus.net>
parents:
39336
diff
changeset
|
674 tested.add(rev) |
37957e07138c
revlogdeltas: move finddeltainfo filtering inside _candidategroups
Boris Feld <boris.feld@octobus.net>
parents:
39336
diff
changeset
|
675 # filter out delta base that will never produce good delta |
37957e07138c
revlogdeltas: move finddeltainfo filtering inside _candidategroups
Boris Feld <boris.feld@octobus.net>
parents:
39336
diff
changeset
|
676 if deltas_limit < revlog.length(rev): |
37957e07138c
revlogdeltas: move finddeltainfo filtering inside _candidategroups
Boris Feld <boris.feld@octobus.net>
parents:
39336
diff
changeset
|
677 continue |
40978
42f59d3f714d
delta: exclude base candidate much smaller than the target
Boris Feld <boris.feld@octobus.net>
parents:
40957
diff
changeset
|
678 if sparse and revlog.rawsize(rev) < (textlen // LIMIT_BASE2TEXT): |
42f59d3f714d
delta: exclude base candidate much smaller than the target
Boris Feld <boris.feld@octobus.net>
parents:
40957
diff
changeset
|
679 continue |
39337
37957e07138c
revlogdeltas: move finddeltainfo filtering inside _candidategroups
Boris Feld <boris.feld@octobus.net>
parents:
39336
diff
changeset
|
680 # no delta for rawtext-changing revs (see "candelta" for why) |
37957e07138c
revlogdeltas: move finddeltainfo filtering inside _candidategroups
Boris Feld <boris.feld@octobus.net>
parents:
39336
diff
changeset
|
681 if revlog.flags(rev) & REVIDX_RAWTEXT_CHANGING_FLAGS: |
37957e07138c
revlogdeltas: move finddeltainfo filtering inside _candidategroups
Boris Feld <boris.feld@octobus.net>
parents:
39336
diff
changeset
|
682 continue |
40979
ba09db267cb6
delta: ignore base whose chains already don't match expectations
Boris Feld <boris.feld@octobus.net>
parents:
40978
diff
changeset
|
683 # If we reach here, we are about to build and test a delta. |
ba09db267cb6
delta: ignore base whose chains already don't match expectations
Boris Feld <boris.feld@octobus.net>
parents:
40978
diff
changeset
|
684 # The delta building process will compute the chaininfo in all |
ba09db267cb6
delta: ignore base whose chains already don't match expectations
Boris Feld <boris.feld@octobus.net>
parents:
40978
diff
changeset
|
685 # case, since that computation is cached, it is fine to access it |
ba09db267cb6
delta: ignore base whose chains already don't match expectations
Boris Feld <boris.feld@octobus.net>
parents:
40978
diff
changeset
|
686 # here too. |
ba09db267cb6
delta: ignore base whose chains already don't match expectations
Boris Feld <boris.feld@octobus.net>
parents:
40978
diff
changeset
|
687 chainlen, chainsize = revlog._chaininfo(rev) |
ba09db267cb6
delta: ignore base whose chains already don't match expectations
Boris Feld <boris.feld@octobus.net>
parents:
40978
diff
changeset
|
688 # if chain will be too long, skip base |
ba09db267cb6
delta: ignore base whose chains already don't match expectations
Boris Feld <boris.feld@octobus.net>
parents:
40978
diff
changeset
|
689 if revlog._maxchainlen and chainlen >= revlog._maxchainlen: |
ba09db267cb6
delta: ignore base whose chains already don't match expectations
Boris Feld <boris.feld@octobus.net>
parents:
40978
diff
changeset
|
690 continue |
ba09db267cb6
delta: ignore base whose chains already don't match expectations
Boris Feld <boris.feld@octobus.net>
parents:
40978
diff
changeset
|
691 # if chain already have too much data, skip base |
ba09db267cb6
delta: ignore base whose chains already don't match expectations
Boris Feld <boris.feld@octobus.net>
parents:
40978
diff
changeset
|
692 if deltas_limit < chainsize: |
ba09db267cb6
delta: ignore base whose chains already don't match expectations
Boris Feld <boris.feld@octobus.net>
parents:
40978
diff
changeset
|
693 continue |
42463
a0b26fc8fbba
deltas: skip if projected delta size does not match text size constraint
Valentin Gatien-Baron <vgatien-baron@janestreet.com>, Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
42057
diff
changeset
|
694 if sparse and revlog.upperboundcomp is not None: |
a0b26fc8fbba
deltas: skip if projected delta size does not match text size constraint
Valentin Gatien-Baron <vgatien-baron@janestreet.com>, Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
42057
diff
changeset
|
695 maxcomp = revlog.upperboundcomp |
a0b26fc8fbba
deltas: skip if projected delta size does not match text size constraint
Valentin Gatien-Baron <vgatien-baron@janestreet.com>, Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
42057
diff
changeset
|
696 basenotsnap = (p1, p2, nullrev) |
a0b26fc8fbba
deltas: skip if projected delta size does not match text size constraint
Valentin Gatien-Baron <vgatien-baron@janestreet.com>, Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
42057
diff
changeset
|
697 if rev not in basenotsnap and revlog.issnapshot(rev): |
a0b26fc8fbba
deltas: skip if projected delta size does not match text size constraint
Valentin Gatien-Baron <vgatien-baron@janestreet.com>, Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
42057
diff
changeset
|
698 snapshotdepth = revlog.snapshotdepth(rev) |
a0b26fc8fbba
deltas: skip if projected delta size does not match text size constraint
Valentin Gatien-Baron <vgatien-baron@janestreet.com>, Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
42057
diff
changeset
|
699 # If text is significantly larger than the base, we can |
a0b26fc8fbba
deltas: skip if projected delta size does not match text size constraint
Valentin Gatien-Baron <vgatien-baron@janestreet.com>, Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
42057
diff
changeset
|
700 # expect the resulting delta to be proportional to the size |
a0b26fc8fbba
deltas: skip if projected delta size does not match text size constraint
Valentin Gatien-Baron <vgatien-baron@janestreet.com>, Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
42057
diff
changeset
|
701 # difference |
a0b26fc8fbba
deltas: skip if projected delta size does not match text size constraint
Valentin Gatien-Baron <vgatien-baron@janestreet.com>, Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
42057
diff
changeset
|
702 revsize = revlog.rawsize(rev) |
a0b26fc8fbba
deltas: skip if projected delta size does not match text size constraint
Valentin Gatien-Baron <vgatien-baron@janestreet.com>, Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
42057
diff
changeset
|
703 rawsizedistance = max(textlen - revsize, 0) |
a0b26fc8fbba
deltas: skip if projected delta size does not match text size constraint
Valentin Gatien-Baron <vgatien-baron@janestreet.com>, Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
42057
diff
changeset
|
704 # use an estimate of the compression upper bound. |
a0b26fc8fbba
deltas: skip if projected delta size does not match text size constraint
Valentin Gatien-Baron <vgatien-baron@janestreet.com>, Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
42057
diff
changeset
|
705 lowestrealisticdeltalen = rawsizedistance // maxcomp |
a0b26fc8fbba
deltas: skip if projected delta size does not match text size constraint
Valentin Gatien-Baron <vgatien-baron@janestreet.com>, Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
42057
diff
changeset
|
706 |
a0b26fc8fbba
deltas: skip if projected delta size does not match text size constraint
Valentin Gatien-Baron <vgatien-baron@janestreet.com>, Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
42057
diff
changeset
|
707 # check the absolute constraint on the delta size |
a0b26fc8fbba
deltas: skip if projected delta size does not match text size constraint
Valentin Gatien-Baron <vgatien-baron@janestreet.com>, Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
42057
diff
changeset
|
708 snapshotlimit = textlen >> snapshotdepth |
a0b26fc8fbba
deltas: skip if projected delta size does not match text size constraint
Valentin Gatien-Baron <vgatien-baron@janestreet.com>, Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
42057
diff
changeset
|
709 if snapshotlimit < lowestrealisticdeltalen: |
a0b26fc8fbba
deltas: skip if projected delta size does not match text size constraint
Valentin Gatien-Baron <vgatien-baron@janestreet.com>, Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
42057
diff
changeset
|
710 # delta lower bound is larger than accepted upper bound |
a0b26fc8fbba
deltas: skip if projected delta size does not match text size constraint
Valentin Gatien-Baron <vgatien-baron@janestreet.com>, Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
42057
diff
changeset
|
711 continue |
a0b26fc8fbba
deltas: skip if projected delta size does not match text size constraint
Valentin Gatien-Baron <vgatien-baron@janestreet.com>, Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
42057
diff
changeset
|
712 |
42464
66c27df1be84
deltas: skip if projected delta size is bigger than previous snapshot
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents:
42463
diff
changeset
|
713 # check the relative constraint on the delta size |
66c27df1be84
deltas: skip if projected delta size is bigger than previous snapshot
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents:
42463
diff
changeset
|
714 revlength = revlog.length(rev) |
66c27df1be84
deltas: skip if projected delta size is bigger than previous snapshot
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents:
42463
diff
changeset
|
715 if revlength < lowestrealisticdeltalen: |
66c27df1be84
deltas: skip if projected delta size is bigger than previous snapshot
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents:
42463
diff
changeset
|
716 # delta probable lower bound is larger than target base |
66c27df1be84
deltas: skip if projected delta size is bigger than previous snapshot
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents:
42463
diff
changeset
|
717 continue |
66c27df1be84
deltas: skip if projected delta size is bigger than previous snapshot
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents:
42463
diff
changeset
|
718 |
39337
37957e07138c
revlogdeltas: move finddeltainfo filtering inside _candidategroups
Boris Feld <boris.feld@octobus.net>
parents:
39336
diff
changeset
|
719 group.append(rev) |
39336
1c6ff52fe9cf
revlogdeltas: split candidate groups selection from the filtering logic
Boris Feld <boris.feld@octobus.net>
parents:
39335
diff
changeset
|
720 if group: |
39493
3ca144f1c8dd
snapshot: search for unrelated but reusable full-snapshot
Boris Feld <boris.feld@octobus.net>
parents:
39492
diff
changeset
|
721 # XXX: in the sparse revlog case, group can become large, |
3ca144f1c8dd
snapshot: search for unrelated but reusable full-snapshot
Boris Feld <boris.feld@octobus.net>
parents:
39492
diff
changeset
|
722 # impacting performances. Some bounding or slicing mecanism |
3ca144f1c8dd
snapshot: search for unrelated but reusable full-snapshot
Boris Feld <boris.feld@octobus.net>
parents:
39492
diff
changeset
|
723 # would help to reduce this impact. |
39498
04b75f3a3f2a
snapshot: add refining logic at the findeltainfo level
Boris Feld <boris.feld@octobus.net>
parents:
39497
diff
changeset
|
724 good = yield tuple(group) |
39497
5b308a4e6d03
snapshot: use None as a stop value when looking for a good delta
Boris Feld <boris.feld@octobus.net>
parents:
39496
diff
changeset
|
725 yield None |
39336
1c6ff52fe9cf
revlogdeltas: split candidate groups selection from the filtering logic
Boris Feld <boris.feld@octobus.net>
parents:
39335
diff
changeset
|
726 |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
727 |
39493
3ca144f1c8dd
snapshot: search for unrelated but reusable full-snapshot
Boris Feld <boris.feld@octobus.net>
parents:
39492
diff
changeset
|
728 def _findsnapshots(revlog, cache, start_rev): |
3ca144f1c8dd
snapshot: search for unrelated but reusable full-snapshot
Boris Feld <boris.feld@octobus.net>
parents:
39492
diff
changeset
|
729 """find snapshot from start_rev to tip""" |
43077
687b865b95ad
formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents:
43076
diff
changeset
|
730 if util.safehasattr(revlog.index, b'findsnapshots'): |
41108
38e88450138c
delta: have a native implementation of _findsnapshot
Boris Feld <boris.feld@octobus.net>
parents:
41079
diff
changeset
|
731 revlog.index.findsnapshots(cache, start_rev) |
38e88450138c
delta: have a native implementation of _findsnapshot
Boris Feld <boris.feld@octobus.net>
parents:
41079
diff
changeset
|
732 else: |
38e88450138c
delta: have a native implementation of _findsnapshot
Boris Feld <boris.feld@octobus.net>
parents:
41079
diff
changeset
|
733 deltaparent = revlog.deltaparent |
38e88450138c
delta: have a native implementation of _findsnapshot
Boris Feld <boris.feld@octobus.net>
parents:
41079
diff
changeset
|
734 issnapshot = revlog.issnapshot |
38e88450138c
delta: have a native implementation of _findsnapshot
Boris Feld <boris.feld@octobus.net>
parents:
41079
diff
changeset
|
735 for rev in revlog.revs(start_rev): |
38e88450138c
delta: have a native implementation of _findsnapshot
Boris Feld <boris.feld@octobus.net>
parents:
41079
diff
changeset
|
736 if issnapshot(rev): |
38e88450138c
delta: have a native implementation of _findsnapshot
Boris Feld <boris.feld@octobus.net>
parents:
41079
diff
changeset
|
737 cache[deltaparent(rev)].append(rev) |
39493
3ca144f1c8dd
snapshot: search for unrelated but reusable full-snapshot
Boris Feld <boris.feld@octobus.net>
parents:
39492
diff
changeset
|
738 |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
739 |
39496
2f9f7889549b
snapshot: introduce an intermediate `_refinedgroups` generator
Boris Feld <boris.feld@octobus.net>
parents:
39495
diff
changeset
|
740 def _refinedgroups(revlog, p1, p2, cachedelta): |
2f9f7889549b
snapshot: introduce an intermediate `_refinedgroups` generator
Boris Feld <boris.feld@octobus.net>
parents:
39495
diff
changeset
|
741 good = None |
39501
993d7e2c8b79
snapshot: make sure we'll never refine delta base from a reused source
Boris Feld <boris.feld@octobus.net>
parents:
39500
diff
changeset
|
742 # First we try to reuse a the delta contained in the bundle. |
993d7e2c8b79
snapshot: make sure we'll never refine delta base from a reused source
Boris Feld <boris.feld@octobus.net>
parents:
39500
diff
changeset
|
743 # (or from the source revlog) |
993d7e2c8b79
snapshot: make sure we'll never refine delta base from a reused source
Boris Feld <boris.feld@octobus.net>
parents:
39500
diff
changeset
|
744 # |
993d7e2c8b79
snapshot: make sure we'll never refine delta base from a reused source
Boris Feld <boris.feld@octobus.net>
parents:
39500
diff
changeset
|
745 # This logic only applies to general delta repositories and can be disabled |
993d7e2c8b79
snapshot: make sure we'll never refine delta base from a reused source
Boris Feld <boris.feld@octobus.net>
parents:
39500
diff
changeset
|
746 # through configuration. Disabling reuse source delta is useful when |
993d7e2c8b79
snapshot: make sure we'll never refine delta base from a reused source
Boris Feld <boris.feld@octobus.net>
parents:
39500
diff
changeset
|
747 # we want to make sure we recomputed "optimal" deltas. |
993d7e2c8b79
snapshot: make sure we'll never refine delta base from a reused source
Boris Feld <boris.feld@octobus.net>
parents:
39500
diff
changeset
|
748 if cachedelta and revlog._generaldelta and revlog._lazydeltabase: |
993d7e2c8b79
snapshot: make sure we'll never refine delta base from a reused source
Boris Feld <boris.feld@octobus.net>
parents:
39500
diff
changeset
|
749 # Assume what we received from the server is a good choice |
993d7e2c8b79
snapshot: make sure we'll never refine delta base from a reused source
Boris Feld <boris.feld@octobus.net>
parents:
39500
diff
changeset
|
750 # build delta will reuse the cache |
993d7e2c8b79
snapshot: make sure we'll never refine delta base from a reused source
Boris Feld <boris.feld@octobus.net>
parents:
39500
diff
changeset
|
751 good = yield (cachedelta[0],) |
993d7e2c8b79
snapshot: make sure we'll never refine delta base from a reused source
Boris Feld <boris.feld@octobus.net>
parents:
39500
diff
changeset
|
752 if good is not None: |
993d7e2c8b79
snapshot: make sure we'll never refine delta base from a reused source
Boris Feld <boris.feld@octobus.net>
parents:
39500
diff
changeset
|
753 yield None |
993d7e2c8b79
snapshot: make sure we'll never refine delta base from a reused source
Boris Feld <boris.feld@octobus.net>
parents:
39500
diff
changeset
|
754 return |
41109
3e1960e23e6b
delta: reuse _findsnapshot call from previous stage
Boris Feld <boris.feld@octobus.net>
parents:
41108
diff
changeset
|
755 snapshots = collections.defaultdict(list) |
3e1960e23e6b
delta: reuse _findsnapshot call from previous stage
Boris Feld <boris.feld@octobus.net>
parents:
41108
diff
changeset
|
756 for candidates in _rawgroups(revlog, p1, p2, cachedelta, snapshots): |
39496
2f9f7889549b
snapshot: introduce an intermediate `_refinedgroups` generator
Boris Feld <boris.feld@octobus.net>
parents:
39495
diff
changeset
|
757 good = yield candidates |
2f9f7889549b
snapshot: introduce an intermediate `_refinedgroups` generator
Boris Feld <boris.feld@octobus.net>
parents:
39495
diff
changeset
|
758 if good is not None: |
2f9f7889549b
snapshot: introduce an intermediate `_refinedgroups` generator
Boris Feld <boris.feld@octobus.net>
parents:
39495
diff
changeset
|
759 break |
39502
e4d4361d0bcd
snapshot: try to refine new snapshot base down the chain
Boris Feld <boris.feld@octobus.net>
parents:
39501
diff
changeset
|
760 |
40428
bafa1c4bb7a8
sparse-revlog: only refine delta candidates in the sparse case (issue6006)
Boris Feld <boris.feld@octobus.net>
parents:
39777
diff
changeset
|
761 # If sparse revlog is enabled, we can try to refine the available deltas |
bafa1c4bb7a8
sparse-revlog: only refine delta candidates in the sparse case (issue6006)
Boris Feld <boris.feld@octobus.net>
parents:
39777
diff
changeset
|
762 if not revlog._sparserevlog: |
bafa1c4bb7a8
sparse-revlog: only refine delta candidates in the sparse case (issue6006)
Boris Feld <boris.feld@octobus.net>
parents:
39777
diff
changeset
|
763 yield None |
bafa1c4bb7a8
sparse-revlog: only refine delta candidates in the sparse case (issue6006)
Boris Feld <boris.feld@octobus.net>
parents:
39777
diff
changeset
|
764 return |
bafa1c4bb7a8
sparse-revlog: only refine delta candidates in the sparse case (issue6006)
Boris Feld <boris.feld@octobus.net>
parents:
39777
diff
changeset
|
765 |
39502
e4d4361d0bcd
snapshot: try to refine new snapshot base down the chain
Boris Feld <boris.feld@octobus.net>
parents:
39501
diff
changeset
|
766 # if we have a refinable value, try to refine it |
e4d4361d0bcd
snapshot: try to refine new snapshot base down the chain
Boris Feld <boris.feld@octobus.net>
parents:
39501
diff
changeset
|
767 if good is not None and good not in (p1, p2) and revlog.issnapshot(good): |
e4d4361d0bcd
snapshot: try to refine new snapshot base down the chain
Boris Feld <boris.feld@octobus.net>
parents:
39501
diff
changeset
|
768 # refine snapshot down |
e4d4361d0bcd
snapshot: try to refine new snapshot base down the chain
Boris Feld <boris.feld@octobus.net>
parents:
39501
diff
changeset
|
769 previous = None |
e4d4361d0bcd
snapshot: try to refine new snapshot base down the chain
Boris Feld <boris.feld@octobus.net>
parents:
39501
diff
changeset
|
770 while previous != good: |
e4d4361d0bcd
snapshot: try to refine new snapshot base down the chain
Boris Feld <boris.feld@octobus.net>
parents:
39501
diff
changeset
|
771 previous = good |
e4d4361d0bcd
snapshot: try to refine new snapshot base down the chain
Boris Feld <boris.feld@octobus.net>
parents:
39501
diff
changeset
|
772 base = revlog.deltaparent(good) |
e4d4361d0bcd
snapshot: try to refine new snapshot base down the chain
Boris Feld <boris.feld@octobus.net>
parents:
39501
diff
changeset
|
773 if base == nullrev: |
e4d4361d0bcd
snapshot: try to refine new snapshot base down the chain
Boris Feld <boris.feld@octobus.net>
parents:
39501
diff
changeset
|
774 break |
e4d4361d0bcd
snapshot: try to refine new snapshot base down the chain
Boris Feld <boris.feld@octobus.net>
parents:
39501
diff
changeset
|
775 good = yield (base,) |
39503
5aef5afa8654
snapshot: refine candidate snapshot base upward
Boris Feld <boris.feld@octobus.net>
parents:
39502
diff
changeset
|
776 # refine snapshot up |
41109
3e1960e23e6b
delta: reuse _findsnapshot call from previous stage
Boris Feld <boris.feld@octobus.net>
parents:
41108
diff
changeset
|
777 if not snapshots: |
3e1960e23e6b
delta: reuse _findsnapshot call from previous stage
Boris Feld <boris.feld@octobus.net>
parents:
41108
diff
changeset
|
778 _findsnapshots(revlog, snapshots, good + 1) |
39503
5aef5afa8654
snapshot: refine candidate snapshot base upward
Boris Feld <boris.feld@octobus.net>
parents:
39502
diff
changeset
|
779 previous = None |
5aef5afa8654
snapshot: refine candidate snapshot base upward
Boris Feld <boris.feld@octobus.net>
parents:
39502
diff
changeset
|
780 while good != previous: |
5aef5afa8654
snapshot: refine candidate snapshot base upward
Boris Feld <boris.feld@octobus.net>
parents:
39502
diff
changeset
|
781 previous = good |
5aef5afa8654
snapshot: refine candidate snapshot base upward
Boris Feld <boris.feld@octobus.net>
parents:
39502
diff
changeset
|
782 children = tuple(sorted(c for c in snapshots[good])) |
5aef5afa8654
snapshot: refine candidate snapshot base upward
Boris Feld <boris.feld@octobus.net>
parents:
39502
diff
changeset
|
783 good = yield children |
5aef5afa8654
snapshot: refine candidate snapshot base upward
Boris Feld <boris.feld@octobus.net>
parents:
39502
diff
changeset
|
784 |
39499
51cec7fb672e
snapshot: also use None as a stop value for `_refinegroup`
Boris Feld <boris.feld@octobus.net>
parents:
39498
diff
changeset
|
785 # we have found nothing |
51cec7fb672e
snapshot: also use None as a stop value for `_refinegroup`
Boris Feld <boris.feld@octobus.net>
parents:
39498
diff
changeset
|
786 yield None |
39496
2f9f7889549b
snapshot: introduce an intermediate `_refinedgroups` generator
Boris Feld <boris.feld@octobus.net>
parents:
39495
diff
changeset
|
787 |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
788 |
41109
3e1960e23e6b
delta: reuse _findsnapshot call from previous stage
Boris Feld <boris.feld@octobus.net>
parents:
41108
diff
changeset
|
789 def _rawgroups(revlog, p1, p2, cachedelta, snapshots=None): |
39336
1c6ff52fe9cf
revlogdeltas: split candidate groups selection from the filtering logic
Boris Feld <boris.feld@octobus.net>
parents:
39335
diff
changeset
|
790 """Provides group of revision to be tested as delta base |
1c6ff52fe9cf
revlogdeltas: split candidate groups selection from the filtering logic
Boris Feld <boris.feld@octobus.net>
parents:
39335
diff
changeset
|
791 |
1c6ff52fe9cf
revlogdeltas: split candidate groups selection from the filtering logic
Boris Feld <boris.feld@octobus.net>
parents:
39335
diff
changeset
|
792 This lower level function focus on emitting delta theorically interresting |
1c6ff52fe9cf
revlogdeltas: split candidate groups selection from the filtering logic
Boris Feld <boris.feld@octobus.net>
parents:
39335
diff
changeset
|
793 without looking it any practical details. |
1c6ff52fe9cf
revlogdeltas: split candidate groups selection from the filtering logic
Boris Feld <boris.feld@octobus.net>
parents:
39335
diff
changeset
|
794 |
1c6ff52fe9cf
revlogdeltas: split candidate groups selection from the filtering logic
Boris Feld <boris.feld@octobus.net>
parents:
39335
diff
changeset
|
795 The group order aims at providing fast or small candidates first. |
39334
507f5b1dd7c8
revlogdeltas: extract _getcandidaterevs in a function
Boris Feld <boris.feld@octobus.net>
parents:
39333
diff
changeset
|
796 """ |
507f5b1dd7c8
revlogdeltas: extract _getcandidaterevs in a function
Boris Feld <boris.feld@octobus.net>
parents:
39333
diff
changeset
|
797 gdelta = revlog._generaldelta |
41447
189e06b2d719
revlog: make sure we never use sparserevlog without general delta (issue6056)
Boris Feld <boris.feld@octobus.net>
parents:
41109
diff
changeset
|
798 # gate sparse behind general-delta because of issue6056 |
189e06b2d719
revlog: make sure we never use sparserevlog without general delta (issue6056)
Boris Feld <boris.feld@octobus.net>
parents:
41109
diff
changeset
|
799 sparse = gdelta and revlog._sparserevlog |
39334
507f5b1dd7c8
revlogdeltas: extract _getcandidaterevs in a function
Boris Feld <boris.feld@octobus.net>
parents:
39333
diff
changeset
|
800 curr = len(revlog) |
507f5b1dd7c8
revlogdeltas: extract _getcandidaterevs in a function
Boris Feld <boris.feld@octobus.net>
parents:
39333
diff
changeset
|
801 prev = curr - 1 |
39492
a33f394b2bfd
snapshot: try intermediate snapshot against parents' base
Boris Feld <boris.feld@octobus.net>
parents:
39488
diff
changeset
|
802 deltachain = lambda rev: revlog._deltachain(rev)[0] |
39334
507f5b1dd7c8
revlogdeltas: extract _getcandidaterevs in a function
Boris Feld <boris.feld@octobus.net>
parents:
39333
diff
changeset
|
803 |
39336
1c6ff52fe9cf
revlogdeltas: split candidate groups selection from the filtering logic
Boris Feld <boris.feld@octobus.net>
parents:
39335
diff
changeset
|
804 if gdelta: |
1c6ff52fe9cf
revlogdeltas: split candidate groups selection from the filtering logic
Boris Feld <boris.feld@octobus.net>
parents:
39335
diff
changeset
|
805 # exclude already lazy tested base if any |
1c6ff52fe9cf
revlogdeltas: split candidate groups selection from the filtering logic
Boris Feld <boris.feld@octobus.net>
parents:
39335
diff
changeset
|
806 parents = [p for p in (p1, p2) if p != nullrev] |
39334
507f5b1dd7c8
revlogdeltas: extract _getcandidaterevs in a function
Boris Feld <boris.feld@octobus.net>
parents:
39333
diff
changeset
|
807 |
39336
1c6ff52fe9cf
revlogdeltas: split candidate groups selection from the filtering logic
Boris Feld <boris.feld@octobus.net>
parents:
39335
diff
changeset
|
808 if not revlog._deltabothparents and len(parents) == 2: |
1c6ff52fe9cf
revlogdeltas: split candidate groups selection from the filtering logic
Boris Feld <boris.feld@octobus.net>
parents:
39335
diff
changeset
|
809 parents.sort() |
1c6ff52fe9cf
revlogdeltas: split candidate groups selection from the filtering logic
Boris Feld <boris.feld@octobus.net>
parents:
39335
diff
changeset
|
810 # To minimize the chance of having to build a fulltext, |
1c6ff52fe9cf
revlogdeltas: split candidate groups selection from the filtering logic
Boris Feld <boris.feld@octobus.net>
parents:
39335
diff
changeset
|
811 # pick first whichever parent is closest to us (max rev) |
1c6ff52fe9cf
revlogdeltas: split candidate groups selection from the filtering logic
Boris Feld <boris.feld@octobus.net>
parents:
39335
diff
changeset
|
812 yield (parents[1],) |
1c6ff52fe9cf
revlogdeltas: split candidate groups selection from the filtering logic
Boris Feld <boris.feld@octobus.net>
parents:
39335
diff
changeset
|
813 # then the other one (min rev) if the first did not fit |
1c6ff52fe9cf
revlogdeltas: split candidate groups selection from the filtering logic
Boris Feld <boris.feld@octobus.net>
parents:
39335
diff
changeset
|
814 yield (parents[0],) |
1c6ff52fe9cf
revlogdeltas: split candidate groups selection from the filtering logic
Boris Feld <boris.feld@octobus.net>
parents:
39335
diff
changeset
|
815 elif len(parents) > 0: |
1c6ff52fe9cf
revlogdeltas: split candidate groups selection from the filtering logic
Boris Feld <boris.feld@octobus.net>
parents:
39335
diff
changeset
|
816 # Test all parents (1 or 2), and keep the best candidate |
1c6ff52fe9cf
revlogdeltas: split candidate groups selection from the filtering logic
Boris Feld <boris.feld@octobus.net>
parents:
39335
diff
changeset
|
817 yield parents |
39334
507f5b1dd7c8
revlogdeltas: extract _getcandidaterevs in a function
Boris Feld <boris.feld@octobus.net>
parents:
39333
diff
changeset
|
818 |
39492
a33f394b2bfd
snapshot: try intermediate snapshot against parents' base
Boris Feld <boris.feld@octobus.net>
parents:
39488
diff
changeset
|
819 if sparse and parents: |
41109
3e1960e23e6b
delta: reuse _findsnapshot call from previous stage
Boris Feld <boris.feld@octobus.net>
parents:
41108
diff
changeset
|
820 if snapshots is None: |
3e1960e23e6b
delta: reuse _findsnapshot call from previous stage
Boris Feld <boris.feld@octobus.net>
parents:
41108
diff
changeset
|
821 # map: base-rev: snapshot-rev |
3e1960e23e6b
delta: reuse _findsnapshot call from previous stage
Boris Feld <boris.feld@octobus.net>
parents:
41108
diff
changeset
|
822 snapshots = collections.defaultdict(list) |
39492
a33f394b2bfd
snapshot: try intermediate snapshot against parents' base
Boris Feld <boris.feld@octobus.net>
parents:
39488
diff
changeset
|
823 # See if we can use an existing snapshot in the parent chains to use as |
a33f394b2bfd
snapshot: try intermediate snapshot against parents' base
Boris Feld <boris.feld@octobus.net>
parents:
39488
diff
changeset
|
824 # a base for a new intermediate-snapshot |
39494
e72130f58f5d
snapshot: consider all snapshots in the parents' chains
Boris Feld <boris.feld@octobus.net>
parents:
39493
diff
changeset
|
825 # |
e72130f58f5d
snapshot: consider all snapshots in the parents' chains
Boris Feld <boris.feld@octobus.net>
parents:
39493
diff
changeset
|
826 # search for snapshot in parents delta chain |
e72130f58f5d
snapshot: consider all snapshots in the parents' chains
Boris Feld <boris.feld@octobus.net>
parents:
39493
diff
changeset
|
827 # map: snapshot-level: snapshot-rev |
e72130f58f5d
snapshot: consider all snapshots in the parents' chains
Boris Feld <boris.feld@octobus.net>
parents:
39493
diff
changeset
|
828 parents_snaps = collections.defaultdict(set) |
39504
05a165dc4f55
snapshot: extract parent chain computation
Boris Feld <boris.feld@octobus.net>
parents:
39503
diff
changeset
|
829 candidate_chains = [deltachain(p) for p in parents] |
05a165dc4f55
snapshot: extract parent chain computation
Boris Feld <boris.feld@octobus.net>
parents:
39503
diff
changeset
|
830 for chain in candidate_chains: |
05a165dc4f55
snapshot: extract parent chain computation
Boris Feld <boris.feld@octobus.net>
parents:
39503
diff
changeset
|
831 for idx, s in enumerate(chain): |
39494
e72130f58f5d
snapshot: consider all snapshots in the parents' chains
Boris Feld <boris.feld@octobus.net>
parents:
39493
diff
changeset
|
832 if not revlog.issnapshot(s): |
e72130f58f5d
snapshot: consider all snapshots in the parents' chains
Boris Feld <boris.feld@octobus.net>
parents:
39493
diff
changeset
|
833 break |
e72130f58f5d
snapshot: consider all snapshots in the parents' chains
Boris Feld <boris.feld@octobus.net>
parents:
39493
diff
changeset
|
834 parents_snaps[idx].add(s) |
39495
6a53842727c1
snapshot: consider unrelated snapshots at a similar level first
Boris Feld <boris.feld@octobus.net>
parents:
39494
diff
changeset
|
835 snapfloor = min(parents_snaps[0]) + 1 |
6a53842727c1
snapshot: consider unrelated snapshots at a similar level first
Boris Feld <boris.feld@octobus.net>
parents:
39494
diff
changeset
|
836 _findsnapshots(revlog, snapshots, snapfloor) |
39505
c6b8eab5db19
snapshot: also consider the snapshot chain of one unrelated revision
Boris Feld <boris.feld@octobus.net>
parents:
39504
diff
changeset
|
837 # search for the highest "unrelated" revision |
c6b8eab5db19
snapshot: also consider the snapshot chain of one unrelated revision
Boris Feld <boris.feld@octobus.net>
parents:
39504
diff
changeset
|
838 # |
c6b8eab5db19
snapshot: also consider the snapshot chain of one unrelated revision
Boris Feld <boris.feld@octobus.net>
parents:
39504
diff
changeset
|
839 # Adding snapshots used by "unrelated" revision increase the odd we |
c6b8eab5db19
snapshot: also consider the snapshot chain of one unrelated revision
Boris Feld <boris.feld@octobus.net>
parents:
39504
diff
changeset
|
840 # reuse an independant, yet better snapshot chain. |
c6b8eab5db19
snapshot: also consider the snapshot chain of one unrelated revision
Boris Feld <boris.feld@octobus.net>
parents:
39504
diff
changeset
|
841 # |
c6b8eab5db19
snapshot: also consider the snapshot chain of one unrelated revision
Boris Feld <boris.feld@octobus.net>
parents:
39504
diff
changeset
|
842 # XXX instead of building a set of revisions, we could lazily enumerate |
c6b8eab5db19
snapshot: also consider the snapshot chain of one unrelated revision
Boris Feld <boris.feld@octobus.net>
parents:
39504
diff
changeset
|
843 # over the chains. That would be more efficient, however we stick to |
c6b8eab5db19
snapshot: also consider the snapshot chain of one unrelated revision
Boris Feld <boris.feld@octobus.net>
parents:
39504
diff
changeset
|
844 # simple code for now. |
c6b8eab5db19
snapshot: also consider the snapshot chain of one unrelated revision
Boris Feld <boris.feld@octobus.net>
parents:
39504
diff
changeset
|
845 all_revs = set() |
c6b8eab5db19
snapshot: also consider the snapshot chain of one unrelated revision
Boris Feld <boris.feld@octobus.net>
parents:
39504
diff
changeset
|
846 for chain in candidate_chains: |
c6b8eab5db19
snapshot: also consider the snapshot chain of one unrelated revision
Boris Feld <boris.feld@octobus.net>
parents:
39504
diff
changeset
|
847 all_revs.update(chain) |
c6b8eab5db19
snapshot: also consider the snapshot chain of one unrelated revision
Boris Feld <boris.feld@octobus.net>
parents:
39504
diff
changeset
|
848 other = None |
c6b8eab5db19
snapshot: also consider the snapshot chain of one unrelated revision
Boris Feld <boris.feld@octobus.net>
parents:
39504
diff
changeset
|
849 for r in revlog.revs(prev, snapfloor): |
c6b8eab5db19
snapshot: also consider the snapshot chain of one unrelated revision
Boris Feld <boris.feld@octobus.net>
parents:
39504
diff
changeset
|
850 if r not in all_revs: |
c6b8eab5db19
snapshot: also consider the snapshot chain of one unrelated revision
Boris Feld <boris.feld@octobus.net>
parents:
39504
diff
changeset
|
851 other = r |
c6b8eab5db19
snapshot: also consider the snapshot chain of one unrelated revision
Boris Feld <boris.feld@octobus.net>
parents:
39504
diff
changeset
|
852 break |
c6b8eab5db19
snapshot: also consider the snapshot chain of one unrelated revision
Boris Feld <boris.feld@octobus.net>
parents:
39504
diff
changeset
|
853 if other is not None: |
c6b8eab5db19
snapshot: also consider the snapshot chain of one unrelated revision
Boris Feld <boris.feld@octobus.net>
parents:
39504
diff
changeset
|
854 # To avoid unfair competition, we won't use unrelated intermediate |
c6b8eab5db19
snapshot: also consider the snapshot chain of one unrelated revision
Boris Feld <boris.feld@octobus.net>
parents:
39504
diff
changeset
|
855 # snapshot that are deeper than the ones from the parent delta |
c6b8eab5db19
snapshot: also consider the snapshot chain of one unrelated revision
Boris Feld <boris.feld@octobus.net>
parents:
39504
diff
changeset
|
856 # chain. |
c6b8eab5db19
snapshot: also consider the snapshot chain of one unrelated revision
Boris Feld <boris.feld@octobus.net>
parents:
39504
diff
changeset
|
857 max_depth = max(parents_snaps.keys()) |
c6b8eab5db19
snapshot: also consider the snapshot chain of one unrelated revision
Boris Feld <boris.feld@octobus.net>
parents:
39504
diff
changeset
|
858 chain = deltachain(other) |
c6b8eab5db19
snapshot: also consider the snapshot chain of one unrelated revision
Boris Feld <boris.feld@octobus.net>
parents:
39504
diff
changeset
|
859 for idx, s in enumerate(chain): |
c6b8eab5db19
snapshot: also consider the snapshot chain of one unrelated revision
Boris Feld <boris.feld@octobus.net>
parents:
39504
diff
changeset
|
860 if s < snapfloor: |
c6b8eab5db19
snapshot: also consider the snapshot chain of one unrelated revision
Boris Feld <boris.feld@octobus.net>
parents:
39504
diff
changeset
|
861 continue |
c6b8eab5db19
snapshot: also consider the snapshot chain of one unrelated revision
Boris Feld <boris.feld@octobus.net>
parents:
39504
diff
changeset
|
862 if max_depth < idx: |
c6b8eab5db19
snapshot: also consider the snapshot chain of one unrelated revision
Boris Feld <boris.feld@octobus.net>
parents:
39504
diff
changeset
|
863 break |
c6b8eab5db19
snapshot: also consider the snapshot chain of one unrelated revision
Boris Feld <boris.feld@octobus.net>
parents:
39504
diff
changeset
|
864 if not revlog.issnapshot(s): |
c6b8eab5db19
snapshot: also consider the snapshot chain of one unrelated revision
Boris Feld <boris.feld@octobus.net>
parents:
39504
diff
changeset
|
865 break |
c6b8eab5db19
snapshot: also consider the snapshot chain of one unrelated revision
Boris Feld <boris.feld@octobus.net>
parents:
39504
diff
changeset
|
866 parents_snaps[idx].add(s) |
39494
e72130f58f5d
snapshot: consider all snapshots in the parents' chains
Boris Feld <boris.feld@octobus.net>
parents:
39493
diff
changeset
|
867 # Test them as possible intermediate snapshot base |
e72130f58f5d
snapshot: consider all snapshots in the parents' chains
Boris Feld <boris.feld@octobus.net>
parents:
39493
diff
changeset
|
868 # We test them from highest to lowest level. High level one are more |
e72130f58f5d
snapshot: consider all snapshots in the parents' chains
Boris Feld <boris.feld@octobus.net>
parents:
39493
diff
changeset
|
869 # likely to result in small delta |
39495
6a53842727c1
snapshot: consider unrelated snapshots at a similar level first
Boris Feld <boris.feld@octobus.net>
parents:
39494
diff
changeset
|
870 floor = None |
39494
e72130f58f5d
snapshot: consider all snapshots in the parents' chains
Boris Feld <boris.feld@octobus.net>
parents:
39493
diff
changeset
|
871 for idx, snaps in sorted(parents_snaps.items(), reverse=True): |
39495
6a53842727c1
snapshot: consider unrelated snapshots at a similar level first
Boris Feld <boris.feld@octobus.net>
parents:
39494
diff
changeset
|
872 siblings = set() |
6a53842727c1
snapshot: consider unrelated snapshots at a similar level first
Boris Feld <boris.feld@octobus.net>
parents:
39494
diff
changeset
|
873 for s in snaps: |
6a53842727c1
snapshot: consider unrelated snapshots at a similar level first
Boris Feld <boris.feld@octobus.net>
parents:
39494
diff
changeset
|
874 siblings.update(snapshots[s]) |
6a53842727c1
snapshot: consider unrelated snapshots at a similar level first
Boris Feld <boris.feld@octobus.net>
parents:
39494
diff
changeset
|
875 # Before considering making a new intermediate snapshot, we check |
6a53842727c1
snapshot: consider unrelated snapshots at a similar level first
Boris Feld <boris.feld@octobus.net>
parents:
39494
diff
changeset
|
876 # if an existing snapshot, children of base we consider, would be |
6a53842727c1
snapshot: consider unrelated snapshots at a similar level first
Boris Feld <boris.feld@octobus.net>
parents:
39494
diff
changeset
|
877 # suitable. |
6a53842727c1
snapshot: consider unrelated snapshots at a similar level first
Boris Feld <boris.feld@octobus.net>
parents:
39494
diff
changeset
|
878 # |
6a53842727c1
snapshot: consider unrelated snapshots at a similar level first
Boris Feld <boris.feld@octobus.net>
parents:
39494
diff
changeset
|
879 # It give a change to reuse a delta chain "unrelated" to the |
6a53842727c1
snapshot: consider unrelated snapshots at a similar level first
Boris Feld <boris.feld@octobus.net>
parents:
39494
diff
changeset
|
880 # current revision instead of starting our own. Without such |
6a53842727c1
snapshot: consider unrelated snapshots at a similar level first
Boris Feld <boris.feld@octobus.net>
parents:
39494
diff
changeset
|
881 # re-use, topological branches would keep reopening new chains. |
6a53842727c1
snapshot: consider unrelated snapshots at a similar level first
Boris Feld <boris.feld@octobus.net>
parents:
39494
diff
changeset
|
882 # Creating more and more snapshot as the repository grow. |
6a53842727c1
snapshot: consider unrelated snapshots at a similar level first
Boris Feld <boris.feld@octobus.net>
parents:
39494
diff
changeset
|
883 |
6a53842727c1
snapshot: consider unrelated snapshots at a similar level first
Boris Feld <boris.feld@octobus.net>
parents:
39494
diff
changeset
|
884 if floor is not None: |
6a53842727c1
snapshot: consider unrelated snapshots at a similar level first
Boris Feld <boris.feld@octobus.net>
parents:
39494
diff
changeset
|
885 # We only do this for siblings created after the one in our |
6a53842727c1
snapshot: consider unrelated snapshots at a similar level first
Boris Feld <boris.feld@octobus.net>
parents:
39494
diff
changeset
|
886 # parent's delta chain. Those created before has less chances |
6a53842727c1
snapshot: consider unrelated snapshots at a similar level first
Boris Feld <boris.feld@octobus.net>
parents:
39494
diff
changeset
|
887 # to be valid base since our ancestors had to create a new |
6a53842727c1
snapshot: consider unrelated snapshots at a similar level first
Boris Feld <boris.feld@octobus.net>
parents:
39494
diff
changeset
|
888 # snapshot. |
6a53842727c1
snapshot: consider unrelated snapshots at a similar level first
Boris Feld <boris.feld@octobus.net>
parents:
39494
diff
changeset
|
889 siblings = [r for r in siblings if floor < r] |
6a53842727c1
snapshot: consider unrelated snapshots at a similar level first
Boris Feld <boris.feld@octobus.net>
parents:
39494
diff
changeset
|
890 yield tuple(sorted(siblings)) |
6a53842727c1
snapshot: consider unrelated snapshots at a similar level first
Boris Feld <boris.feld@octobus.net>
parents:
39494
diff
changeset
|
891 # then test the base from our parent's delta chain. |
39494
e72130f58f5d
snapshot: consider all snapshots in the parents' chains
Boris Feld <boris.feld@octobus.net>
parents:
39493
diff
changeset
|
892 yield tuple(sorted(snaps)) |
39495
6a53842727c1
snapshot: consider unrelated snapshots at a similar level first
Boris Feld <boris.feld@octobus.net>
parents:
39494
diff
changeset
|
893 floor = min(snaps) |
39493
3ca144f1c8dd
snapshot: search for unrelated but reusable full-snapshot
Boris Feld <boris.feld@octobus.net>
parents:
39492
diff
changeset
|
894 # No suitable base found in the parent chain, search if any full |
3ca144f1c8dd
snapshot: search for unrelated but reusable full-snapshot
Boris Feld <boris.feld@octobus.net>
parents:
39492
diff
changeset
|
895 # snapshots emitted since parent's base would be a suitable base for an |
3ca144f1c8dd
snapshot: search for unrelated but reusable full-snapshot
Boris Feld <boris.feld@octobus.net>
parents:
39492
diff
changeset
|
896 # intermediate snapshot. |
3ca144f1c8dd
snapshot: search for unrelated but reusable full-snapshot
Boris Feld <boris.feld@octobus.net>
parents:
39492
diff
changeset
|
897 # |
3ca144f1c8dd
snapshot: search for unrelated but reusable full-snapshot
Boris Feld <boris.feld@octobus.net>
parents:
39492
diff
changeset
|
898 # It give a chance to reuse a delta chain unrelated to the current |
3ca144f1c8dd
snapshot: search for unrelated but reusable full-snapshot
Boris Feld <boris.feld@octobus.net>
parents:
39492
diff
changeset
|
899 # revisions instead of starting our own. Without such re-use, |
3ca144f1c8dd
snapshot: search for unrelated but reusable full-snapshot
Boris Feld <boris.feld@octobus.net>
parents:
39492
diff
changeset
|
900 # topological branches would keep reopening new full chains. Creating |
3ca144f1c8dd
snapshot: search for unrelated but reusable full-snapshot
Boris Feld <boris.feld@octobus.net>
parents:
39492
diff
changeset
|
901 # more and more snapshot as the repository grow. |
3ca144f1c8dd
snapshot: search for unrelated but reusable full-snapshot
Boris Feld <boris.feld@octobus.net>
parents:
39492
diff
changeset
|
902 yield tuple(snapshots[nullrev]) |
39492
a33f394b2bfd
snapshot: try intermediate snapshot against parents' base
Boris Feld <boris.feld@octobus.net>
parents:
39488
diff
changeset
|
903 |
39505
c6b8eab5db19
snapshot: also consider the snapshot chain of one unrelated revision
Boris Feld <boris.feld@octobus.net>
parents:
39504
diff
changeset
|
904 if not sparse: |
c6b8eab5db19
snapshot: also consider the snapshot chain of one unrelated revision
Boris Feld <boris.feld@octobus.net>
parents:
39504
diff
changeset
|
905 # other approach failed try against prev to hopefully save us a |
c6b8eab5db19
snapshot: also consider the snapshot chain of one unrelated revision
Boris Feld <boris.feld@octobus.net>
parents:
39504
diff
changeset
|
906 # fulltext. |
c6b8eab5db19
snapshot: also consider the snapshot chain of one unrelated revision
Boris Feld <boris.feld@octobus.net>
parents:
39504
diff
changeset
|
907 yield (prev,) |
39334
507f5b1dd7c8
revlogdeltas: extract _getcandidaterevs in a function
Boris Feld <boris.feld@octobus.net>
parents:
39333
diff
changeset
|
908 |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
909 |
39330
655b5b465953
revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents:
39329
diff
changeset
|
910 class deltacomputer(object): |
35738
f90f6fd130c1
revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents:
35737
diff
changeset
|
911 def __init__(self, revlog): |
f90f6fd130c1
revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents:
35737
diff
changeset
|
912 self.revlog = revlog |
f90f6fd130c1
revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents:
35737
diff
changeset
|
913 |
f90f6fd130c1
revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents:
35737
diff
changeset
|
914 def buildtext(self, revinfo, fh): |
f90f6fd130c1
revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents:
35737
diff
changeset
|
915 """Builds a fulltext version of a revision |
f90f6fd130c1
revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents:
35737
diff
changeset
|
916 |
f90f6fd130c1
revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents:
35737
diff
changeset
|
917 revinfo: _revisioninfo instance that contains all needed info |
f90f6fd130c1
revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents:
35737
diff
changeset
|
918 fh: file handle to either the .i or the .d revlog file, |
f90f6fd130c1
revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents:
35737
diff
changeset
|
919 depending on whether it is inlined or not |
f90f6fd130c1
revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents:
35737
diff
changeset
|
920 """ |
f90f6fd130c1
revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents:
35737
diff
changeset
|
921 btext = revinfo.btext |
f90f6fd130c1
revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents:
35737
diff
changeset
|
922 if btext[0] is not None: |
f90f6fd130c1
revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents:
35737
diff
changeset
|
923 return btext[0] |
f90f6fd130c1
revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents:
35737
diff
changeset
|
924 |
f90f6fd130c1
revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents:
35737
diff
changeset
|
925 revlog = self.revlog |
f90f6fd130c1
revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents:
35737
diff
changeset
|
926 cachedelta = revinfo.cachedelta |
f90f6fd130c1
revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents:
35737
diff
changeset
|
927 baserev = cachedelta[0] |
f90f6fd130c1
revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents:
35737
diff
changeset
|
928 delta = cachedelta[1] |
f90f6fd130c1
revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents:
35737
diff
changeset
|
929 |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
930 fulltext = btext[0] = _textfromdelta( |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
931 fh, |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
932 revlog, |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
933 baserev, |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
934 delta, |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
935 revinfo.p1, |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
936 revinfo.p2, |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
937 revinfo.flags, |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
938 revinfo.node, |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
939 ) |
39331
fd0150a3c2fe
revlogdeltas: extra fulltext building in its own function
Boris Feld <boris.feld@octobus.net>
parents:
39330
diff
changeset
|
940 return fulltext |
35738
f90f6fd130c1
revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents:
35737
diff
changeset
|
941 |
f90f6fd130c1
revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents:
35737
diff
changeset
|
942 def _builddeltadiff(self, base, revinfo, fh): |
f90f6fd130c1
revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents:
35737
diff
changeset
|
943 revlog = self.revlog |
f90f6fd130c1
revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents:
35737
diff
changeset
|
944 t = self.buildtext(revinfo, fh) |
f90f6fd130c1
revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents:
35737
diff
changeset
|
945 if revlog.iscensored(base): |
f90f6fd130c1
revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents:
35737
diff
changeset
|
946 # deltas based on a censored revision must replace the |
f90f6fd130c1
revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents:
35737
diff
changeset
|
947 # full content in one patch, so delta works everywhere |
f90f6fd130c1
revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents:
35737
diff
changeset
|
948 header = mdiff.replacediffheader(revlog.rawsize(base), len(t)) |
f90f6fd130c1
revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents:
35737
diff
changeset
|
949 delta = header + t |
f90f6fd130c1
revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents:
35737
diff
changeset
|
950 else: |
42780
7a89b044eea4
rawdata: update callers in delta utils
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
42468
diff
changeset
|
951 ptext = revlog.rawdata(base, _df=fh) |
35738
f90f6fd130c1
revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents:
35737
diff
changeset
|
952 delta = mdiff.textdiff(ptext, t) |
f90f6fd130c1
revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents:
35737
diff
changeset
|
953 |
f90f6fd130c1
revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents:
35737
diff
changeset
|
954 return delta |
f90f6fd130c1
revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents:
35737
diff
changeset
|
955 |
f90f6fd130c1
revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents:
35737
diff
changeset
|
956 def _builddeltainfo(self, revinfo, base, fh): |
f90f6fd130c1
revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents:
35737
diff
changeset
|
957 # can we use the cached delta? |
42465
6e9ba867a946
delta: move some delta chain related computation earlier in deltainfo
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents:
42464
diff
changeset
|
958 revlog = self.revlog |
6e9ba867a946
delta: move some delta chain related computation earlier in deltainfo
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents:
42464
diff
changeset
|
959 chainbase = revlog.chainbase(base) |
6e9ba867a946
delta: move some delta chain related computation earlier in deltainfo
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents:
42464
diff
changeset
|
960 if revlog._generaldelta: |
6e9ba867a946
delta: move some delta chain related computation earlier in deltainfo
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents:
42464
diff
changeset
|
961 deltabase = base |
6e9ba867a946
delta: move some delta chain related computation earlier in deltainfo
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents:
42464
diff
changeset
|
962 else: |
6e9ba867a946
delta: move some delta chain related computation earlier in deltainfo
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents:
42464
diff
changeset
|
963 deltabase = chainbase |
6e9ba867a946
delta: move some delta chain related computation earlier in deltainfo
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents:
42464
diff
changeset
|
964 snapshotdepth = None |
6e9ba867a946
delta: move some delta chain related computation earlier in deltainfo
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents:
42464
diff
changeset
|
965 if revlog._sparserevlog and deltabase == nullrev: |
6e9ba867a946
delta: move some delta chain related computation earlier in deltainfo
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents:
42464
diff
changeset
|
966 snapshotdepth = 0 |
6e9ba867a946
delta: move some delta chain related computation earlier in deltainfo
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents:
42464
diff
changeset
|
967 elif revlog._sparserevlog and revlog.issnapshot(deltabase): |
6e9ba867a946
delta: move some delta chain related computation earlier in deltainfo
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents:
42464
diff
changeset
|
968 # A delta chain should always be one full snapshot, |
6e9ba867a946
delta: move some delta chain related computation earlier in deltainfo
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents:
42464
diff
changeset
|
969 # zero or more semi-snapshots, and zero or more deltas |
6e9ba867a946
delta: move some delta chain related computation earlier in deltainfo
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents:
42464
diff
changeset
|
970 p1, p2 = revlog.rev(revinfo.p1), revlog.rev(revinfo.p2) |
6e9ba867a946
delta: move some delta chain related computation earlier in deltainfo
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents:
42464
diff
changeset
|
971 if deltabase not in (p1, p2) and revlog.issnapshot(deltabase): |
6e9ba867a946
delta: move some delta chain related computation earlier in deltainfo
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents:
42464
diff
changeset
|
972 snapshotdepth = len(revlog._deltachain(deltabase)[0]) |
39595
a911932d5003
revlog: reuse cached delta for identical base revision (issue5975)
Boris Feld <boris.feld@octobus.net>
parents:
39594
diff
changeset
|
973 delta = None |
a911932d5003
revlog: reuse cached delta for identical base revision (issue5975)
Boris Feld <boris.feld@octobus.net>
parents:
39594
diff
changeset
|
974 if revinfo.cachedelta: |
a911932d5003
revlog: reuse cached delta for identical base revision (issue5975)
Boris Feld <boris.feld@octobus.net>
parents:
39594
diff
changeset
|
975 cachebase, cachediff = revinfo.cachedelta |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
976 # check if the diff still apply |
39595
a911932d5003
revlog: reuse cached delta for identical base revision (issue5975)
Boris Feld <boris.feld@octobus.net>
parents:
39594
diff
changeset
|
977 currentbase = cachebase |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
978 while ( |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
979 currentbase != nullrev |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
980 and currentbase != base |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
981 and self.revlog.length(currentbase) == 0 |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
982 ): |
39595
a911932d5003
revlog: reuse cached delta for identical base revision (issue5975)
Boris Feld <boris.feld@octobus.net>
parents:
39594
diff
changeset
|
983 currentbase = self.revlog.deltaparent(currentbase) |
41819
688fc33e105d
storage: introduce a `revlog.reuse-external-delta` config
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
41447
diff
changeset
|
984 if self.revlog._lazydelta and currentbase == base: |
39595
a911932d5003
revlog: reuse cached delta for identical base revision (issue5975)
Boris Feld <boris.feld@octobus.net>
parents:
39594
diff
changeset
|
985 delta = revinfo.cachedelta[1] |
a911932d5003
revlog: reuse cached delta for identical base revision (issue5975)
Boris Feld <boris.feld@octobus.net>
parents:
39594
diff
changeset
|
986 if delta is None: |
35738
f90f6fd130c1
revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents:
35737
diff
changeset
|
987 delta = self._builddeltadiff(base, revinfo, fh) |
42467
c1c1872d25d1
deltas: skip if projected compressed size does not match text size constraint
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents:
42466
diff
changeset
|
988 # snapshotdept need to be neither None nor 0 level snapshot |
c1c1872d25d1
deltas: skip if projected compressed size does not match text size constraint
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents:
42466
diff
changeset
|
989 if revlog.upperboundcomp is not None and snapshotdepth: |
c1c1872d25d1
deltas: skip if projected compressed size does not match text size constraint
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents:
42466
diff
changeset
|
990 lowestrealisticdeltalen = len(delta) // revlog.upperboundcomp |
c1c1872d25d1
deltas: skip if projected compressed size does not match text size constraint
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents:
42466
diff
changeset
|
991 snapshotlimit = revinfo.textlen >> snapshotdepth |
c1c1872d25d1
deltas: skip if projected compressed size does not match text size constraint
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents:
42466
diff
changeset
|
992 if snapshotlimit < lowestrealisticdeltalen: |
c1c1872d25d1
deltas: skip if projected compressed size does not match text size constraint
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents:
42466
diff
changeset
|
993 return None |
42468
9b5fbe5ead89
deltas: skip if projected compressed size is bigger than previous snapshot
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents:
42467
diff
changeset
|
994 if revlog.length(base) < lowestrealisticdeltalen: |
9b5fbe5ead89
deltas: skip if projected compressed size is bigger than previous snapshot
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents:
42467
diff
changeset
|
995 return None |
35738
f90f6fd130c1
revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents:
35737
diff
changeset
|
996 header, data = revlog.compress(delta) |
f90f6fd130c1
revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents:
35737
diff
changeset
|
997 deltalen = len(header) + len(data) |
f90f6fd130c1
revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents:
35737
diff
changeset
|
998 offset = revlog.end(len(revlog) - 1) |
f90f6fd130c1
revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents:
35737
diff
changeset
|
999 dist = deltalen + offset - revlog.start(chainbase) |
f90f6fd130c1
revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents:
35737
diff
changeset
|
1000 chainlen, compresseddeltalen = revlog._chaininfo(base) |
f90f6fd130c1
revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents:
35737
diff
changeset
|
1001 chainlen += 1 |
f90f6fd130c1
revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents:
35737
diff
changeset
|
1002 compresseddeltalen += deltalen |
39154
e0da43e2f71f
revlog: compute snapshot depth on delta info
Boris Feld <boris.feld@octobus.net>
parents:
39152
diff
changeset
|
1003 |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
1004 return _deltainfo( |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
1005 dist, |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
1006 deltalen, |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
1007 (header, data), |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
1008 deltabase, |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
1009 chainbase, |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
1010 chainlen, |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
1011 compresseddeltalen, |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
1012 snapshotdepth, |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
1013 ) |
35738
f90f6fd130c1
revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents:
35737
diff
changeset
|
1014 |
39333
5d343a24bff5
revlogdeltas: always return a delta info object in finddeltainfo
Boris Feld <boris.feld@octobus.net>
parents:
39332
diff
changeset
|
1015 def _fullsnapshotinfo(self, fh, revinfo): |
5d343a24bff5
revlogdeltas: always return a delta info object in finddeltainfo
Boris Feld <boris.feld@octobus.net>
parents:
39332
diff
changeset
|
1016 curr = len(self.revlog) |
5d343a24bff5
revlogdeltas: always return a delta info object in finddeltainfo
Boris Feld <boris.feld@octobus.net>
parents:
39332
diff
changeset
|
1017 rawtext = self.buildtext(revinfo, fh) |
5d343a24bff5
revlogdeltas: always return a delta info object in finddeltainfo
Boris Feld <boris.feld@octobus.net>
parents:
39332
diff
changeset
|
1018 data = self.revlog.compress(rawtext) |
5d343a24bff5
revlogdeltas: always return a delta info object in finddeltainfo
Boris Feld <boris.feld@octobus.net>
parents:
39332
diff
changeset
|
1019 compresseddeltalen = deltalen = dist = len(data[1]) + len(data[0]) |
5d343a24bff5
revlogdeltas: always return a delta info object in finddeltainfo
Boris Feld <boris.feld@octobus.net>
parents:
39332
diff
changeset
|
1020 deltabase = chainbase = curr |
5d343a24bff5
revlogdeltas: always return a delta info object in finddeltainfo
Boris Feld <boris.feld@octobus.net>
parents:
39332
diff
changeset
|
1021 snapshotdepth = 0 |
5d343a24bff5
revlogdeltas: always return a delta info object in finddeltainfo
Boris Feld <boris.feld@octobus.net>
parents:
39332
diff
changeset
|
1022 chainlen = 1 |
5d343a24bff5
revlogdeltas: always return a delta info object in finddeltainfo
Boris Feld <boris.feld@octobus.net>
parents:
39332
diff
changeset
|
1023 |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
1024 return _deltainfo( |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
1025 dist, |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
1026 deltalen, |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
1027 data, |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
1028 deltabase, |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
1029 chainbase, |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
1030 chainlen, |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
1031 compresseddeltalen, |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
1032 snapshotdepth, |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
1033 ) |
39333
5d343a24bff5
revlogdeltas: always return a delta info object in finddeltainfo
Boris Feld <boris.feld@octobus.net>
parents:
39332
diff
changeset
|
1034 |
35738
f90f6fd130c1
revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents:
35737
diff
changeset
|
1035 def finddeltainfo(self, revinfo, fh): |
f90f6fd130c1
revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents:
35737
diff
changeset
|
1036 """Find an acceptable delta against a candidate revision |
f90f6fd130c1
revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents:
35737
diff
changeset
|
1037 |
f90f6fd130c1
revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents:
35737
diff
changeset
|
1038 revinfo: information about the revision (instance of _revisioninfo) |
f90f6fd130c1
revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents:
35737
diff
changeset
|
1039 fh: file handle to either the .i or the .d revlog file, |
f90f6fd130c1
revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents:
35737
diff
changeset
|
1040 depending on whether it is inlined or not |
f90f6fd130c1
revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents:
35737
diff
changeset
|
1041 |
f90f6fd130c1
revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents:
35737
diff
changeset
|
1042 Returns the first acceptable candidate revision, as ordered by |
39334
507f5b1dd7c8
revlogdeltas: extract _getcandidaterevs in a function
Boris Feld <boris.feld@octobus.net>
parents:
39333
diff
changeset
|
1043 _candidategroups |
39333
5d343a24bff5
revlogdeltas: always return a delta info object in finddeltainfo
Boris Feld <boris.feld@octobus.net>
parents:
39332
diff
changeset
|
1044 |
5d343a24bff5
revlogdeltas: always return a delta info object in finddeltainfo
Boris Feld <boris.feld@octobus.net>
parents:
39332
diff
changeset
|
1045 If no suitable deltabase is found, we return delta info for a full |
5d343a24bff5
revlogdeltas: always return a delta info object in finddeltainfo
Boris Feld <boris.feld@octobus.net>
parents:
39332
diff
changeset
|
1046 snapshot. |
35738
f90f6fd130c1
revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents:
35737
diff
changeset
|
1047 """ |
39085
dbb3e9e44fce
revlog: do not search for delta for empty content
Boris Feld <boris.feld@octobus.net>
parents:
39084
diff
changeset
|
1048 if not revinfo.textlen: |
39333
5d343a24bff5
revlogdeltas: always return a delta info object in finddeltainfo
Boris Feld <boris.feld@octobus.net>
parents:
39332
diff
changeset
|
1049 return self._fullsnapshotinfo(fh, revinfo) |
39085
dbb3e9e44fce
revlog: do not search for delta for empty content
Boris Feld <boris.feld@octobus.net>
parents:
39084
diff
changeset
|
1050 |
39332
6f4b8f607a31
revlogdeltas: move special cases around raw revisions in finddeltainfo
Boris Feld <boris.feld@octobus.net>
parents:
39331
diff
changeset
|
1051 # no delta for flag processor revision (see "candelta" for why) |
6f4b8f607a31
revlogdeltas: move special cases around raw revisions in finddeltainfo
Boris Feld <boris.feld@octobus.net>
parents:
39331
diff
changeset
|
1052 # not calling candelta since only one revision needs test, also to |
6f4b8f607a31
revlogdeltas: move special cases around raw revisions in finddeltainfo
Boris Feld <boris.feld@octobus.net>
parents:
39331
diff
changeset
|
1053 # avoid overhead fetching flags again. |
6f4b8f607a31
revlogdeltas: move special cases around raw revisions in finddeltainfo
Boris Feld <boris.feld@octobus.net>
parents:
39331
diff
changeset
|
1054 if revinfo.flags & REVIDX_RAWTEXT_CHANGING_FLAGS: |
39333
5d343a24bff5
revlogdeltas: always return a delta info object in finddeltainfo
Boris Feld <boris.feld@octobus.net>
parents:
39332
diff
changeset
|
1055 return self._fullsnapshotinfo(fh, revinfo) |
39332
6f4b8f607a31
revlogdeltas: move special cases around raw revisions in finddeltainfo
Boris Feld <boris.feld@octobus.net>
parents:
39331
diff
changeset
|
1056 |
35738
f90f6fd130c1
revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents:
35737
diff
changeset
|
1057 cachedelta = revinfo.cachedelta |
f90f6fd130c1
revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents:
35737
diff
changeset
|
1058 p1 = revinfo.p1 |
f90f6fd130c1
revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents:
35737
diff
changeset
|
1059 p2 = revinfo.p2 |
f90f6fd130c1
revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents:
35737
diff
changeset
|
1060 revlog = self.revlog |
f90f6fd130c1
revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents:
35737
diff
changeset
|
1061 |
f90f6fd130c1
revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents:
35737
diff
changeset
|
1062 deltainfo = None |
39335
1441eb38849f
revlogdeltas: pass revision number to _candidatesgroups
Boris Feld <boris.feld@octobus.net>
parents:
39334
diff
changeset
|
1063 p1r, p2r = revlog.rev(p1), revlog.rev(p2) |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
1064 groups = _candidategroups( |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
1065 self.revlog, revinfo.textlen, p1r, p2r, cachedelta |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42992
diff
changeset
|
1066 ) |
39497
5b308a4e6d03
snapshot: use None as a stop value when looking for a good delta
Boris Feld <boris.feld@octobus.net>
parents:
39496
diff
changeset
|
1067 candidaterevs = next(groups) |
5b308a4e6d03
snapshot: use None as a stop value when looking for a good delta
Boris Feld <boris.feld@octobus.net>
parents:
39496
diff
changeset
|
1068 while candidaterevs is not None: |
35738
f90f6fd130c1
revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents:
35737
diff
changeset
|
1069 nominateddeltas = [] |
39498
04b75f3a3f2a
snapshot: add refining logic at the findeltainfo level
Boris Feld <boris.feld@octobus.net>
parents:
39497
diff
changeset
|
1070 if deltainfo is not None: |
04b75f3a3f2a
snapshot: add refining logic at the findeltainfo level
Boris Feld <boris.feld@octobus.net>
parents:
39497
diff
changeset
|
1071 # if we already found a good delta, |
04b75f3a3f2a
snapshot: add refining logic at the findeltainfo level
Boris Feld <boris.feld@octobus.net>
parents:
39497
diff
changeset
|
1072 # challenge it against refined candidates |
04b75f3a3f2a
snapshot: add refining logic at the findeltainfo level
Boris Feld <boris.feld@octobus.net>
parents:
39497
diff
changeset
|
1073 nominateddeltas.append(deltainfo) |
35738
f90f6fd130c1
revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents:
35737
diff
changeset
|
1074 for candidaterev in candidaterevs: |
f90f6fd130c1
revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents:
35737
diff
changeset
|
1075 candidatedelta = self._builddeltainfo(revinfo, candidaterev, fh) |
42466
465f2d0df9ae
deltas: accept and skip None return for delta info
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents:
42465
diff
changeset
|
1076 if candidatedelta is not None: |
465f2d0df9ae
deltas: accept and skip None return for delta info
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents:
42465
diff
changeset
|
1077 if isgooddeltainfo(self.revlog, candidatedelta, revinfo): |
465f2d0df9ae
deltas: accept and skip None return for delta info
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents:
42465
diff
changeset
|
1078 nominateddeltas.append(candidatedelta) |
35738
f90f6fd130c1
revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents:
35737
diff
changeset
|
1079 if nominateddeltas: |
f90f6fd130c1
revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents:
35737
diff
changeset
|
1080 deltainfo = min(nominateddeltas, key=lambda x: x.deltalen) |
39498
04b75f3a3f2a
snapshot: add refining logic at the findeltainfo level
Boris Feld <boris.feld@octobus.net>
parents:
39497
diff
changeset
|
1081 if deltainfo is not None: |
04b75f3a3f2a
snapshot: add refining logic at the findeltainfo level
Boris Feld <boris.feld@octobus.net>
parents:
39497
diff
changeset
|
1082 candidaterevs = groups.send(deltainfo.base) |
04b75f3a3f2a
snapshot: add refining logic at the findeltainfo level
Boris Feld <boris.feld@octobus.net>
parents:
39497
diff
changeset
|
1083 else: |
04b75f3a3f2a
snapshot: add refining logic at the findeltainfo level
Boris Feld <boris.feld@octobus.net>
parents:
39497
diff
changeset
|
1084 candidaterevs = next(groups) |
35738
f90f6fd130c1
revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents:
35737
diff
changeset
|
1085 |
39333
5d343a24bff5
revlogdeltas: always return a delta info object in finddeltainfo
Boris Feld <boris.feld@octobus.net>
parents:
39332
diff
changeset
|
1086 if deltainfo is None: |
5d343a24bff5
revlogdeltas: always return a delta info object in finddeltainfo
Boris Feld <boris.feld@octobus.net>
parents:
39332
diff
changeset
|
1087 deltainfo = self._fullsnapshotinfo(fh, revinfo) |
35738
f90f6fd130c1
revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents:
35737
diff
changeset
|
1088 return deltainfo |