Mercurial > hg
annotate mercurial/revlogutils/sidedata.py @ 51310:f0e7d51bb454 stable
pycompat: fix bytestr(bytes) in Python 3.11
In Python 3.10, the `bytes` type itself does not have a `__bytes__`
attribute, but it does in 3.11. Yet `bytes(bytes)` does not give
the wished output, so we have to add an exceptional case.
The added case in the doctest reproduces the problem with Python 3.11.
Impact: error treatment in expressions such as `repo[b'invalid']` gets
broken.
author | Georges Racinet <georges.racinet@octobus.net> |
---|---|
date | Wed, 03 Jan 2024 18:33:39 +0100 |
parents | 6000f5b25c9b |
children |
rev | line source |
---|---|
43033
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
1 # sidedata.py - Logic around store extra data alongside revlog revisions |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
2 # |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
3 # Copyright 2019 Pierre-Yves David <pierre-yves.david@octobus.net) |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
4 # |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
5 # This software may be used and distributed according to the terms of the |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
6 # GNU General Public License version 2 or any later version. |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
7 """core code for "sidedata" support |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
8 |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
9 The "sidedata" are stored alongside the revision without actually being part of |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
10 its content and not affecting its hash. It's main use cases is to cache |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
11 important information related to a changesets. |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
12 |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
13 The current implementation is experimental and subject to changes. Do not rely |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
14 on it in production. |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
15 |
46709
3d740058b467
sidedata: move to new sidedata storage in revlogv2
Raphaël Gomès <rgomes@octobus.net>
parents:
46195
diff
changeset
|
16 Sidedata are stored in the revlog itself, thanks to a new version of the |
3d740058b467
sidedata: move to new sidedata storage in revlogv2
Raphaël Gomès <rgomes@octobus.net>
parents:
46195
diff
changeset
|
17 revlog. The following format is currently used:: |
43033
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
18 |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
19 initial header: |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
20 <number of sidedata; 2 bytes> |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
21 sidedata (repeated N times): |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
22 <sidedata-key; 2 bytes> |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
23 <sidedata-entry-length: 4 bytes> |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
24 <sidedata-content-sha1-digest: 20 bytes> |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
25 <sidedata-content; X bytes> |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
26 normal raw text: |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
27 <all bytes remaining in the rawtext> |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
28 |
46195 | 29 This is a simple and effective format. It should be enough to experiment with |
43033
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
30 the concept. |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
31 """ |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
32 |
43034
294afb982a88
sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43033
diff
changeset
|
33 |
47085
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
34 import collections |
43034
294afb982a88
sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43033
diff
changeset
|
35 import struct |
294afb982a88
sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43033
diff
changeset
|
36 |
47085
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
37 from .. import error, requirements as requirementsmod |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
38 from ..revlogutils import constants, flagutil |
44060
a61287a95dc3
core: migrate uses of hashlib.sha1 to hashutil.sha1
Augie Fackler <augie@google.com>
parents:
43506
diff
changeset
|
39 from ..utils import hashutil |
43034
294afb982a88
sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43033
diff
changeset
|
40 |
43040
ba4072c0a911
sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43037
diff
changeset
|
41 ## sidedata type constant |
ba4072c0a911
sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43037
diff
changeset
|
42 # reserve a block for testing purposes. |
ba4072c0a911
sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43037
diff
changeset
|
43 SD_TEST1 = 1 |
ba4072c0a911
sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43037
diff
changeset
|
44 SD_TEST2 = 2 |
ba4072c0a911
sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43037
diff
changeset
|
45 SD_TEST3 = 3 |
ba4072c0a911
sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43037
diff
changeset
|
46 SD_TEST4 = 4 |
ba4072c0a911
sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43037
diff
changeset
|
47 SD_TEST5 = 5 |
ba4072c0a911
sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43037
diff
changeset
|
48 SD_TEST6 = 6 |
ba4072c0a911
sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43037
diff
changeset
|
49 SD_TEST7 = 7 |
ba4072c0a911
sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43037
diff
changeset
|
50 |
43142
beed7ce61681
sidedatacopies: write copies information in sidedata when applicable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43077
diff
changeset
|
51 # key to store copies related information |
beed7ce61681
sidedatacopies: write copies information in sidedata when applicable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43077
diff
changeset
|
52 SD_P1COPIES = 8 |
beed7ce61681
sidedatacopies: write copies information in sidedata when applicable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43077
diff
changeset
|
53 SD_P2COPIES = 9 |
beed7ce61681
sidedatacopies: write copies information in sidedata when applicable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43077
diff
changeset
|
54 SD_FILESADDED = 10 |
beed7ce61681
sidedatacopies: write copies information in sidedata when applicable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43077
diff
changeset
|
55 SD_FILESREMOVED = 11 |
45634
9a6b409b8ebc
changing-files: rework the way we store changed files in side-data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44060
diff
changeset
|
56 SD_FILES = 12 |
43142
beed7ce61681
sidedatacopies: write copies information in sidedata when applicable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43077
diff
changeset
|
57 |
43040
ba4072c0a911
sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43037
diff
changeset
|
58 # internal format constant |
43506
9f70512ae2cf
cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents:
43142
diff
changeset
|
59 SIDEDATA_HEADER = struct.Struct('>H') |
9f70512ae2cf
cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents:
43142
diff
changeset
|
60 SIDEDATA_ENTRY = struct.Struct('>HL20s') |
43034
294afb982a88
sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43033
diff
changeset
|
61 |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
43048
diff
changeset
|
62 |
46709
3d740058b467
sidedata: move to new sidedata storage in revlogv2
Raphaël Gomès <rgomes@octobus.net>
parents:
46195
diff
changeset
|
63 def serialize_sidedata(sidedata): |
43035
ea83abf95630
sidedata: add a function to write sidedata into a raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43034
diff
changeset
|
64 sidedata = list(sidedata.items()) |
ea83abf95630
sidedata: add a function to write sidedata into a raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43034
diff
changeset
|
65 sidedata.sort() |
46709
3d740058b467
sidedata: move to new sidedata storage in revlogv2
Raphaël Gomès <rgomes@octobus.net>
parents:
46195
diff
changeset
|
66 buf = [SIDEDATA_HEADER.pack(len(sidedata))] |
43035
ea83abf95630
sidedata: add a function to write sidedata into a raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43034
diff
changeset
|
67 for key, value in sidedata: |
44060
a61287a95dc3
core: migrate uses of hashlib.sha1 to hashutil.sha1
Augie Fackler <augie@google.com>
parents:
43506
diff
changeset
|
68 digest = hashutil.sha1(value).digest() |
46709
3d740058b467
sidedata: move to new sidedata storage in revlogv2
Raphaël Gomès <rgomes@octobus.net>
parents:
46195
diff
changeset
|
69 buf.append(SIDEDATA_ENTRY.pack(key, len(value), digest)) |
43035
ea83abf95630
sidedata: add a function to write sidedata into a raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43034
diff
changeset
|
70 for key, value in sidedata: |
46709
3d740058b467
sidedata: move to new sidedata storage in revlogv2
Raphaël Gomès <rgomes@octobus.net>
parents:
46195
diff
changeset
|
71 buf.append(value) |
3d740058b467
sidedata: move to new sidedata storage in revlogv2
Raphaël Gomès <rgomes@octobus.net>
parents:
46195
diff
changeset
|
72 buf = b''.join(buf) |
3d740058b467
sidedata: move to new sidedata storage in revlogv2
Raphaël Gomès <rgomes@octobus.net>
parents:
46195
diff
changeset
|
73 return buf |
43035
ea83abf95630
sidedata: add a function to write sidedata into a raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43034
diff
changeset
|
74 |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
43048
diff
changeset
|
75 |
46709
3d740058b467
sidedata: move to new sidedata storage in revlogv2
Raphaël Gomès <rgomes@octobus.net>
parents:
46195
diff
changeset
|
76 def deserialize_sidedata(blob): |
43034
294afb982a88
sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43033
diff
changeset
|
77 sidedata = {} |
294afb982a88
sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43033
diff
changeset
|
78 offset = 0 |
46709
3d740058b467
sidedata: move to new sidedata storage in revlogv2
Raphaël Gomès <rgomes@octobus.net>
parents:
46195
diff
changeset
|
79 (nbentry,) = SIDEDATA_HEADER.unpack(blob[: SIDEDATA_HEADER.size]) |
43034
294afb982a88
sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43033
diff
changeset
|
80 offset += SIDEDATA_HEADER.size |
294afb982a88
sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43033
diff
changeset
|
81 dataoffset = SIDEDATA_HEADER.size + (SIDEDATA_ENTRY.size * nbentry) |
294afb982a88
sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43033
diff
changeset
|
82 for i in range(nbentry): |
294afb982a88
sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43033
diff
changeset
|
83 nextoffset = offset + SIDEDATA_ENTRY.size |
46709
3d740058b467
sidedata: move to new sidedata storage in revlogv2
Raphaël Gomès <rgomes@octobus.net>
parents:
46195
diff
changeset
|
84 key, size, storeddigest = SIDEDATA_ENTRY.unpack(blob[offset:nextoffset]) |
43034
294afb982a88
sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43033
diff
changeset
|
85 offset = nextoffset |
294afb982a88
sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43033
diff
changeset
|
86 # read the data associated with that entry |
294afb982a88
sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43033
diff
changeset
|
87 nextdataoffset = dataoffset + size |
46709
3d740058b467
sidedata: move to new sidedata storage in revlogv2
Raphaël Gomès <rgomes@octobus.net>
parents:
46195
diff
changeset
|
88 entrytext = bytes(blob[dataoffset:nextdataoffset]) |
44060
a61287a95dc3
core: migrate uses of hashlib.sha1 to hashutil.sha1
Augie Fackler <augie@google.com>
parents:
43506
diff
changeset
|
89 readdigest = hashutil.sha1(entrytext).digest() |
43034
294afb982a88
sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43033
diff
changeset
|
90 if storeddigest != readdigest: |
294afb982a88
sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43033
diff
changeset
|
91 raise error.SidedataHashError(key, storeddigest, readdigest) |
294afb982a88
sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43033
diff
changeset
|
92 sidedata[key] = entrytext |
294afb982a88
sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43033
diff
changeset
|
93 dataoffset = nextdataoffset |
46709
3d740058b467
sidedata: move to new sidedata storage in revlogv2
Raphaël Gomès <rgomes@octobus.net>
parents:
46195
diff
changeset
|
94 return sidedata |
47085
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
95 |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
96 |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
97 def get_sidedata_helpers(repo, remote_sd_categories, pull=False): |
47086
8bd769b5c941
sidedata: move documentation about sidedata helpers to sidedata module
Raphaël Gomès <rgomes@octobus.net>
parents:
47085
diff
changeset
|
98 """ |
8bd769b5c941
sidedata: move documentation about sidedata helpers to sidedata module
Raphaël Gomès <rgomes@octobus.net>
parents:
47085
diff
changeset
|
99 Returns a dictionary mapping revlog types to tuples of |
8bd769b5c941
sidedata: move documentation about sidedata helpers to sidedata module
Raphaël Gomès <rgomes@octobus.net>
parents:
47085
diff
changeset
|
100 `(repo, computers, removers)`: |
8bd769b5c941
sidedata: move documentation about sidedata helpers to sidedata module
Raphaël Gomès <rgomes@octobus.net>
parents:
47085
diff
changeset
|
101 * `repo` is used as an argument for computers |
8bd769b5c941
sidedata: move documentation about sidedata helpers to sidedata module
Raphaël Gomès <rgomes@octobus.net>
parents:
47085
diff
changeset
|
102 * `computers` is a list of `(category, (keys, computer, flags)` that |
8bd769b5c941
sidedata: move documentation about sidedata helpers to sidedata module
Raphaël Gomès <rgomes@octobus.net>
parents:
47085
diff
changeset
|
103 compute the missing sidedata categories that were asked: |
8bd769b5c941
sidedata: move documentation about sidedata helpers to sidedata module
Raphaël Gomès <rgomes@octobus.net>
parents:
47085
diff
changeset
|
104 * `category` is the sidedata category |
8bd769b5c941
sidedata: move documentation about sidedata helpers to sidedata module
Raphaël Gomès <rgomes@octobus.net>
parents:
47085
diff
changeset
|
105 * `keys` are the sidedata keys to be affected |
8bd769b5c941
sidedata: move documentation about sidedata helpers to sidedata module
Raphaël Gomès <rgomes@octobus.net>
parents:
47085
diff
changeset
|
106 * `flags` is a bitmask (an integer) of flags to remove when |
8bd769b5c941
sidedata: move documentation about sidedata helpers to sidedata module
Raphaël Gomès <rgomes@octobus.net>
parents:
47085
diff
changeset
|
107 removing the category. |
8bd769b5c941
sidedata: move documentation about sidedata helpers to sidedata module
Raphaël Gomès <rgomes@octobus.net>
parents:
47085
diff
changeset
|
108 * `computer` is the function `(repo, store, rev, sidedata)` that |
8bd769b5c941
sidedata: move documentation about sidedata helpers to sidedata module
Raphaël Gomès <rgomes@octobus.net>
parents:
47085
diff
changeset
|
109 returns a tuple of |
8bd769b5c941
sidedata: move documentation about sidedata helpers to sidedata module
Raphaël Gomès <rgomes@octobus.net>
parents:
47085
diff
changeset
|
110 `(new sidedata dict, (flags to add, flags to remove))`. |
8bd769b5c941
sidedata: move documentation about sidedata helpers to sidedata module
Raphaël Gomès <rgomes@octobus.net>
parents:
47085
diff
changeset
|
111 For example, it will return `({}, (0, 1 << 15))` to return no |
8bd769b5c941
sidedata: move documentation about sidedata helpers to sidedata module
Raphaël Gomès <rgomes@octobus.net>
parents:
47085
diff
changeset
|
112 sidedata, with no flags to add and one flag to remove. |
8bd769b5c941
sidedata: move documentation about sidedata helpers to sidedata module
Raphaël Gomès <rgomes@octobus.net>
parents:
47085
diff
changeset
|
113 * `removers` will remove the keys corresponding to the categories |
8bd769b5c941
sidedata: move documentation about sidedata helpers to sidedata module
Raphaël Gomès <rgomes@octobus.net>
parents:
47085
diff
changeset
|
114 that are present, but not needed. |
8bd769b5c941
sidedata: move documentation about sidedata helpers to sidedata module
Raphaël Gomès <rgomes@octobus.net>
parents:
47085
diff
changeset
|
115 If both `computers` and `removers` are empty, sidedata will simply not |
8bd769b5c941
sidedata: move documentation about sidedata helpers to sidedata module
Raphaël Gomès <rgomes@octobus.net>
parents:
47085
diff
changeset
|
116 be transformed. |
8bd769b5c941
sidedata: move documentation about sidedata helpers to sidedata module
Raphaël Gomès <rgomes@octobus.net>
parents:
47085
diff
changeset
|
117 """ |
47085
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
118 # Computers for computing sidedata on-the-fly |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
119 sd_computers = collections.defaultdict(list) |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
120 # Computers for categories to remove from sidedata |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
121 sd_removers = collections.defaultdict(list) |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
122 to_generate = remote_sd_categories - repo._wanted_sidedata |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
123 to_remove = repo._wanted_sidedata - remote_sd_categories |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
124 if pull: |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
125 to_generate, to_remove = to_remove, to_generate |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
126 |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
127 for revlog_kind, computers in repo._sidedata_computers.items(): |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
128 for category, computer in computers.items(): |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
129 if category in to_generate: |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
130 sd_computers[revlog_kind].append(computer) |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
131 if category in to_remove: |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
132 sd_removers[revlog_kind].append(computer) |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
133 |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
134 sidedata_helpers = (repo, sd_computers, sd_removers) |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
135 return sidedata_helpers |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
136 |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
137 |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
138 def run_sidedata_helpers(store, sidedata_helpers, sidedata, rev): |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
139 """Returns the sidedata for the given revision after running through |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
140 the given helpers. |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
141 - `store`: the revlog this applies to (changelog, manifest, or filelog |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
142 instance) |
47086
8bd769b5c941
sidedata: move documentation about sidedata helpers to sidedata module
Raphaël Gomès <rgomes@octobus.net>
parents:
47085
diff
changeset
|
143 - `sidedata_helpers`: see `get_sidedata_helpers` |
47085
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
144 - `sidedata`: previous sidedata at the given rev, if any |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
145 - `rev`: affected rev of `store` |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
146 """ |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
147 repo, sd_computers, sd_removers = sidedata_helpers |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
148 kind = store.revlog_kind |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
149 flags_to_add = 0 |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
150 flags_to_remove = 0 |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
151 for _keys, sd_computer, _flags in sd_computers.get(kind, []): |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
152 sidedata, flags = sd_computer(repo, store, rev, sidedata) |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
153 flags_to_add |= flags[0] |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
154 flags_to_remove |= flags[1] |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
155 for keys, _computer, flags in sd_removers.get(kind, []): |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
156 for key in keys: |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
157 sidedata.pop(key, None) |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
158 flags_to_remove |= flags |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
159 return sidedata, (flags_to_add, flags_to_remove) |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
160 |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
161 |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
162 def set_sidedata_spec_for_repo(repo): |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
163 # prevent cycle metadata -> revlogutils.sidedata -> metadata |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
164 from .. import metadata |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
165 |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
166 if requirementsmod.COPIESSDC_REQUIREMENT in repo.requirements: |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
167 repo.register_wanted_sidedata(SD_FILES) |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
168 repo.register_sidedata_computer( |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
169 constants.KIND_CHANGELOG, |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
170 SD_FILES, |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
171 (SD_FILES,), |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
172 metadata.copies_sidedata_computer, |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
173 flagutil.REVIDX_HASCOPIESINFO, |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
174 ) |