author | Matt Harbison <matt_harbison@yahoo.com> |
Thu, 11 Jul 2024 20:54:06 -0400 | |
changeset 51685 | 0eb515c7bec8 |
parent 48875 | 6000f5b25c9b |
child 51863 | f4733654f144 |
permissions | -rw-r--r-- |
43033
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
1 |
# sidedata.py - Logic around store extra data alongside revlog revisions |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
2 |
# |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
3 |
# Copyright 2019 Pierre-Yves David <pierre-yves.david@octobus.net) |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
4 |
# |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
5 |
# This software may be used and distributed according to the terms of the |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
6 |
# GNU General Public License version 2 or any later version. |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
7 |
"""core code for "sidedata" support |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
8 |
|
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
9 |
The "sidedata" are stored alongside the revision without actually being part of |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
10 |
its content and not affecting its hash. It's main use cases is to cache |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
11 |
important information related to a changesets. |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
12 |
|
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
13 |
The current implementation is experimental and subject to changes. Do not rely |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
14 |
on it in production. |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
15 |
|
46709
3d740058b467
sidedata: move to new sidedata storage in revlogv2
Raphaël Gomès <rgomes@octobus.net>
parents:
46195
diff
changeset
|
16 |
Sidedata are stored in the revlog itself, thanks to a new version of the |
3d740058b467
sidedata: move to new sidedata storage in revlogv2
Raphaël Gomès <rgomes@octobus.net>
parents:
46195
diff
changeset
|
17 |
revlog. The following format is currently used:: |
43033
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
18 |
|
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
19 |
initial header: |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
20 |
<number of sidedata; 2 bytes> |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
21 |
sidedata (repeated N times): |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
22 |
<sidedata-key; 2 bytes> |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
23 |
<sidedata-entry-length: 4 bytes> |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
24 |
<sidedata-content-sha1-digest: 20 bytes> |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
25 |
<sidedata-content; X bytes> |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
26 |
normal raw text: |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
27 |
<all bytes remaining in the rawtext> |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
28 |
|
46195 | 29 |
This is a simple and effective format. It should be enough to experiment with |
43033
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
30 |
the concept. |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
31 |
""" |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
32 |
|
43034
294afb982a88
sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43033
diff
changeset
|
33 |
|
47085
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
34 |
import collections |
43034
294afb982a88
sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43033
diff
changeset
|
35 |
import struct |
294afb982a88
sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43033
diff
changeset
|
36 |
|
47085
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
37 |
from .. import error, requirements as requirementsmod |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
38 |
from ..revlogutils import constants, flagutil |
44060
a61287a95dc3
core: migrate uses of hashlib.sha1 to hashutil.sha1
Augie Fackler <augie@google.com>
parents:
43506
diff
changeset
|
39 |
from ..utils import hashutil |
43034
294afb982a88
sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43033
diff
changeset
|
40 |
|
43040
ba4072c0a911
sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43037
diff
changeset
|
41 |
## sidedata type constant |
ba4072c0a911
sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43037
diff
changeset
|
42 |
# reserve a block for testing purposes. |
ba4072c0a911
sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43037
diff
changeset
|
43 |
SD_TEST1 = 1 |
ba4072c0a911
sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43037
diff
changeset
|
44 |
SD_TEST2 = 2 |
ba4072c0a911
sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43037
diff
changeset
|
45 |
SD_TEST3 = 3 |
ba4072c0a911
sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43037
diff
changeset
|
46 |
SD_TEST4 = 4 |
ba4072c0a911
sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43037
diff
changeset
|
47 |
SD_TEST5 = 5 |
ba4072c0a911
sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43037
diff
changeset
|
48 |
SD_TEST6 = 6 |
ba4072c0a911
sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43037
diff
changeset
|
49 |
SD_TEST7 = 7 |
ba4072c0a911
sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43037
diff
changeset
|
50 |
|
43142
beed7ce61681
sidedatacopies: write copies information in sidedata when applicable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43077
diff
changeset
|
51 |
# key to store copies related information |
beed7ce61681
sidedatacopies: write copies information in sidedata when applicable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43077
diff
changeset
|
52 |
SD_P1COPIES = 8 |
beed7ce61681
sidedatacopies: write copies information in sidedata when applicable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43077
diff
changeset
|
53 |
SD_P2COPIES = 9 |
beed7ce61681
sidedatacopies: write copies information in sidedata when applicable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43077
diff
changeset
|
54 |
SD_FILESADDED = 10 |
beed7ce61681
sidedatacopies: write copies information in sidedata when applicable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43077
diff
changeset
|
55 |
SD_FILESREMOVED = 11 |
45634
9a6b409b8ebc
changing-files: rework the way we store changed files in side-data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44060
diff
changeset
|
56 |
SD_FILES = 12 |
43142
beed7ce61681
sidedatacopies: write copies information in sidedata when applicable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43077
diff
changeset
|
57 |
|
43040
ba4072c0a911
sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43037
diff
changeset
|
58 |
# internal format constant |
43506
9f70512ae2cf
cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents:
43142
diff
changeset
|
59 |
SIDEDATA_HEADER = struct.Struct('>H') |
9f70512ae2cf
cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents:
43142
diff
changeset
|
60 |
SIDEDATA_ENTRY = struct.Struct('>HL20s') |
43034
294afb982a88
sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43033
diff
changeset
|
61 |
|
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
43048
diff
changeset
|
62 |
|
46709
3d740058b467
sidedata: move to new sidedata storage in revlogv2
Raphaël Gomès <rgomes@octobus.net>
parents:
46195
diff
changeset
|
63 |
def serialize_sidedata(sidedata): |
43035
ea83abf95630
sidedata: add a function to write sidedata into a raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43034
diff
changeset
|
64 |
sidedata = list(sidedata.items()) |
ea83abf95630
sidedata: add a function to write sidedata into a raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43034
diff
changeset
|
65 |
sidedata.sort() |
46709
3d740058b467
sidedata: move to new sidedata storage in revlogv2
Raphaël Gomès <rgomes@octobus.net>
parents:
46195
diff
changeset
|
66 |
buf = [SIDEDATA_HEADER.pack(len(sidedata))] |
43035
ea83abf95630
sidedata: add a function to write sidedata into a raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43034
diff
changeset
|
67 |
for key, value in sidedata: |
44060
a61287a95dc3
core: migrate uses of hashlib.sha1 to hashutil.sha1
Augie Fackler <augie@google.com>
parents:
43506
diff
changeset
|
68 |
digest = hashutil.sha1(value).digest() |
46709
3d740058b467
sidedata: move to new sidedata storage in revlogv2
Raphaël Gomès <rgomes@octobus.net>
parents:
46195
diff
changeset
|
69 |
buf.append(SIDEDATA_ENTRY.pack(key, len(value), digest)) |
43035
ea83abf95630
sidedata: add a function to write sidedata into a raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43034
diff
changeset
|
70 |
for key, value in sidedata: |
46709
3d740058b467
sidedata: move to new sidedata storage in revlogv2
Raphaël Gomès <rgomes@octobus.net>
parents:
46195
diff
changeset
|
71 |
buf.append(value) |
3d740058b467
sidedata: move to new sidedata storage in revlogv2
Raphaël Gomès <rgomes@octobus.net>
parents:
46195
diff
changeset
|
72 |
buf = b''.join(buf) |
3d740058b467
sidedata: move to new sidedata storage in revlogv2
Raphaël Gomès <rgomes@octobus.net>
parents:
46195
diff
changeset
|
73 |
return buf |
43035
ea83abf95630
sidedata: add a function to write sidedata into a raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43034
diff
changeset
|
74 |
|
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
43048
diff
changeset
|
75 |
|
46709
3d740058b467
sidedata: move to new sidedata storage in revlogv2
Raphaël Gomès <rgomes@octobus.net>
parents:
46195
diff
changeset
|
76 |
def deserialize_sidedata(blob): |
43034
294afb982a88
sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43033
diff
changeset
|
77 |
sidedata = {} |
294afb982a88
sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43033
diff
changeset
|
78 |
offset = 0 |
46709
3d740058b467
sidedata: move to new sidedata storage in revlogv2
Raphaël Gomès <rgomes@octobus.net>
parents:
46195
diff
changeset
|
79 |
(nbentry,) = SIDEDATA_HEADER.unpack(blob[: SIDEDATA_HEADER.size]) |
43034
294afb982a88
sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43033
diff
changeset
|
80 |
offset += SIDEDATA_HEADER.size |
294afb982a88
sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43033
diff
changeset
|
81 |
dataoffset = SIDEDATA_HEADER.size + (SIDEDATA_ENTRY.size * nbentry) |
294afb982a88
sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43033
diff
changeset
|
82 |
for i in range(nbentry): |
294afb982a88
sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43033
diff
changeset
|
83 |
nextoffset = offset + SIDEDATA_ENTRY.size |
46709
3d740058b467
sidedata: move to new sidedata storage in revlogv2
Raphaël Gomès <rgomes@octobus.net>
parents:
46195
diff
changeset
|
84 |
key, size, storeddigest = SIDEDATA_ENTRY.unpack(blob[offset:nextoffset]) |
43034
294afb982a88
sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43033
diff
changeset
|
85 |
offset = nextoffset |
294afb982a88
sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43033
diff
changeset
|
86 |
# read the data associated with that entry |
294afb982a88
sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43033
diff
changeset
|
87 |
nextdataoffset = dataoffset + size |
46709
3d740058b467
sidedata: move to new sidedata storage in revlogv2
Raphaël Gomès <rgomes@octobus.net>
parents:
46195
diff
changeset
|
88 |
entrytext = bytes(blob[dataoffset:nextdataoffset]) |
44060
a61287a95dc3
core: migrate uses of hashlib.sha1 to hashutil.sha1
Augie Fackler <augie@google.com>
parents:
43506
diff
changeset
|
89 |
readdigest = hashutil.sha1(entrytext).digest() |
43034
294afb982a88
sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43033
diff
changeset
|
90 |
if storeddigest != readdigest: |
294afb982a88
sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43033
diff
changeset
|
91 |
raise error.SidedataHashError(key, storeddigest, readdigest) |
294afb982a88
sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43033
diff
changeset
|
92 |
sidedata[key] = entrytext |
294afb982a88
sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43033
diff
changeset
|
93 |
dataoffset = nextdataoffset |
46709
3d740058b467
sidedata: move to new sidedata storage in revlogv2
Raphaël Gomès <rgomes@octobus.net>
parents:
46195
diff
changeset
|
94 |
return sidedata |
47085
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
95 |
|
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
96 |
|
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
97 |
def get_sidedata_helpers(repo, remote_sd_categories, pull=False): |
47086
8bd769b5c941
sidedata: move documentation about sidedata helpers to sidedata module
Raphaël Gomès <rgomes@octobus.net>
parents:
47085
diff
changeset
|
98 |
""" |
8bd769b5c941
sidedata: move documentation about sidedata helpers to sidedata module
Raphaël Gomès <rgomes@octobus.net>
parents:
47085
diff
changeset
|
99 |
Returns a dictionary mapping revlog types to tuples of |
8bd769b5c941
sidedata: move documentation about sidedata helpers to sidedata module
Raphaël Gomès <rgomes@octobus.net>
parents:
47085
diff
changeset
|
100 |
`(repo, computers, removers)`: |
8bd769b5c941
sidedata: move documentation about sidedata helpers to sidedata module
Raphaël Gomès <rgomes@octobus.net>
parents:
47085
diff
changeset
|
101 |
* `repo` is used as an argument for computers |
8bd769b5c941
sidedata: move documentation about sidedata helpers to sidedata module
Raphaël Gomès <rgomes@octobus.net>
parents:
47085
diff
changeset
|
102 |
* `computers` is a list of `(category, (keys, computer, flags)` that |
8bd769b5c941
sidedata: move documentation about sidedata helpers to sidedata module
Raphaël Gomès <rgomes@octobus.net>
parents:
47085
diff
changeset
|
103 |
compute the missing sidedata categories that were asked: |
8bd769b5c941
sidedata: move documentation about sidedata helpers to sidedata module
Raphaël Gomès <rgomes@octobus.net>
parents:
47085
diff
changeset
|
104 |
* `category` is the sidedata category |
8bd769b5c941
sidedata: move documentation about sidedata helpers to sidedata module
Raphaël Gomès <rgomes@octobus.net>
parents:
47085
diff
changeset
|
105 |
* `keys` are the sidedata keys to be affected |
8bd769b5c941
sidedata: move documentation about sidedata helpers to sidedata module
Raphaël Gomès <rgomes@octobus.net>
parents:
47085
diff
changeset
|
106 |
* `flags` is a bitmask (an integer) of flags to remove when |
8bd769b5c941
sidedata: move documentation about sidedata helpers to sidedata module
Raphaël Gomès <rgomes@octobus.net>
parents:
47085
diff
changeset
|
107 |
removing the category. |
8bd769b5c941
sidedata: move documentation about sidedata helpers to sidedata module
Raphaël Gomès <rgomes@octobus.net>
parents:
47085
diff
changeset
|
108 |
* `computer` is the function `(repo, store, rev, sidedata)` that |
8bd769b5c941
sidedata: move documentation about sidedata helpers to sidedata module
Raphaël Gomès <rgomes@octobus.net>
parents:
47085
diff
changeset
|
109 |
returns a tuple of |
8bd769b5c941
sidedata: move documentation about sidedata helpers to sidedata module
Raphaël Gomès <rgomes@octobus.net>
parents:
47085
diff
changeset
|
110 |
`(new sidedata dict, (flags to add, flags to remove))`. |
8bd769b5c941
sidedata: move documentation about sidedata helpers to sidedata module
Raphaël Gomès <rgomes@octobus.net>
parents:
47085
diff
changeset
|
111 |
For example, it will return `({}, (0, 1 << 15))` to return no |
8bd769b5c941
sidedata: move documentation about sidedata helpers to sidedata module
Raphaël Gomès <rgomes@octobus.net>
parents:
47085
diff
changeset
|
112 |
sidedata, with no flags to add and one flag to remove. |
8bd769b5c941
sidedata: move documentation about sidedata helpers to sidedata module
Raphaël Gomès <rgomes@octobus.net>
parents:
47085
diff
changeset
|
113 |
* `removers` will remove the keys corresponding to the categories |
8bd769b5c941
sidedata: move documentation about sidedata helpers to sidedata module
Raphaël Gomès <rgomes@octobus.net>
parents:
47085
diff
changeset
|
114 |
that are present, but not needed. |
8bd769b5c941
sidedata: move documentation about sidedata helpers to sidedata module
Raphaël Gomès <rgomes@octobus.net>
parents:
47085
diff
changeset
|
115 |
If both `computers` and `removers` are empty, sidedata will simply not |
8bd769b5c941
sidedata: move documentation about sidedata helpers to sidedata module
Raphaël Gomès <rgomes@octobus.net>
parents:
47085
diff
changeset
|
116 |
be transformed. |
8bd769b5c941
sidedata: move documentation about sidedata helpers to sidedata module
Raphaël Gomès <rgomes@octobus.net>
parents:
47085
diff
changeset
|
117 |
""" |
47085
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
118 |
# Computers for computing sidedata on-the-fly |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
119 |
sd_computers = collections.defaultdict(list) |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
120 |
# Computers for categories to remove from sidedata |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
121 |
sd_removers = collections.defaultdict(list) |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
122 |
to_generate = remote_sd_categories - repo._wanted_sidedata |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
123 |
to_remove = repo._wanted_sidedata - remote_sd_categories |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
124 |
if pull: |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
125 |
to_generate, to_remove = to_remove, to_generate |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
126 |
|
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
127 |
for revlog_kind, computers in repo._sidedata_computers.items(): |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
128 |
for category, computer in computers.items(): |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
129 |
if category in to_generate: |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
130 |
sd_computers[revlog_kind].append(computer) |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
131 |
if category in to_remove: |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
132 |
sd_removers[revlog_kind].append(computer) |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
133 |
|
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
134 |
sidedata_helpers = (repo, sd_computers, sd_removers) |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
135 |
return sidedata_helpers |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
136 |
|
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
137 |
|
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
138 |
def run_sidedata_helpers(store, sidedata_helpers, sidedata, rev): |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
139 |
"""Returns the sidedata for the given revision after running through |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
140 |
the given helpers. |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
141 |
- `store`: the revlog this applies to (changelog, manifest, or filelog |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
142 |
instance) |
47086
8bd769b5c941
sidedata: move documentation about sidedata helpers to sidedata module
Raphaël Gomès <rgomes@octobus.net>
parents:
47085
diff
changeset
|
143 |
- `sidedata_helpers`: see `get_sidedata_helpers` |
47085
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
144 |
- `sidedata`: previous sidedata at the given rev, if any |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
145 |
- `rev`: affected rev of `store` |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
146 |
""" |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
147 |
repo, sd_computers, sd_removers = sidedata_helpers |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
148 |
kind = store.revlog_kind |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
149 |
flags_to_add = 0 |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
150 |
flags_to_remove = 0 |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
151 |
for _keys, sd_computer, _flags in sd_computers.get(kind, []): |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
152 |
sidedata, flags = sd_computer(repo, store, rev, sidedata) |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
153 |
flags_to_add |= flags[0] |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
154 |
flags_to_remove |= flags[1] |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
155 |
for keys, _computer, flags in sd_removers.get(kind, []): |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
156 |
for key in keys: |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
157 |
sidedata.pop(key, None) |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
158 |
flags_to_remove |= flags |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
159 |
return sidedata, (flags_to_add, flags_to_remove) |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
160 |
|
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
161 |
|
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
162 |
def set_sidedata_spec_for_repo(repo): |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
163 |
# prevent cycle metadata -> revlogutils.sidedata -> metadata |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
164 |
from .. import metadata |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
165 |
|
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
166 |
if requirementsmod.COPIESSDC_REQUIREMENT in repo.requirements: |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
167 |
repo.register_wanted_sidedata(SD_FILES) |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
168 |
repo.register_sidedata_computer( |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
169 |
constants.KIND_CHANGELOG, |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
170 |
SD_FILES, |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
171 |
(SD_FILES,), |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
172 |
metadata.copies_sidedata_computer, |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
173 |
flagutil.REVIDX_HASCOPIESINFO, |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents:
46709
diff
changeset
|
174 |
) |