mercurial/revlogutils/sidedata.py
author Matt Harbison <matt_harbison@yahoo.com>
Thu, 11 Jul 2024 20:54:06 -0400
changeset 51685 0eb515c7bec8
parent 48875 6000f5b25c9b
child 51863 f4733654f144
permissions -rw-r--r--
typing: add trivial type hints to the convert extension's common modules This started as ensuring that the `encoding` and `orig_encoding` attributes has a type other than `Any`, so pytype can catch problems where it needs to be str for stdlib encoding and decoding. It turns out that adding the hint in `mercurial.encoding` is what was needed, but I picked a bunch of low hanging fruit while here. There's definitely more to do, and I see a problem where `shlex.shlex` is being fed bytes instead of str, but there are not enough type hints yet to make pytype notice.
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
43033
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
     1
# sidedata.py - Logic around store extra data alongside revlog revisions
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
     2
#
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
     3
# Copyright 2019 Pierre-Yves David <pierre-yves.david@octobus.net)
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
     4
#
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
     5
# This software may be used and distributed according to the terms of the
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
     6
# GNU General Public License version 2 or any later version.
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
     7
"""core code for "sidedata" support
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
     8
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
     9
The "sidedata" are stored alongside the revision without actually being part of
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
    10
its content and not affecting its hash. It's main use cases is to cache
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
    11
important information related to a changesets.
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
    12
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
    13
The current implementation is experimental and subject to changes. Do not rely
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
    14
on it in production.
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
    15
46709
3d740058b467 sidedata: move to new sidedata storage in revlogv2
Raphaël Gomès <rgomes@octobus.net>
parents: 46195
diff changeset
    16
Sidedata are stored in the revlog itself, thanks to a new version of the
3d740058b467 sidedata: move to new sidedata storage in revlogv2
Raphaël Gomès <rgomes@octobus.net>
parents: 46195
diff changeset
    17
revlog. The following format is currently used::
43033
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
    18
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
    19
    initial header:
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
    20
        <number of sidedata; 2 bytes>
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
    21
    sidedata (repeated N times):
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
    22
        <sidedata-key; 2 bytes>
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
    23
        <sidedata-entry-length: 4 bytes>
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
    24
        <sidedata-content-sha1-digest: 20 bytes>
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
    25
        <sidedata-content; X bytes>
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
    26
    normal raw text:
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
    27
        <all bytes remaining in the rawtext>
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
    28
46195
d6a9e690d620 comments: fix typos
Joerg Sonnenberger <joerg@bec.de>
parents: 45634
diff changeset
    29
This is a simple and effective format. It should be enough to experiment with
43033
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
    30
the concept.
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
    31
"""
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
    32
43034
294afb982a88 sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43033
diff changeset
    33
47085
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents: 46709
diff changeset
    34
import collections
43034
294afb982a88 sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43033
diff changeset
    35
import struct
294afb982a88 sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43033
diff changeset
    36
47085
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents: 46709
diff changeset
    37
from .. import error, requirements as requirementsmod
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents: 46709
diff changeset
    38
from ..revlogutils import constants, flagutil
44060
a61287a95dc3 core: migrate uses of hashlib.sha1 to hashutil.sha1
Augie Fackler <augie@google.com>
parents: 43506
diff changeset
    39
from ..utils import hashutil
43034
294afb982a88 sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43033
diff changeset
    40
43040
ba4072c0a911 sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43037
diff changeset
    41
## sidedata type constant
ba4072c0a911 sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43037
diff changeset
    42
# reserve a block for testing purposes.
ba4072c0a911 sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43037
diff changeset
    43
SD_TEST1 = 1
ba4072c0a911 sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43037
diff changeset
    44
SD_TEST2 = 2
ba4072c0a911 sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43037
diff changeset
    45
SD_TEST3 = 3
ba4072c0a911 sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43037
diff changeset
    46
SD_TEST4 = 4
ba4072c0a911 sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43037
diff changeset
    47
SD_TEST5 = 5
ba4072c0a911 sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43037
diff changeset
    48
SD_TEST6 = 6
ba4072c0a911 sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43037
diff changeset
    49
SD_TEST7 = 7
ba4072c0a911 sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43037
diff changeset
    50
43142
beed7ce61681 sidedatacopies: write copies information in sidedata when applicable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43077
diff changeset
    51
# key to store copies related information
beed7ce61681 sidedatacopies: write copies information in sidedata when applicable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43077
diff changeset
    52
SD_P1COPIES = 8
beed7ce61681 sidedatacopies: write copies information in sidedata when applicable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43077
diff changeset
    53
SD_P2COPIES = 9
beed7ce61681 sidedatacopies: write copies information in sidedata when applicable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43077
diff changeset
    54
SD_FILESADDED = 10
beed7ce61681 sidedatacopies: write copies information in sidedata when applicable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43077
diff changeset
    55
SD_FILESREMOVED = 11
45634
9a6b409b8ebc changing-files: rework the way we store changed files in side-data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44060
diff changeset
    56
SD_FILES = 12
43142
beed7ce61681 sidedatacopies: write copies information in sidedata when applicable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43077
diff changeset
    57
43040
ba4072c0a911 sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43037
diff changeset
    58
# internal format constant
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43142
diff changeset
    59
SIDEDATA_HEADER = struct.Struct('>H')
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43142
diff changeset
    60
SIDEDATA_ENTRY = struct.Struct('>HL20s')
43034
294afb982a88 sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43033
diff changeset
    61
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43048
diff changeset
    62
46709
3d740058b467 sidedata: move to new sidedata storage in revlogv2
Raphaël Gomès <rgomes@octobus.net>
parents: 46195
diff changeset
    63
def serialize_sidedata(sidedata):
43035
ea83abf95630 sidedata: add a function to write sidedata into a raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43034
diff changeset
    64
    sidedata = list(sidedata.items())
ea83abf95630 sidedata: add a function to write sidedata into a raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43034
diff changeset
    65
    sidedata.sort()
46709
3d740058b467 sidedata: move to new sidedata storage in revlogv2
Raphaël Gomès <rgomes@octobus.net>
parents: 46195
diff changeset
    66
    buf = [SIDEDATA_HEADER.pack(len(sidedata))]
43035
ea83abf95630 sidedata: add a function to write sidedata into a raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43034
diff changeset
    67
    for key, value in sidedata:
44060
a61287a95dc3 core: migrate uses of hashlib.sha1 to hashutil.sha1
Augie Fackler <augie@google.com>
parents: 43506
diff changeset
    68
        digest = hashutil.sha1(value).digest()
46709
3d740058b467 sidedata: move to new sidedata storage in revlogv2
Raphaël Gomès <rgomes@octobus.net>
parents: 46195
diff changeset
    69
        buf.append(SIDEDATA_ENTRY.pack(key, len(value), digest))
43035
ea83abf95630 sidedata: add a function to write sidedata into a raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43034
diff changeset
    70
    for key, value in sidedata:
46709
3d740058b467 sidedata: move to new sidedata storage in revlogv2
Raphaël Gomès <rgomes@octobus.net>
parents: 46195
diff changeset
    71
        buf.append(value)
3d740058b467 sidedata: move to new sidedata storage in revlogv2
Raphaël Gomès <rgomes@octobus.net>
parents: 46195
diff changeset
    72
    buf = b''.join(buf)
3d740058b467 sidedata: move to new sidedata storage in revlogv2
Raphaël Gomès <rgomes@octobus.net>
parents: 46195
diff changeset
    73
    return buf
43035
ea83abf95630 sidedata: add a function to write sidedata into a raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43034
diff changeset
    74
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43048
diff changeset
    75
46709
3d740058b467 sidedata: move to new sidedata storage in revlogv2
Raphaël Gomès <rgomes@octobus.net>
parents: 46195
diff changeset
    76
def deserialize_sidedata(blob):
43034
294afb982a88 sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43033
diff changeset
    77
    sidedata = {}
294afb982a88 sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43033
diff changeset
    78
    offset = 0
46709
3d740058b467 sidedata: move to new sidedata storage in revlogv2
Raphaël Gomès <rgomes@octobus.net>
parents: 46195
diff changeset
    79
    (nbentry,) = SIDEDATA_HEADER.unpack(blob[: SIDEDATA_HEADER.size])
43034
294afb982a88 sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43033
diff changeset
    80
    offset += SIDEDATA_HEADER.size
294afb982a88 sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43033
diff changeset
    81
    dataoffset = SIDEDATA_HEADER.size + (SIDEDATA_ENTRY.size * nbentry)
294afb982a88 sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43033
diff changeset
    82
    for i in range(nbentry):
294afb982a88 sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43033
diff changeset
    83
        nextoffset = offset + SIDEDATA_ENTRY.size
46709
3d740058b467 sidedata: move to new sidedata storage in revlogv2
Raphaël Gomès <rgomes@octobus.net>
parents: 46195
diff changeset
    84
        key, size, storeddigest = SIDEDATA_ENTRY.unpack(blob[offset:nextoffset])
43034
294afb982a88 sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43033
diff changeset
    85
        offset = nextoffset
294afb982a88 sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43033
diff changeset
    86
        # read the data associated with that entry
294afb982a88 sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43033
diff changeset
    87
        nextdataoffset = dataoffset + size
46709
3d740058b467 sidedata: move to new sidedata storage in revlogv2
Raphaël Gomès <rgomes@octobus.net>
parents: 46195
diff changeset
    88
        entrytext = bytes(blob[dataoffset:nextdataoffset])
44060
a61287a95dc3 core: migrate uses of hashlib.sha1 to hashutil.sha1
Augie Fackler <augie@google.com>
parents: 43506
diff changeset
    89
        readdigest = hashutil.sha1(entrytext).digest()
43034
294afb982a88 sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43033
diff changeset
    90
        if storeddigest != readdigest:
294afb982a88 sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43033
diff changeset
    91
            raise error.SidedataHashError(key, storeddigest, readdigest)
294afb982a88 sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43033
diff changeset
    92
        sidedata[key] = entrytext
294afb982a88 sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43033
diff changeset
    93
        dataoffset = nextdataoffset
46709
3d740058b467 sidedata: move to new sidedata storage in revlogv2
Raphaël Gomès <rgomes@octobus.net>
parents: 46195
diff changeset
    94
    return sidedata
47085
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents: 46709
diff changeset
    95
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents: 46709
diff changeset
    96
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents: 46709
diff changeset
    97
def get_sidedata_helpers(repo, remote_sd_categories, pull=False):
47086
8bd769b5c941 sidedata: move documentation about sidedata helpers to sidedata module
Raphaël Gomès <rgomes@octobus.net>
parents: 47085
diff changeset
    98
    """
8bd769b5c941 sidedata: move documentation about sidedata helpers to sidedata module
Raphaël Gomès <rgomes@octobus.net>
parents: 47085
diff changeset
    99
    Returns a dictionary mapping revlog types to tuples of
8bd769b5c941 sidedata: move documentation about sidedata helpers to sidedata module
Raphaël Gomès <rgomes@octobus.net>
parents: 47085
diff changeset
   100
    `(repo, computers, removers)`:
8bd769b5c941 sidedata: move documentation about sidedata helpers to sidedata module
Raphaël Gomès <rgomes@octobus.net>
parents: 47085
diff changeset
   101
        * `repo` is used as an argument for computers
8bd769b5c941 sidedata: move documentation about sidedata helpers to sidedata module
Raphaël Gomès <rgomes@octobus.net>
parents: 47085
diff changeset
   102
        * `computers` is a list of `(category, (keys, computer, flags)` that
8bd769b5c941 sidedata: move documentation about sidedata helpers to sidedata module
Raphaël Gomès <rgomes@octobus.net>
parents: 47085
diff changeset
   103
           compute the missing sidedata categories that were asked:
8bd769b5c941 sidedata: move documentation about sidedata helpers to sidedata module
Raphaël Gomès <rgomes@octobus.net>
parents: 47085
diff changeset
   104
           * `category` is the sidedata category
8bd769b5c941 sidedata: move documentation about sidedata helpers to sidedata module
Raphaël Gomès <rgomes@octobus.net>
parents: 47085
diff changeset
   105
           * `keys` are the sidedata keys to be affected
8bd769b5c941 sidedata: move documentation about sidedata helpers to sidedata module
Raphaël Gomès <rgomes@octobus.net>
parents: 47085
diff changeset
   106
           * `flags` is a bitmask (an integer) of flags to remove when
8bd769b5c941 sidedata: move documentation about sidedata helpers to sidedata module
Raphaël Gomès <rgomes@octobus.net>
parents: 47085
diff changeset
   107
              removing the category.
8bd769b5c941 sidedata: move documentation about sidedata helpers to sidedata module
Raphaël Gomès <rgomes@octobus.net>
parents: 47085
diff changeset
   108
           * `computer` is the function `(repo, store, rev, sidedata)` that
8bd769b5c941 sidedata: move documentation about sidedata helpers to sidedata module
Raphaël Gomès <rgomes@octobus.net>
parents: 47085
diff changeset
   109
             returns a tuple of
8bd769b5c941 sidedata: move documentation about sidedata helpers to sidedata module
Raphaël Gomès <rgomes@octobus.net>
parents: 47085
diff changeset
   110
             `(new sidedata dict, (flags to add, flags to remove))`.
8bd769b5c941 sidedata: move documentation about sidedata helpers to sidedata module
Raphaël Gomès <rgomes@octobus.net>
parents: 47085
diff changeset
   111
             For example, it will return `({}, (0, 1 << 15))` to return no
8bd769b5c941 sidedata: move documentation about sidedata helpers to sidedata module
Raphaël Gomès <rgomes@octobus.net>
parents: 47085
diff changeset
   112
             sidedata, with no flags to add and one flag to remove.
8bd769b5c941 sidedata: move documentation about sidedata helpers to sidedata module
Raphaël Gomès <rgomes@octobus.net>
parents: 47085
diff changeset
   113
        * `removers` will remove the keys corresponding to the categories
8bd769b5c941 sidedata: move documentation about sidedata helpers to sidedata module
Raphaël Gomès <rgomes@octobus.net>
parents: 47085
diff changeset
   114
          that are present, but not needed.
8bd769b5c941 sidedata: move documentation about sidedata helpers to sidedata module
Raphaël Gomès <rgomes@octobus.net>
parents: 47085
diff changeset
   115
        If both `computers` and `removers` are empty, sidedata will simply not
8bd769b5c941 sidedata: move documentation about sidedata helpers to sidedata module
Raphaël Gomès <rgomes@octobus.net>
parents: 47085
diff changeset
   116
        be transformed.
8bd769b5c941 sidedata: move documentation about sidedata helpers to sidedata module
Raphaël Gomès <rgomes@octobus.net>
parents: 47085
diff changeset
   117
    """
47085
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents: 46709
diff changeset
   118
    # Computers for computing sidedata on-the-fly
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents: 46709
diff changeset
   119
    sd_computers = collections.defaultdict(list)
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents: 46709
diff changeset
   120
    # Computers for categories to remove from sidedata
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents: 46709
diff changeset
   121
    sd_removers = collections.defaultdict(list)
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents: 46709
diff changeset
   122
    to_generate = remote_sd_categories - repo._wanted_sidedata
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents: 46709
diff changeset
   123
    to_remove = repo._wanted_sidedata - remote_sd_categories
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents: 46709
diff changeset
   124
    if pull:
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents: 46709
diff changeset
   125
        to_generate, to_remove = to_remove, to_generate
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents: 46709
diff changeset
   126
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents: 46709
diff changeset
   127
    for revlog_kind, computers in repo._sidedata_computers.items():
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents: 46709
diff changeset
   128
        for category, computer in computers.items():
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents: 46709
diff changeset
   129
            if category in to_generate:
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents: 46709
diff changeset
   130
                sd_computers[revlog_kind].append(computer)
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents: 46709
diff changeset
   131
            if category in to_remove:
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents: 46709
diff changeset
   132
                sd_removers[revlog_kind].append(computer)
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents: 46709
diff changeset
   133
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents: 46709
diff changeset
   134
    sidedata_helpers = (repo, sd_computers, sd_removers)
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents: 46709
diff changeset
   135
    return sidedata_helpers
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents: 46709
diff changeset
   136
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents: 46709
diff changeset
   137
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents: 46709
diff changeset
   138
def run_sidedata_helpers(store, sidedata_helpers, sidedata, rev):
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents: 46709
diff changeset
   139
    """Returns the sidedata for the given revision after running through
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents: 46709
diff changeset
   140
    the given helpers.
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents: 46709
diff changeset
   141
    - `store`: the revlog this applies to (changelog, manifest, or filelog
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents: 46709
diff changeset
   142
      instance)
47086
8bd769b5c941 sidedata: move documentation about sidedata helpers to sidedata module
Raphaël Gomès <rgomes@octobus.net>
parents: 47085
diff changeset
   143
    - `sidedata_helpers`: see `get_sidedata_helpers`
47085
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents: 46709
diff changeset
   144
    - `sidedata`: previous sidedata at the given rev, if any
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents: 46709
diff changeset
   145
    - `rev`: affected rev of `store`
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents: 46709
diff changeset
   146
    """
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents: 46709
diff changeset
   147
    repo, sd_computers, sd_removers = sidedata_helpers
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents: 46709
diff changeset
   148
    kind = store.revlog_kind
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents: 46709
diff changeset
   149
    flags_to_add = 0
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents: 46709
diff changeset
   150
    flags_to_remove = 0
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents: 46709
diff changeset
   151
    for _keys, sd_computer, _flags in sd_computers.get(kind, []):
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents: 46709
diff changeset
   152
        sidedata, flags = sd_computer(repo, store, rev, sidedata)
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents: 46709
diff changeset
   153
        flags_to_add |= flags[0]
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents: 46709
diff changeset
   154
        flags_to_remove |= flags[1]
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents: 46709
diff changeset
   155
    for keys, _computer, flags in sd_removers.get(kind, []):
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents: 46709
diff changeset
   156
        for key in keys:
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents: 46709
diff changeset
   157
            sidedata.pop(key, None)
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents: 46709
diff changeset
   158
        flags_to_remove |= flags
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents: 46709
diff changeset
   159
    return sidedata, (flags_to_add, flags_to_remove)
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents: 46709
diff changeset
   160
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents: 46709
diff changeset
   161
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents: 46709
diff changeset
   162
def set_sidedata_spec_for_repo(repo):
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents: 46709
diff changeset
   163
    # prevent cycle metadata -> revlogutils.sidedata -> metadata
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents: 46709
diff changeset
   164
    from .. import metadata
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents: 46709
diff changeset
   165
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents: 46709
diff changeset
   166
    if requirementsmod.COPIESSDC_REQUIREMENT in repo.requirements:
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents: 46709
diff changeset
   167
        repo.register_wanted_sidedata(SD_FILES)
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents: 46709
diff changeset
   168
    repo.register_sidedata_computer(
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents: 46709
diff changeset
   169
        constants.KIND_CHANGELOG,
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents: 46709
diff changeset
   170
        SD_FILES,
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents: 46709
diff changeset
   171
        (SD_FILES,),
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents: 46709
diff changeset
   172
        metadata.copies_sidedata_computer,
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents: 46709
diff changeset
   173
        flagutil.REVIDX_HASCOPIESINFO,
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Raphaël Gomès <rgomes@octobus.net>
parents: 46709
diff changeset
   174
    )