mercurial/revlogutils/nodemap.py
author Matt Harbison <matt_harbison@yahoo.com>
Thu, 11 Jul 2024 20:54:06 -0400
changeset 51685 0eb515c7bec8
parent 51656 21442f87f7fa
child 51690 493034cc3265
permissions -rw-r--r--
typing: add trivial type hints to the convert extension's common modules This started as ensuring that the `encoding` and `orig_encoding` attributes has a type other than `Any`, so pytype can catch problems where it needs to be str for stdlib encoding and decoding. It turns out that adding the hint in `mercurial.encoding` is what was needed, but I picked a bunch of low hanging fruit while here. There's definitely more to do, and I see a problem where `shlex.shlex` is being fed bytes instead of str, but there are not enough type hints yet to make pytype notice.
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
44034
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
     1
# nodemap.py - nodemap related code and utilities
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
     2
#
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
     3
# Copyright 2019 Pierre-Yves David <pierre-yves.david@octobus.net>
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
     4
# Copyright 2019 George Racinet <georges.racinet@octobus.net>
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
     5
#
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
     6
# This software may be used and distributed according to the terms of the
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
     7
# GNU General Public License version 2 or any later version.
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
     8
44307
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
     9
44312
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44311
diff changeset
    10
import re
44307
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
    11
import struct
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
    12
46113
59fa3890d40a node: import symbols explicitly
Joerg Sonnenberger <joerg@bec.de>
parents: 46089
diff changeset
    13
from ..node import hex
44788
6493f0a567c2 nodemap: add a new mode option, with an optional "warn" value
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44745
diff changeset
    14
44307
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
    15
from .. import (
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
    16
    error,
48693
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48690
diff changeset
    17
    requirements,
44314
7f4f7ef3133e nodemap: add a optional `nodemap_add_full` method on indexes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44312
diff changeset
    18
    util,
44307
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
    19
)
47312
7ea39d633cf3 docket: move the uid logic in the `revlogutils.docket` module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47311
diff changeset
    20
from . import docket as docket_mod
44034
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
    21
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
    22
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
    23
class NodeMap(dict):
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
    24
    def __missing__(self, x):
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
    25
        raise error.RevlogError(b'unknown node: %s' % x)
44307
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
    26
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
    27
48038
52018f8ef020 persistent-nodemap: introduce a test to highlight possible race
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47451
diff changeset
    28
def test_race_hook_1():
52018f8ef020 persistent-nodemap: introduce a test to highlight possible race
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47451
diff changeset
    29
    """hook point for test
52018f8ef020 persistent-nodemap: introduce a test to highlight possible race
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47451
diff changeset
    30
52018f8ef020 persistent-nodemap: introduce a test to highlight possible race
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47451
diff changeset
    31
    This let tests to have things happens between the docket reading and the
52018f8ef020 persistent-nodemap: introduce a test to highlight possible race
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47451
diff changeset
    32
    data reading"""
52018f8ef020 persistent-nodemap: introduce a test to highlight possible race
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47451
diff changeset
    33
    pass
52018f8ef020 persistent-nodemap: introduce a test to highlight possible race
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47451
diff changeset
    34
52018f8ef020 persistent-nodemap: introduce a test to highlight possible race
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47451
diff changeset
    35
48693
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48690
diff changeset
    36
def post_stream_cleanup(repo):
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48690
diff changeset
    37
    """The stream clone might needs to remove some file if persisten nodemap
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48690
diff changeset
    38
    was dropped while stream cloning
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48690
diff changeset
    39
    """
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48690
diff changeset
    40
    if requirements.REVLOGV1_REQUIREMENT not in repo.requirements:
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48690
diff changeset
    41
        return
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48690
diff changeset
    42
    if requirements.NODEMAP_REQUIREMENT in repo.requirements:
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48690
diff changeset
    43
        return
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48690
diff changeset
    44
    unfi = repo.unfiltered()
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48690
diff changeset
    45
    delete_nodemap(None, unfi, unfi.changelog)
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48690
diff changeset
    46
    delete_nodemap(None, repo, unfi.manifestlog._rootstore._revlog)
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48690
diff changeset
    47
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48690
diff changeset
    48
44309
6c07480d6659 nodemap: add a function to read the data from disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44308
diff changeset
    49
def persisted_data(revlog):
6c07480d6659 nodemap: add a function to read the data from disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44308
diff changeset
    50
    """read the nodemap for a revlog from disk"""
47164
04f2f94836a5 revlog: rename `nodemap_file` to `_nodemap_file`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47152
diff changeset
    51
    if revlog._nodemap_file is None:
44309
6c07480d6659 nodemap: add a function to read the data from disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44308
diff changeset
    52
        return None
47164
04f2f94836a5 revlog: rename `nodemap_file` to `_nodemap_file`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47152
diff changeset
    53
    pdata = revlog.opener.tryread(revlog._nodemap_file)
44311
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
    54
    if not pdata:
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
    55
        return None
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
    56
    offset = 0
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
    57
    (version,) = S_VERSION.unpack(pdata[offset : offset + S_VERSION.size])
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
    58
    if version != ONDISK_VERSION:
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
    59
        return None
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
    60
    offset += S_VERSION.size
44335
e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44333
diff changeset
    61
    headers = S_HEADER.unpack(pdata[offset : offset + S_HEADER.size])
44515
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44497
diff changeset
    62
    uid_size, tip_rev, data_length, data_unused, tip_node_size = headers
44311
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
    63
    offset += S_HEADER.size
44322
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44321
diff changeset
    64
    docket = NodeMapDocket(pdata[offset : offset + uid_size])
44515
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44497
diff changeset
    65
    offset += uid_size
44335
e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44333
diff changeset
    66
    docket.tip_rev = tip_rev
44515
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44497
diff changeset
    67
    docket.tip_node = pdata[offset : offset + tip_node_size]
44336
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44335
diff changeset
    68
    docket.data_length = data_length
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44335
diff changeset
    69
    docket.data_unused = data_unused
44311
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
    70
44322
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44321
diff changeset
    71
    filename = _rawdata_filepath(revlog, docket)
44792
5e3c718692bb nodemap: drop the 'exp-' prefix for internal opener option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44790
diff changeset
    72
    use_mmap = revlog.opener.options.get(b"persistent-nodemap.mmap")
48038
52018f8ef020 persistent-nodemap: introduce a test to highlight possible race
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47451
diff changeset
    73
52018f8ef020 persistent-nodemap: introduce a test to highlight possible race
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47451
diff changeset
    74
    test_race_hook_1()
44363
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44340
diff changeset
    75
    try:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44340
diff changeset
    76
        with revlog.opener(filename) as fd:
51656
21442f87f7fa mmap: only use mmap to read revlog persistent nodemap if it is safe
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51655
diff changeset
    77
            if use_mmap and revlog.opener.is_mmap_safe(filename):
46971
a3720569a43f nodemap: deal with data mmap error
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46794
diff changeset
    78
                try:
a3720569a43f nodemap: deal with data mmap error
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46794
diff changeset
    79
                    data = util.buffer(util.mmapread(fd, data_length))
a3720569a43f nodemap: deal with data mmap error
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46794
diff changeset
    80
                except ValueError:
a3720569a43f nodemap: deal with data mmap error
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46794
diff changeset
    81
                    # raised when the read file is too small
a3720569a43f nodemap: deal with data mmap error
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46794
diff changeset
    82
                    data = b''
44363
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44340
diff changeset
    83
            else:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44340
diff changeset
    84
                data = fd.read(data_length)
49306
2e726c934fcd py3: catch FileNotFoundError instead of checking errno == ENOENT
Manuel Jacob <me@manueljacob.de>
parents: 48946
diff changeset
    85
    except FileNotFoundError:
2e726c934fcd py3: catch FileNotFoundError instead of checking errno == ENOENT
Manuel Jacob <me@manueljacob.de>
parents: 48946
diff changeset
    86
        return None
44339
c7eebdb15139 nodemap: never read more than the expected data amount
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44338
diff changeset
    87
    if len(data) < data_length:
c7eebdb15139 nodemap: never read more than the expected data amount
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44338
diff changeset
    88
        return None
c7eebdb15139 nodemap: never read more than the expected data amount
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44338
diff changeset
    89
    return docket, data
44309
6c07480d6659 nodemap: add a function to read the data from disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44308
diff changeset
    90
6c07480d6659 nodemap: add a function to read the data from disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44308
diff changeset
    91
44308
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44307
diff changeset
    92
def setup_persistent_nodemap(tr, revlog):
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44307
diff changeset
    93
    """Install whatever is needed transaction side to persist a nodemap on disk
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44307
diff changeset
    94
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44307
diff changeset
    95
    (only actually persist the nodemap if this is relevant for this revlog)
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44307
diff changeset
    96
    """
44310
daad3aace942 nodemap: only use persistent nodemap for non-inlined revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44309
diff changeset
    97
    if revlog._inline:
daad3aace942 nodemap: only use persistent nodemap for non-inlined revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44309
diff changeset
    98
        return  # inlined revlog are too small for this to be relevant
47164
04f2f94836a5 revlog: rename `nodemap_file` to `_nodemap_file`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47152
diff changeset
    99
    if revlog._nodemap_file is None:
44308
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44307
diff changeset
   100
        return  # we do not use persistent_nodemap on this revlog
44517
448d700e0d27 nodemap: make sure the nodemap docket is updated after the changelog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44516
diff changeset
   101
448d700e0d27 nodemap: make sure the nodemap docket is updated after the changelog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44516
diff changeset
   102
    # we need to happen after the changelog finalization, in that use "cl-"
47164
04f2f94836a5 revlog: rename `nodemap_file` to `_nodemap_file`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47152
diff changeset
   103
    callback_id = b"nm-revlog-persistent-nodemap-%s" % revlog._nodemap_file
44308
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44307
diff changeset
   104
    if tr.hasfinalize(callback_id):
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44307
diff changeset
   105
        return  # no need to register again
44516
64e2f603de9d nodemap: make sure hooks have access to an up-to-date version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44515
diff changeset
   106
    tr.addpending(
46468
83f037acf71a nodemap: make `_persist_nodemap` a public function
Pulkit Goyal <7895pulkit@gmail.com>
parents: 46321
diff changeset
   107
        callback_id, lambda tr: persist_nodemap(tr, revlog, pending=True)
44516
64e2f603de9d nodemap: make sure hooks have access to an up-to-date version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44515
diff changeset
   108
    )
46468
83f037acf71a nodemap: make `_persist_nodemap` a public function
Pulkit Goyal <7895pulkit@gmail.com>
parents: 46321
diff changeset
   109
    tr.addfinalize(callback_id, lambda tr: persist_nodemap(tr, revlog))
44497
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44494
diff changeset
   110
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44494
diff changeset
   111
48946
642e31cb55f0 py3: use class X: instead of class X(object):
Gregory Szorc <gregory.szorc@gmail.com>
parents: 48875
diff changeset
   112
class _NoTransaction:
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 44792
diff changeset
   113
    """transaction like object to update the nodemap outside a transaction"""
44497
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44494
diff changeset
   114
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44494
diff changeset
   115
    def __init__(self):
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44494
diff changeset
   116
        self._postclose = {}
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44494
diff changeset
   117
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44494
diff changeset
   118
    def addpostclose(self, callback_id, callback_func):
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44494
diff changeset
   119
        self._postclose[callback_id] = callback_func
44308
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44307
diff changeset
   120
44634
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44517
diff changeset
   121
    def registertmp(self, *args, **kwargs):
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44517
diff changeset
   122
        pass
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44517
diff changeset
   123
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44517
diff changeset
   124
    def addbackup(self, *args, **kwargs):
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44517
diff changeset
   125
        pass
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44517
diff changeset
   126
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44517
diff changeset
   127
    def add(self, *args, **kwargs):
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44517
diff changeset
   128
        pass
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44517
diff changeset
   129
44635
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44634
diff changeset
   130
    def addabort(self, *args, **kwargs):
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44634
diff changeset
   131
        pass
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44634
diff changeset
   132
44788
6493f0a567c2 nodemap: add a new mode option, with an optional "warn" value
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44745
diff changeset
   133
    def _report(self, *args):
6493f0a567c2 nodemap: add a new mode option, with an optional "warn" value
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44745
diff changeset
   134
        pass
6493f0a567c2 nodemap: add a new mode option, with an optional "warn" value
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44745
diff changeset
   135
44308
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44307
diff changeset
   136
44445
336ec75ed1ac nodemap: warm the persistent nodemap on disk with debugupdatecache
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44363
diff changeset
   137
def update_persistent_nodemap(revlog):
336ec75ed1ac nodemap: warm the persistent nodemap on disk with debugupdatecache
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44363
diff changeset
   138
    """update the persistent nodemap right now
336ec75ed1ac nodemap: warm the persistent nodemap on disk with debugupdatecache
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44363
diff changeset
   139
336ec75ed1ac nodemap: warm the persistent nodemap on disk with debugupdatecache
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44363
diff changeset
   140
    To be used for updating the nodemap on disk outside of a normal transaction
336ec75ed1ac nodemap: warm the persistent nodemap on disk with debugupdatecache
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44363
diff changeset
   141
    setup (eg, `debugupdatecache`).
336ec75ed1ac nodemap: warm the persistent nodemap on disk with debugupdatecache
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44363
diff changeset
   142
    """
44745
b4537125eb3c nodemap: skip persistent nodemap warming for revlog not using it
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44697
diff changeset
   143
    if revlog._inline:
b4537125eb3c nodemap: skip persistent nodemap warming for revlog not using it
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44697
diff changeset
   144
        return  # inlined revlog are too small for this to be relevant
47164
04f2f94836a5 revlog: rename `nodemap_file` to `_nodemap_file`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47152
diff changeset
   145
    if revlog._nodemap_file is None:
44745
b4537125eb3c nodemap: skip persistent nodemap warming for revlog not using it
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44697
diff changeset
   146
        return  # we do not use persistent_nodemap on this revlog
b4537125eb3c nodemap: skip persistent nodemap warming for revlog not using it
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44697
diff changeset
   147
44497
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44494
diff changeset
   148
    notr = _NoTransaction()
46468
83f037acf71a nodemap: make `_persist_nodemap` a public function
Pulkit Goyal <7895pulkit@gmail.com>
parents: 46321
diff changeset
   149
    persist_nodemap(notr, revlog)
44497
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44494
diff changeset
   150
    for k in sorted(notr._postclose):
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44494
diff changeset
   151
        notr._postclose[k](None)
44445
336ec75ed1ac nodemap: warm the persistent nodemap on disk with debugupdatecache
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44363
diff changeset
   152
336ec75ed1ac nodemap: warm the persistent nodemap on disk with debugupdatecache
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44363
diff changeset
   153
46526
67b5fafd3a46 upgrade: speed up when we have only nodemap to downgrade
Pulkit Goyal <7895pulkit@gmail.com>
parents: 46472
diff changeset
   154
def delete_nodemap(tr, repo, revlog):
47451
f70ca39d0ab8 nodemap: fix some comment formatting
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47312
diff changeset
   155
    """Delete nodemap data on disk for a given revlog"""
48690
d55c4472bbb6 persistent-nodemap: properly delete all nodemap files on downgrade
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48424
diff changeset
   156
    prefix = revlog.radix
d55c4472bbb6 persistent-nodemap: properly delete all nodemap files on downgrade
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48424
diff changeset
   157
    pattern = re.compile(br"(^|/)%s(-[0-9a-f]+\.nd|\.n(\.a)?)$" % prefix)
d55c4472bbb6 persistent-nodemap: properly delete all nodemap files on downgrade
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48424
diff changeset
   158
    dirpath = revlog.opener.dirname(revlog._indexfile)
d55c4472bbb6 persistent-nodemap: properly delete all nodemap files on downgrade
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48424
diff changeset
   159
    for f in revlog.opener.listdir(dirpath):
d55c4472bbb6 persistent-nodemap: properly delete all nodemap files on downgrade
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48424
diff changeset
   160
        if pattern.match(f):
d55c4472bbb6 persistent-nodemap: properly delete all nodemap files on downgrade
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48424
diff changeset
   161
            repo.svfs.tryunlink(f)
46526
67b5fafd3a46 upgrade: speed up when we have only nodemap to downgrade
Pulkit Goyal <7895pulkit@gmail.com>
parents: 46472
diff changeset
   162
67b5fafd3a46 upgrade: speed up when we have only nodemap to downgrade
Pulkit Goyal <7895pulkit@gmail.com>
parents: 46472
diff changeset
   163
46472
98e39f04d60e upgrade: implement partial upgrade for upgrading persistent-nodemap
Pulkit Goyal <7895pulkit@gmail.com>
parents: 46470
diff changeset
   164
def persist_nodemap(tr, revlog, pending=False, force=False):
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 44792
diff changeset
   165
    """Write nodemap data on disk for a given revlog"""
51175
1486d8c63f64 persistent-nodemap: avoid writing nodemap for empty revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50928
diff changeset
   166
    if len(revlog.index) <= 0:
1486d8c63f64 persistent-nodemap: avoid writing nodemap for empty revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50928
diff changeset
   167
        return
44308
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44307
diff changeset
   168
    if getattr(revlog, 'filteredrevs', ()):
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44307
diff changeset
   169
        raise error.ProgrammingError(
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44307
diff changeset
   170
            "cannot persist nodemap of a filtered changelog"
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44307
diff changeset
   171
        )
47164
04f2f94836a5 revlog: rename `nodemap_file` to `_nodemap_file`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47152
diff changeset
   172
    if revlog._nodemap_file is None:
46472
98e39f04d60e upgrade: implement partial upgrade for upgrading persistent-nodemap
Pulkit Goyal <7895pulkit@gmail.com>
parents: 46470
diff changeset
   173
        if force:
47164
04f2f94836a5 revlog: rename `nodemap_file` to `_nodemap_file`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47152
diff changeset
   174
            revlog._nodemap_file = get_nodemap_file(revlog)
46472
98e39f04d60e upgrade: implement partial upgrade for upgrading persistent-nodemap
Pulkit Goyal <7895pulkit@gmail.com>
parents: 46470
diff changeset
   175
        else:
98e39f04d60e upgrade: implement partial upgrade for upgrading persistent-nodemap
Pulkit Goyal <7895pulkit@gmail.com>
parents: 46470
diff changeset
   176
            msg = "calling persist nodemap on a revlog without the feature enabled"
98e39f04d60e upgrade: implement partial upgrade for upgrading persistent-nodemap
Pulkit Goyal <7895pulkit@gmail.com>
parents: 46470
diff changeset
   177
            raise error.ProgrammingError(msg)
44333
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   178
50928
d718eddf01d9 safehasattr: drop usage in favor of hasattr
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50510
diff changeset
   179
    can_incremental = hasattr(revlog.index, "nodemap_data_incremental")
44333
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   180
    ondisk_docket = revlog._nodemap_docket
50928
d718eddf01d9 safehasattr: drop usage in favor of hasattr
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50510
diff changeset
   181
    feed_data = hasattr(revlog.index, "update_nodemap_data")
44792
5e3c718692bb nodemap: drop the 'exp-' prefix for internal opener option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44790
diff changeset
   182
    use_mmap = revlog.opener.options.get(b"persistent-nodemap.mmap")
44333
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   183
44337
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44336
diff changeset
   184
    data = None
44333
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   185
    # first attemp an incremental update of the data
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   186
    if can_incremental and ondisk_docket is not None:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   187
        target_docket = revlog._nodemap_docket.copy()
44337
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44336
diff changeset
   188
        (
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44336
diff changeset
   189
            src_docket,
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44336
diff changeset
   190
            data_changed_count,
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44336
diff changeset
   191
            data,
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44336
diff changeset
   192
        ) = revlog.index.nodemap_data_incremental()
44636
c70bcaf7927b nodemap: automatically "vacuum" the persistent nodemap when too sparse
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44635
diff changeset
   193
        new_length = target_docket.data_length + len(data)
c70bcaf7927b nodemap: automatically "vacuum" the persistent nodemap when too sparse
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44635
diff changeset
   194
        new_unused = target_docket.data_unused + data_changed_count
44337
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44336
diff changeset
   195
        if src_docket != target_docket:
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44336
diff changeset
   196
            data = None
44636
c70bcaf7927b nodemap: automatically "vacuum" the persistent nodemap when too sparse
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44635
diff changeset
   197
        elif new_length <= (new_unused * 10):  # under 10% of unused data
c70bcaf7927b nodemap: automatically "vacuum" the persistent nodemap when too sparse
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44635
diff changeset
   198
            data = None
44337
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44336
diff changeset
   199
        else:
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44336
diff changeset
   200
            datafile = _rawdata_filepath(revlog, target_docket)
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44336
diff changeset
   201
            # EXP-TODO: if this is a cache, this should use a cache vfs, not a
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44336
diff changeset
   202
            # store vfs
44634
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44517
diff changeset
   203
            tr.add(datafile, target_docket.data_length)
44338
2ea6a67ff502 nodemap: write new data from the expected current data length
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44337
diff changeset
   204
            with revlog.opener(datafile, b'r+') as fd:
2ea6a67ff502 nodemap: write new data from the expected current data length
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44337
diff changeset
   205
                fd.seek(target_docket.data_length)
44337
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44336
diff changeset
   206
                fd.write(data)
44363
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44340
diff changeset
   207
                if feed_data:
51656
21442f87f7fa mmap: only use mmap to read revlog persistent nodemap if it is safe
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51655
diff changeset
   208
                    if use_mmap and revlog.opener.is_mmap_safe(datafile):
51265
85d96517e650 persistent-nodemap: respect the mmap setting when refreshing data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51175
diff changeset
   209
                        fd.flush()
85d96517e650 persistent-nodemap: respect the mmap setting when refreshing data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51175
diff changeset
   210
                        new_data = util.buffer(util.mmapread(fd, new_length))
85d96517e650 persistent-nodemap: respect the mmap setting when refreshing data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51175
diff changeset
   211
                    else:
44363
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44340
diff changeset
   212
                        fd.seek(0)
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44340
diff changeset
   213
                        new_data = fd.read(new_length)
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44340
diff changeset
   214
            target_docket.data_length = new_length
44636
c70bcaf7927b nodemap: automatically "vacuum" the persistent nodemap when too sparse
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44635
diff changeset
   215
            target_docket.data_unused = new_unused
44337
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44336
diff changeset
   216
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44336
diff changeset
   217
    if data is None:
44333
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   218
        # otherwise fallback to a full new export
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   219
        target_docket = NodeMapDocket()
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   220
        datafile = _rawdata_filepath(revlog, target_docket)
50928
d718eddf01d9 safehasattr: drop usage in favor of hasattr
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50510
diff changeset
   221
        if hasattr(revlog.index, "nodemap_data_all"):
44333
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   222
            data = revlog.index.nodemap_data_all()
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   223
        else:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   224
            data = persistent_data(revlog.index)
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   225
        # EXP-TODO: if this is a cache, this should use a cache vfs, not a
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   226
        # store vfs
44635
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44634
diff changeset
   227
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44634
diff changeset
   228
        tryunlink = revlog.opener.tryunlink
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44634
diff changeset
   229
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44634
diff changeset
   230
        def abortck(tr):
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44634
diff changeset
   231
            tryunlink(datafile)
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44634
diff changeset
   232
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44634
diff changeset
   233
        callback_id = b"delete-%s" % datafile
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44634
diff changeset
   234
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44634
diff changeset
   235
        # some flavor of the transaction abort does not cleanup new file, it
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44634
diff changeset
   236
        # simply empty them.
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44634
diff changeset
   237
        tr.addabort(callback_id, abortck)
44363
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44340
diff changeset
   238
        with revlog.opener(datafile, b'w+') as fd:
44333
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   239
            fd.write(data)
44363
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44340
diff changeset
   240
            if feed_data:
51656
21442f87f7fa mmap: only use mmap to read revlog persistent nodemap if it is safe
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51655
diff changeset
   241
                if use_mmap and revlog.opener.is_mmap_safe(datafile):
44363
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44340
diff changeset
   242
                    fd.flush()
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44340
diff changeset
   243
                    new_data = util.buffer(util.mmapread(fd, len(data)))
51655
8e24f4f86ba8 mmap: fix another instance of reverse mmap logic in persistent nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51265
diff changeset
   244
                else:
8e24f4f86ba8 mmap: fix another instance of reverse mmap logic in persistent nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51265
diff changeset
   245
                    new_data = data
44336
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44335
diff changeset
   246
        target_docket.data_length = len(data)
44335
e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44333
diff changeset
   247
    target_docket.tip_rev = revlog.tiprev()
44515
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44497
diff changeset
   248
    target_docket.tip_node = revlog.node(target_docket.tip_rev)
44333
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   249
    # EXP-TODO: if this is a cache, this should use a cache vfs, not a
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   250
    # store vfs
47164
04f2f94836a5 revlog: rename `nodemap_file` to `_nodemap_file`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47152
diff changeset
   251
    file_path = revlog._nodemap_file
44516
64e2f603de9d nodemap: make sure hooks have access to an up-to-date version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44515
diff changeset
   252
    if pending:
64e2f603de9d nodemap: make sure hooks have access to an up-to-date version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44515
diff changeset
   253
        file_path += b'.a'
44634
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44517
diff changeset
   254
        tr.registertmp(file_path)
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44517
diff changeset
   255
    else:
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44517
diff changeset
   256
        tr.addbackup(file_path)
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44517
diff changeset
   257
44516
64e2f603de9d nodemap: make sure hooks have access to an up-to-date version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44515
diff changeset
   258
    with revlog.opener(file_path, b'w', atomictemp=True) as fp:
44333
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   259
        fp.write(target_docket.serialize())
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   260
    revlog._nodemap_docket = target_docket
44363
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44340
diff changeset
   261
    if feed_data:
44340
6ecc34b31137 nodemap: update the index with the newly written data (when appropriate)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44339
diff changeset
   262
        revlog.index.update_nodemap_data(target_docket, new_data)
6ecc34b31137 nodemap: update the index with the newly written data (when appropriate)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44339
diff changeset
   263
44333
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   264
    # search for old index file in all cases, some older process might have
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   265
    # left one behind.
44322
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44321
diff changeset
   266
    olds = _other_rawdata_filepath(revlog, target_docket)
44312
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44311
diff changeset
   267
    if olds:
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44311
diff changeset
   268
        realvfs = getattr(revlog, '_realopener', revlog.opener)
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44311
diff changeset
   269
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44311
diff changeset
   270
        def cleanup(tr):
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44311
diff changeset
   271
            for oldfile in olds:
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44311
diff changeset
   272
                realvfs.tryunlink(oldfile)
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44311
diff changeset
   273
47164
04f2f94836a5 revlog: rename `nodemap_file` to `_nodemap_file`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47152
diff changeset
   274
        callback_id = b"revlog-cleanup-nodemap-%s" % revlog._nodemap_file
44497
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44494
diff changeset
   275
        tr.addpostclose(callback_id, cleanup)
44311
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
   276
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
   277
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
   278
### Nodemap docket file
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
   279
#
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
   280
# The nodemap data are stored on disk using 2 files:
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
   281
#
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
   282
# * a raw data files containing a persistent nodemap
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
   283
#   (see `Nodemap Trie` section)
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
   284
#
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
   285
# * a small "docket" file containing medatadata
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
   286
#
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
   287
# While the nodemap data can be multiple tens of megabytes, the "docket" is
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
   288
# small, it is easy to update it automatically or to duplicated its content
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
   289
# during a transaction.
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
   290
#
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
   291
# Multiple raw data can exist at the same time (The currently valid one and a
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
   292
# new one beind used by an in progress transaction). To accomodate this, the
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
   293
# filename hosting the raw data has a variable parts. The exact filename is
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
   294
# specified inside the "docket" file.
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
   295
#
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
   296
# The docket file contains information to find, qualify and validate the raw
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
   297
# data. Its content is currently very light, but it will expand as the on disk
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
   298
# nodemap gains the necessary features to be used in production.
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
   299
44790
261e71752d1f nodemap: move on disk file to version 1
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44789
diff changeset
   300
ONDISK_VERSION = 1
44311
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
   301
S_VERSION = struct.Struct(">B")
44515
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44497
diff changeset
   302
S_HEADER = struct.Struct(">BQQQQ")
44311
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
   303
47310
9a3aa54774ff persistent-nodemap: add a way to make the picked uid predictable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47240
diff changeset
   304
48946
642e31cb55f0 py3: use class X: instead of class X(object):
Gregory Szorc <gregory.szorc@gmail.com>
parents: 48875
diff changeset
   305
class NodeMapDocket:
44322
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44321
diff changeset
   306
    """metadata associated with persistent nodemap data
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44321
diff changeset
   307
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44321
diff changeset
   308
    The persistent data may come from disk or be on their way to disk.
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44321
diff changeset
   309
    """
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44321
diff changeset
   310
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44321
diff changeset
   311
    def __init__(self, uid=None):
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44321
diff changeset
   312
        if uid is None:
47312
7ea39d633cf3 docket: move the uid logic in the `revlogutils.docket` module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47311
diff changeset
   313
            uid = docket_mod.make_uid()
44494
283fd803afa5 nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44462
diff changeset
   314
        # a unique identifier for the data file:
283fd803afa5 nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44462
diff changeset
   315
        #   - When new data are appended, it is preserved.
283fd803afa5 nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44462
diff changeset
   316
        #   - When a new data file is created, a new identifier is generated.
44322
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44321
diff changeset
   317
        self.uid = uid
44494
283fd803afa5 nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44462
diff changeset
   318
        # the tipmost revision stored in the data file. This revision and all
283fd803afa5 nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44462
diff changeset
   319
        # revision before it are expected to be encoded in the data file.
44335
e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44333
diff changeset
   320
        self.tip_rev = None
44515
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44497
diff changeset
   321
        # the node of that tipmost revision, if it mismatch the current index
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44497
diff changeset
   322
        # data the docket is not valid for the current index and should be
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44497
diff changeset
   323
        # discarded.
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44497
diff changeset
   324
        #
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44497
diff changeset
   325
        # note: this method is not perfect as some destructive operation could
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44497
diff changeset
   326
        # preserve the same tip_rev + tip_node while altering lower revision.
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44497
diff changeset
   327
        # However this multiple other caches have the same vulnerability (eg:
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44497
diff changeset
   328
        # brancmap cache).
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44497
diff changeset
   329
        self.tip_node = None
44494
283fd803afa5 nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44462
diff changeset
   330
        # the size (in bytes) of the persisted data to encode the nodemap valid
283fd803afa5 nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44462
diff changeset
   331
        # for `tip_rev`.
283fd803afa5 nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44462
diff changeset
   332
        #   - data file shorter than this are corrupted,
283fd803afa5 nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44462
diff changeset
   333
        #   - any extra data should be ignored.
44336
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44335
diff changeset
   334
        self.data_length = None
44494
283fd803afa5 nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44462
diff changeset
   335
        # the amount (in bytes) of "dead" data, still in the data file but no
283fd803afa5 nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44462
diff changeset
   336
        # longer used for the nodemap.
44336
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44335
diff changeset
   337
        self.data_unused = 0
44322
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44321
diff changeset
   338
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44321
diff changeset
   339
    def copy(self):
44335
e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44333
diff changeset
   340
        new = NodeMapDocket(uid=self.uid)
e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44333
diff changeset
   341
        new.tip_rev = self.tip_rev
44515
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44497
diff changeset
   342
        new.tip_node = self.tip_node
44336
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44335
diff changeset
   343
        new.data_length = self.data_length
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44335
diff changeset
   344
        new.data_unused = self.data_unused
44335
e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44333
diff changeset
   345
        return new
44322
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44321
diff changeset
   346
44337
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44336
diff changeset
   347
    def __cmp__(self, other):
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44336
diff changeset
   348
        if self.uid < other.uid:
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44336
diff changeset
   349
            return -1
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44336
diff changeset
   350
        if self.uid > other.uid:
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44336
diff changeset
   351
            return 1
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44336
diff changeset
   352
        elif self.data_length < other.data_length:
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44336
diff changeset
   353
            return -1
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44336
diff changeset
   354
        elif self.data_length > other.data_length:
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44336
diff changeset
   355
            return 1
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44336
diff changeset
   356
        return 0
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44336
diff changeset
   357
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44336
diff changeset
   358
    def __eq__(self, other):
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44336
diff changeset
   359
        return self.uid == other.uid and self.data_length == other.data_length
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44336
diff changeset
   360
44322
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44321
diff changeset
   361
    def serialize(self):
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44321
diff changeset
   362
        """return serialized bytes for a docket using the passed uid"""
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44321
diff changeset
   363
        data = []
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44321
diff changeset
   364
        data.append(S_VERSION.pack(ONDISK_VERSION))
44336
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44335
diff changeset
   365
        headers = (
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44335
diff changeset
   366
            len(self.uid),
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44335
diff changeset
   367
            self.tip_rev,
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44335
diff changeset
   368
            self.data_length,
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44335
diff changeset
   369
            self.data_unused,
44515
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44497
diff changeset
   370
            len(self.tip_node),
44336
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44335
diff changeset
   371
        )
44335
e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44333
diff changeset
   372
        data.append(S_HEADER.pack(*headers))
44322
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44321
diff changeset
   373
        data.append(self.uid)
44515
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44497
diff changeset
   374
        data.append(self.tip_node)
44322
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44321
diff changeset
   375
        return b''.join(data)
44311
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
   376
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
   377
44322
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44321
diff changeset
   378
def _rawdata_filepath(revlog, docket):
44311
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
   379
    """The (vfs relative) nodemap's rawdata file for a given uid"""
47152
09338a2d5c14 revlog: also use radix when computing nodemap data file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47151
diff changeset
   380
    prefix = revlog.radix
44322
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44321
diff changeset
   381
    return b"%s-%s.nd" % (prefix, docket.uid)
44308
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44307
diff changeset
   382
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44307
diff changeset
   383
44322
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44321
diff changeset
   384
def _other_rawdata_filepath(revlog, docket):
47152
09338a2d5c14 revlog: also use radix when computing nodemap data file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47151
diff changeset
   385
    prefix = revlog.radix
44462
6aee0647e026 nodemap: fix missing r-prefix on regular expression
Augie Fackler <augie@google.com>
parents: 44445
diff changeset
   386
    pattern = re.compile(br"(^|/)%s-[0-9a-f]+\.nd$" % prefix)
44322
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44321
diff changeset
   387
    new_file_path = _rawdata_filepath(revlog, docket)
44312
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44311
diff changeset
   388
    new_file_name = revlog.opener.basename(new_file_path)
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44311
diff changeset
   389
    dirpath = revlog.opener.dirname(new_file_path)
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44311
diff changeset
   390
    others = []
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44311
diff changeset
   391
    for f in revlog.opener.listdir(dirpath):
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44311
diff changeset
   392
        if pattern.match(f) and f != new_file_name:
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44311
diff changeset
   393
            others.append(f)
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44311
diff changeset
   394
    return others
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44311
diff changeset
   395
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44311
diff changeset
   396
44307
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   397
### Nodemap Trie
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   398
#
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   399
# This is a simple reference implementation to compute and persist a nodemap
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   400
# trie. This reference implementation is write only. The python version of this
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   401
# is not expected to be actually used, since it wont provide performance
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   402
# improvement over existing non-persistent C implementation.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   403
#
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   404
# The nodemap is persisted as Trie using 4bits-address/16-entries block. each
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   405
# revision can be adressed using its node shortest prefix.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   406
#
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   407
# The trie is stored as a sequence of block. Each block contains 16 entries
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   408
# (signed 64bit integer, big endian). Each entry can be one of the following:
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   409
#
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   410
#  * value >=  0 -> index of sub-block
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   411
#  * value == -1 -> no value
46199
8e7ce6555ea7 nodemap: match comment to actual code
Joerg Sonnenberger <joerg@bec.de>
parents: 46113
diff changeset
   412
#  * value <  -1 -> encoded revision: rev = -(value+2)
8e7ce6555ea7 nodemap: match comment to actual code
Joerg Sonnenberger <joerg@bec.de>
parents: 46113
diff changeset
   413
#
8e7ce6555ea7 nodemap: match comment to actual code
Joerg Sonnenberger <joerg@bec.de>
parents: 46113
diff changeset
   414
# See REV_OFFSET and _transform_rev below.
44307
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   415
#
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   416
# The implementation focus on simplicity, not on performance. A Rust
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   417
# implementation should provide a efficient version of the same binary
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   418
# persistence. This reference python implementation is never meant to be
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   419
# extensively use in production.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   420
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   421
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   422
def persistent_data(index):
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 44792
diff changeset
   423
    """return the persistent binary form for a nodemap for a given index"""
44307
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   424
    trie = _build_trie(index)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   425
    return _persist_trie(trie)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   426
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   427
44333
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   428
def update_persistent_data(index, root, max_idx, last_rev):
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 44792
diff changeset
   429
    """return the incremental update for persistent nodemap from a given index"""
44336
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44335
diff changeset
   430
    changed_block, trie = _update_trie(index, root, last_rev)
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44335
diff changeset
   431
    return (
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44335
diff changeset
   432
        changed_block * S_BLOCK.size,
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44335
diff changeset
   433
        _persist_trie(trie, existing_idx=max_idx),
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44335
diff changeset
   434
    )
44333
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   435
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   436
44307
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   437
S_BLOCK = struct.Struct(">" + ("l" * 16))
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   438
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   439
NO_ENTRY = -1
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   440
# rev 0 need to be -2 because 0 is used by block, -1 is a special value.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   441
REV_OFFSET = 2
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   442
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   443
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   444
def _transform_rev(rev):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   445
    """Return the number used to represent the rev in the tree.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   446
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   447
    (or retrieve a rev number from such representation)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   448
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   449
    Note that this is an involution, a function equal to its inverse (i.e.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   450
    which gives the identity when applied to itself).
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   451
    """
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   452
    return -(rev + REV_OFFSET)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   453
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   454
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   455
def _to_int(hex_digit):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   456
    """turn an hexadecimal digit into a proper integer"""
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   457
    return int(hex_digit, 16)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   458
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   459
44315
7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44314
diff changeset
   460
class Block(dict):
7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44314
diff changeset
   461
    """represent a block of the Trie
7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44314
diff changeset
   462
7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44314
diff changeset
   463
    contains up to 16 entry indexed from 0 to 15"""
7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44314
diff changeset
   464
44321
f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44319
diff changeset
   465
    def __init__(self):
f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44319
diff changeset
   466
        super(Block, self).__init__()
f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44319
diff changeset
   467
        # If this block exist on disk, here is its ID
f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44319
diff changeset
   468
        self.ondisk_id = None
f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44319
diff changeset
   469
44316
55b12f2593c1 nodemap: move the iteratio inside the Block object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44315
diff changeset
   470
    def __iter__(self):
55b12f2593c1 nodemap: move the iteratio inside the Block object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44315
diff changeset
   471
        return iter(self.get(i) for i in range(16))
55b12f2593c1 nodemap: move the iteratio inside the Block object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44315
diff changeset
   472
44315
7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44314
diff changeset
   473
44307
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   474
def _build_trie(index):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   475
    """build a nodemap trie
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   476
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   477
    The nodemap stores revision number for each unique prefix.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   478
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   479
    Each block is a dictionary with keys in `[0, 15]`. Values are either
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   480
    another block or a revision number.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   481
    """
44315
7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44314
diff changeset
   482
    root = Block()
44307
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   483
    for rev in range(len(index)):
46113
59fa3890d40a node: import symbols explicitly
Joerg Sonnenberger <joerg@bec.de>
parents: 46089
diff changeset
   484
        current_hex = hex(index[rev][7])
59fa3890d40a node: import symbols explicitly
Joerg Sonnenberger <joerg@bec.de>
parents: 46089
diff changeset
   485
        _insert_into_block(index, 0, root, rev, current_hex)
44307
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   486
    return root
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   487
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   488
44333
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   489
def _update_trie(index, root, last_rev):
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   490
    """consume"""
44336
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44335
diff changeset
   491
    changed = 0
44333
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   492
    for rev in range(last_rev + 1, len(index)):
46113
59fa3890d40a node: import symbols explicitly
Joerg Sonnenberger <joerg@bec.de>
parents: 46089
diff changeset
   493
        current_hex = hex(index[rev][7])
59fa3890d40a node: import symbols explicitly
Joerg Sonnenberger <joerg@bec.de>
parents: 46089
diff changeset
   494
        changed += _insert_into_block(index, 0, root, rev, current_hex)
44336
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44335
diff changeset
   495
    return changed, root
44333
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   496
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   497
44307
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   498
def _insert_into_block(index, level, block, current_rev, current_hex):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   499
    """insert a new revision in a block
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   500
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   501
    index: the index we are adding revision for
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   502
    level: the depth of the current block in the trie
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   503
    block: the block currently being considered
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   504
    current_rev: the revision number we are adding
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   505
    current_hex: the hexadecimal representation of the of that revision
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   506
    """
44336
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44335
diff changeset
   507
    changed = 1
44333
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   508
    if block.ondisk_id is not None:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   509
        block.ondisk_id = None
44307
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   510
    hex_digit = _to_int(current_hex[level : level + 1])
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   511
    entry = block.get(hex_digit)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   512
    if entry is None:
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   513
        # no entry, simply store the revision number
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   514
        block[hex_digit] = current_rev
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   515
    elif isinstance(entry, dict):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   516
        # need to recurse to an underlying block
44336
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44335
diff changeset
   517
        changed += _insert_into_block(
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44335
diff changeset
   518
            index, level + 1, entry, current_rev, current_hex
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44335
diff changeset
   519
        )
44307
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   520
    else:
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   521
        # collision with a previously unique prefix, inserting new
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   522
        # vertices to fit both entry.
46113
59fa3890d40a node: import symbols explicitly
Joerg Sonnenberger <joerg@bec.de>
parents: 46089
diff changeset
   523
        other_hex = hex(index[entry][7])
44307
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   524
        other_rev = entry
44315
7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44314
diff changeset
   525
        new = Block()
44307
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   526
        block[hex_digit] = new
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   527
        _insert_into_block(index, level + 1, new, other_rev, other_hex)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   528
        _insert_into_block(index, level + 1, new, current_rev, current_hex)
44336
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44335
diff changeset
   529
    return changed
44307
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   530
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   531
44333
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   532
def _persist_trie(root, existing_idx=None):
44307
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   533
    """turn a nodemap trie into persistent binary data
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   534
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   535
    See `_build_trie` for nodemap trie structure"""
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   536
    block_map = {}
44333
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   537
    if existing_idx is not None:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   538
        base_idx = existing_idx + 1
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   539
    else:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   540
        base_idx = 0
44307
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   541
    chunks = []
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   542
    for tn in _walk_trie(root):
44333
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   543
        if tn.ondisk_id is not None:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   544
            block_map[id(tn)] = tn.ondisk_id
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   545
        else:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   546
            block_map[id(tn)] = len(chunks) + base_idx
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   547
            chunks.append(_persist_block(tn, block_map))
44307
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   548
    return b''.join(chunks)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   549
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   550
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   551
def _walk_trie(block):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   552
    """yield all the block in a trie
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   553
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   554
    Children blocks are always yield before their parent block.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   555
    """
44696
ed475420e0af nodemap: rename a variable to prevent shawoding "_"
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44636
diff changeset
   556
    for (__, item) in sorted(block.items()):
44307
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   557
        if isinstance(item, dict):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   558
            for sub_block in _walk_trie(item):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   559
                yield sub_block
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   560
    yield block
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   561
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   562
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   563
def _persist_block(block_node, block_map):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   564
    """produce persistent binary data for a single block
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   565
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   566
    Children block are assumed to be already persisted and present in
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   567
    block_map.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   568
    """
44316
55b12f2593c1 nodemap: move the iteratio inside the Block object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44315
diff changeset
   569
    data = tuple(_to_value(v, block_map) for v in block_node)
44307
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   570
    return S_BLOCK.pack(*data)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   571
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   572
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   573
def _to_value(item, block_map):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   574
    """persist any value as an integer"""
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   575
    if item is None:
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   576
        return NO_ENTRY
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   577
    elif isinstance(item, dict):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   578
        return block_map[id(item)]
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   579
    else:
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   580
        return _transform_rev(item)
44317
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44316
diff changeset
   581
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44316
diff changeset
   582
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44316
diff changeset
   583
def parse_data(data):
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44316
diff changeset
   584
    """parse parse nodemap data into a nodemap Trie"""
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44316
diff changeset
   585
    if (len(data) % S_BLOCK.size) != 0:
46684
98c816e89fac nodemap: convert error message to bytes
Matt Harbison <matt_harbison@yahoo.com>
parents: 46321
diff changeset
   586
        msg = b"nodemap data size is not a multiple of block size (%d): %d"
44317
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44316
diff changeset
   587
        raise error.Abort(msg % (S_BLOCK.size, len(data)))
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44316
diff changeset
   588
    if not data:
44333
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   589
        return Block(), None
44317
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44316
diff changeset
   590
    block_map = {}
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44316
diff changeset
   591
    new_blocks = []
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44316
diff changeset
   592
    for i in range(0, len(data), S_BLOCK.size):
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44316
diff changeset
   593
        block = Block()
44321
f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44319
diff changeset
   594
        block.ondisk_id = len(block_map)
f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44319
diff changeset
   595
        block_map[block.ondisk_id] = block
44317
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44316
diff changeset
   596
        block_data = data[i : i + S_BLOCK.size]
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44316
diff changeset
   597
        values = S_BLOCK.unpack(block_data)
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44316
diff changeset
   598
        new_blocks.append((block, values))
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44316
diff changeset
   599
    for b, values in new_blocks:
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44316
diff changeset
   600
        for idx, v in enumerate(values):
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44316
diff changeset
   601
            if v == NO_ENTRY:
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44316
diff changeset
   602
                continue
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44316
diff changeset
   603
            elif v >= 0:
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44316
diff changeset
   604
                b[idx] = block_map[v]
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44316
diff changeset
   605
            else:
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44316
diff changeset
   606
                b[idx] = _transform_rev(v)
44333
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   607
    return block, i // S_BLOCK.size
44318
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44317
diff changeset
   608
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44317
diff changeset
   609
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44317
diff changeset
   610
# debug utility
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44317
diff changeset
   611
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44317
diff changeset
   612
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44317
diff changeset
   613
def check_data(ui, index, data):
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44317
diff changeset
   614
    """verify that the provided nodemap data are valid for the given idex"""
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44317
diff changeset
   615
    ret = 0
50510
4fe46f96b56b cli: fix spelling in `debugnodemap` error messages
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 49306
diff changeset
   616
    ui.status((b"revisions in index:   %d\n") % len(index))
44333
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   617
    root, __ = parse_data(data)
44318
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44317
diff changeset
   618
    all_revs = set(_all_revisions(root))
50510
4fe46f96b56b cli: fix spelling in `debugnodemap` error messages
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 49306
diff changeset
   619
    ui.status((b"revisions in nodemap: %d\n") % len(all_revs))
44318
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44317
diff changeset
   620
    for r in range(len(index)):
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44317
diff changeset
   621
        if r not in all_revs:
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44317
diff changeset
   622
            msg = b"  revision missing from nodemap: %d\n" % r
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44317
diff changeset
   623
            ui.write_err(msg)
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44317
diff changeset
   624
            ret = 1
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44317
diff changeset
   625
        else:
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44317
diff changeset
   626
            all_revs.remove(r)
46113
59fa3890d40a node: import symbols explicitly
Joerg Sonnenberger <joerg@bec.de>
parents: 46089
diff changeset
   627
        nm_rev = _find_node(root, hex(index[r][7]))
44319
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44318
diff changeset
   628
        if nm_rev is None:
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44318
diff changeset
   629
            msg = b"  revision node does not match any entries: %d\n" % r
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44318
diff changeset
   630
            ui.write_err(msg)
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44318
diff changeset
   631
            ret = 1
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44318
diff changeset
   632
        elif nm_rev != r:
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44318
diff changeset
   633
            msg = (
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44318
diff changeset
   634
                b"  revision node does not match the expected revision: "
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44318
diff changeset
   635
                b"%d != %d\n" % (r, nm_rev)
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44318
diff changeset
   636
            )
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44318
diff changeset
   637
            ui.write_err(msg)
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44318
diff changeset
   638
            ret = 1
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44318
diff changeset
   639
44318
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44317
diff changeset
   640
    if all_revs:
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44317
diff changeset
   641
        for r in sorted(all_revs):
50510
4fe46f96b56b cli: fix spelling in `debugnodemap` error messages
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 49306
diff changeset
   642
            msg = b"  extra revisions in  nodemap: %d\n" % r
44318
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44317
diff changeset
   643
            ui.write_err(msg)
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44317
diff changeset
   644
        ret = 1
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44317
diff changeset
   645
    return ret
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44317
diff changeset
   646
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44317
diff changeset
   647
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44317
diff changeset
   648
def _all_revisions(root):
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44317
diff changeset
   649
    """return all revisions stored in a Trie"""
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44317
diff changeset
   650
    for block in _walk_trie(root):
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44317
diff changeset
   651
        for v in block:
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44317
diff changeset
   652
            if v is None or isinstance(v, Block):
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44317
diff changeset
   653
                continue
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44317
diff changeset
   654
            yield v
44319
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44318
diff changeset
   655
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44318
diff changeset
   656
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44318
diff changeset
   657
def _find_node(block, node):
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44318
diff changeset
   658
    """find the revision associated with a given node"""
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44318
diff changeset
   659
    entry = block.get(_to_int(node[0:1]))
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44318
diff changeset
   660
    if isinstance(entry, dict):
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44318
diff changeset
   661
        return _find_node(entry, node[1:])
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44318
diff changeset
   662
    return entry
46469
835aafb2543f revlog: refactor logic to compute nodemap file in separate function
Pulkit Goyal <7895pulkit@gmail.com>
parents: 46468
diff changeset
   663
835aafb2543f revlog: refactor logic to compute nodemap file in separate function
Pulkit Goyal <7895pulkit@gmail.com>
parents: 46468
diff changeset
   664
47151
24be247a13b4 revlog: stop usage of `_indexfile` to computing nodemap path
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47148
diff changeset
   665
def get_nodemap_file(revlog):
47240
4f38ada3fc26 revlog: move the `trypending` logic from the `changelog` to the `revlog`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47164
diff changeset
   666
    if revlog._trypending:
47151
24be247a13b4 revlog: stop usage of `_indexfile` to computing nodemap path
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47148
diff changeset
   667
        pending_path = revlog.radix + b".n.a"
24be247a13b4 revlog: stop usage of `_indexfile` to computing nodemap path
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47148
diff changeset
   668
        if revlog.opener.exists(pending_path):
46469
835aafb2543f revlog: refactor logic to compute nodemap file in separate function
Pulkit Goyal <7895pulkit@gmail.com>
parents: 46468
diff changeset
   669
            return pending_path
47151
24be247a13b4 revlog: stop usage of `_indexfile` to computing nodemap path
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47148
diff changeset
   670
    return revlog.radix + b".n"