Mercurial > hg
annotate mercurial/revlogutils/nodemap.py @ 46089:8ff2d8359d0f
persistent-nodemap: properly ignore non-existent `.nd` data file
This code was meant to handle the case of a nodemap docket file
pointing to a nodemap data file that doesn’t exist (anymore),
but most likely caused an `UnboundLocalError` exception instead
when `data` was used on the next line without being defined.
This case is theoretically possible with a race condition
between two hg processes, but is hard to reproduce or test:
* Process A reads a docket file and finds a UID in it
that points to a given data file name.
* Process B decides that this same data file needs compacting.
It writes a new one with a different UID,
overwrites the docket file,
then removes the old data file.
* Only then process A tries to a open a file that doesn’t exist anymore.
Differential Revision: https://phab.mercurial-scm.org/D9533
author | Simon Sapin <simon-commits@exyr.org> |
---|---|
date | Mon, 07 Dec 2020 18:06:53 +0100 |
parents | 89a2afe31e82 |
children | 59fa3890d40a |
rev | line source |
---|---|
44034
ab595920de0e
revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
1 # nodemap.py - nodemap related code and utilities |
ab595920de0e
revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
2 # |
ab595920de0e
revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
3 # Copyright 2019 Pierre-Yves David <pierre-yves.david@octobus.net> |
ab595920de0e
revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
4 # Copyright 2019 George Racinet <georges.racinet@octobus.net> |
ab595920de0e
revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
5 # |
ab595920de0e
revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
6 # This software may be used and distributed according to the terms of the |
ab595920de0e
revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
7 # GNU General Public License version 2 or any later version. |
ab595920de0e
revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
8 |
ab595920de0e
revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
9 from __future__ import absolute_import |
44307
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
10 |
44363
f7459da77f23
nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44340
diff
changeset
|
11 import errno |
44311
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
12 import os |
44312
563dfdfd01a4
nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44311
diff
changeset
|
13 import re |
44307
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
14 import struct |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
15 |
44788
6493f0a567c2
nodemap: add a new mode option, with an optional "warn" value
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44745
diff
changeset
|
16 from ..i18n import _ |
6493f0a567c2
nodemap: add a new mode option, with an optional "warn" value
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44745
diff
changeset
|
17 |
44307
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
18 from .. import ( |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
19 error, |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
20 node as nodemod, |
44314
7f4f7ef3133e
nodemap: add a optional `nodemap_add_full` method on indexes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44312
diff
changeset
|
21 util, |
44307
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
22 ) |
44034
ab595920de0e
revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
23 |
ab595920de0e
revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
24 |
ab595920de0e
revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
25 class NodeMap(dict): |
ab595920de0e
revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
26 def __missing__(self, x): |
ab595920de0e
revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
27 raise error.RevlogError(b'unknown node: %s' % x) |
44307
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
28 |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
29 |
44309
6c07480d6659
nodemap: add a function to read the data from disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44308
diff
changeset
|
30 def persisted_data(revlog): |
6c07480d6659
nodemap: add a function to read the data from disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44308
diff
changeset
|
31 """read the nodemap for a revlog from disk""" |
6c07480d6659
nodemap: add a function to read the data from disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44308
diff
changeset
|
32 if revlog.nodemap_file is None: |
6c07480d6659
nodemap: add a function to read the data from disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44308
diff
changeset
|
33 return None |
44311
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
34 pdata = revlog.opener.tryread(revlog.nodemap_file) |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
35 if not pdata: |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
36 return None |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
37 offset = 0 |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
38 (version,) = S_VERSION.unpack(pdata[offset : offset + S_VERSION.size]) |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
39 if version != ONDISK_VERSION: |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
40 return None |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
41 offset += S_VERSION.size |
44335
e41a164db7a9
nodemap: track the maximum revision tracked in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44333
diff
changeset
|
42 headers = S_HEADER.unpack(pdata[offset : offset + S_HEADER.size]) |
44515
6c906eaedd0d
nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44497
diff
changeset
|
43 uid_size, tip_rev, data_length, data_unused, tip_node_size = headers |
44311
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
44 offset += S_HEADER.size |
44322
72c15641c8b4
nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44321
diff
changeset
|
45 docket = NodeMapDocket(pdata[offset : offset + uid_size]) |
44515
6c906eaedd0d
nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44497
diff
changeset
|
46 offset += uid_size |
44335
e41a164db7a9
nodemap: track the maximum revision tracked in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44333
diff
changeset
|
47 docket.tip_rev = tip_rev |
44515
6c906eaedd0d
nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44497
diff
changeset
|
48 docket.tip_node = pdata[offset : offset + tip_node_size] |
44336
8374b69aef75
nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44335
diff
changeset
|
49 docket.data_length = data_length |
8374b69aef75
nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44335
diff
changeset
|
50 docket.data_unused = data_unused |
44311
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
51 |
44322
72c15641c8b4
nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44321
diff
changeset
|
52 filename = _rawdata_filepath(revlog, docket) |
44792
5e3c718692bb
nodemap: drop the 'exp-' prefix for internal opener option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44790
diff
changeset
|
53 use_mmap = revlog.opener.options.get(b"persistent-nodemap.mmap") |
44363
f7459da77f23
nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44340
diff
changeset
|
54 try: |
f7459da77f23
nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44340
diff
changeset
|
55 with revlog.opener(filename) as fd: |
f7459da77f23
nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44340
diff
changeset
|
56 if use_mmap: |
f7459da77f23
nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44340
diff
changeset
|
57 data = util.buffer(util.mmapread(fd, data_length)) |
f7459da77f23
nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44340
diff
changeset
|
58 else: |
f7459da77f23
nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44340
diff
changeset
|
59 data = fd.read(data_length) |
f7459da77f23
nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44340
diff
changeset
|
60 except OSError as e: |
46089
8ff2d8359d0f
persistent-nodemap: properly ignore non-existent `.nd` data file
Simon Sapin <simon-commits@exyr.org>
parents:
45942
diff
changeset
|
61 if e.errno == errno.ENOENT: |
8ff2d8359d0f
persistent-nodemap: properly ignore non-existent `.nd` data file
Simon Sapin <simon-commits@exyr.org>
parents:
45942
diff
changeset
|
62 return None |
8ff2d8359d0f
persistent-nodemap: properly ignore non-existent `.nd` data file
Simon Sapin <simon-commits@exyr.org>
parents:
45942
diff
changeset
|
63 else: |
44363
f7459da77f23
nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44340
diff
changeset
|
64 raise |
44339
c7eebdb15139
nodemap: never read more than the expected data amount
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44338
diff
changeset
|
65 if len(data) < data_length: |
c7eebdb15139
nodemap: never read more than the expected data amount
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44338
diff
changeset
|
66 return None |
c7eebdb15139
nodemap: never read more than the expected data amount
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44338
diff
changeset
|
67 return docket, data |
44309
6c07480d6659
nodemap: add a function to read the data from disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44308
diff
changeset
|
68 |
6c07480d6659
nodemap: add a function to read the data from disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44308
diff
changeset
|
69 |
44308
5962fd0d1045
nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44307
diff
changeset
|
70 def setup_persistent_nodemap(tr, revlog): |
5962fd0d1045
nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44307
diff
changeset
|
71 """Install whatever is needed transaction side to persist a nodemap on disk |
5962fd0d1045
nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44307
diff
changeset
|
72 |
5962fd0d1045
nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44307
diff
changeset
|
73 (only actually persist the nodemap if this is relevant for this revlog) |
5962fd0d1045
nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44307
diff
changeset
|
74 """ |
44310
daad3aace942
nodemap: only use persistent nodemap for non-inlined revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44309
diff
changeset
|
75 if revlog._inline: |
daad3aace942
nodemap: only use persistent nodemap for non-inlined revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44309
diff
changeset
|
76 return # inlined revlog are too small for this to be relevant |
44308
5962fd0d1045
nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44307
diff
changeset
|
77 if revlog.nodemap_file is None: |
5962fd0d1045
nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44307
diff
changeset
|
78 return # we do not use persistent_nodemap on this revlog |
44517
448d700e0d27
nodemap: make sure the nodemap docket is updated after the changelog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44516
diff
changeset
|
79 |
448d700e0d27
nodemap: make sure the nodemap docket is updated after the changelog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44516
diff
changeset
|
80 # we need to happen after the changelog finalization, in that use "cl-" |
448d700e0d27
nodemap: make sure the nodemap docket is updated after the changelog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44516
diff
changeset
|
81 callback_id = b"nm-revlog-persistent-nodemap-%s" % revlog.nodemap_file |
44308
5962fd0d1045
nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44307
diff
changeset
|
82 if tr.hasfinalize(callback_id): |
5962fd0d1045
nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44307
diff
changeset
|
83 return # no need to register again |
44516
64e2f603de9d
nodemap: make sure hooks have access to an up-to-date version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44515
diff
changeset
|
84 tr.addpending( |
64e2f603de9d
nodemap: make sure hooks have access to an up-to-date version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44515
diff
changeset
|
85 callback_id, lambda tr: _persist_nodemap(tr, revlog, pending=True) |
64e2f603de9d
nodemap: make sure hooks have access to an up-to-date version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44515
diff
changeset
|
86 ) |
44497
3265c92f7d13
nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44494
diff
changeset
|
87 tr.addfinalize(callback_id, lambda tr: _persist_nodemap(tr, revlog)) |
3265c92f7d13
nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44494
diff
changeset
|
88 |
3265c92f7d13
nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44494
diff
changeset
|
89 |
3265c92f7d13
nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44494
diff
changeset
|
90 class _NoTransaction(object): |
45942
89a2afe31e82
formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents:
44792
diff
changeset
|
91 """transaction like object to update the nodemap outside a transaction""" |
44497
3265c92f7d13
nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44494
diff
changeset
|
92 |
3265c92f7d13
nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44494
diff
changeset
|
93 def __init__(self): |
3265c92f7d13
nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44494
diff
changeset
|
94 self._postclose = {} |
3265c92f7d13
nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44494
diff
changeset
|
95 |
3265c92f7d13
nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44494
diff
changeset
|
96 def addpostclose(self, callback_id, callback_func): |
3265c92f7d13
nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44494
diff
changeset
|
97 self._postclose[callback_id] = callback_func |
44308
5962fd0d1045
nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44307
diff
changeset
|
98 |
44634
01b0805534bb
nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44517
diff
changeset
|
99 def registertmp(self, *args, **kwargs): |
01b0805534bb
nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44517
diff
changeset
|
100 pass |
01b0805534bb
nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44517
diff
changeset
|
101 |
01b0805534bb
nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44517
diff
changeset
|
102 def addbackup(self, *args, **kwargs): |
01b0805534bb
nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44517
diff
changeset
|
103 pass |
01b0805534bb
nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44517
diff
changeset
|
104 |
01b0805534bb
nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44517
diff
changeset
|
105 def add(self, *args, **kwargs): |
01b0805534bb
nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44517
diff
changeset
|
106 pass |
01b0805534bb
nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44517
diff
changeset
|
107 |
44635
99ea74cbed74
nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44634
diff
changeset
|
108 def addabort(self, *args, **kwargs): |
99ea74cbed74
nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44634
diff
changeset
|
109 pass |
99ea74cbed74
nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44634
diff
changeset
|
110 |
44788
6493f0a567c2
nodemap: add a new mode option, with an optional "warn" value
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44745
diff
changeset
|
111 def _report(self, *args): |
6493f0a567c2
nodemap: add a new mode option, with an optional "warn" value
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44745
diff
changeset
|
112 pass |
6493f0a567c2
nodemap: add a new mode option, with an optional "warn" value
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44745
diff
changeset
|
113 |
44308
5962fd0d1045
nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44307
diff
changeset
|
114 |
44445
336ec75ed1ac
nodemap: warm the persistent nodemap on disk with debugupdatecache
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44363
diff
changeset
|
115 def update_persistent_nodemap(revlog): |
336ec75ed1ac
nodemap: warm the persistent nodemap on disk with debugupdatecache
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44363
diff
changeset
|
116 """update the persistent nodemap right now |
336ec75ed1ac
nodemap: warm the persistent nodemap on disk with debugupdatecache
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44363
diff
changeset
|
117 |
336ec75ed1ac
nodemap: warm the persistent nodemap on disk with debugupdatecache
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44363
diff
changeset
|
118 To be used for updating the nodemap on disk outside of a normal transaction |
336ec75ed1ac
nodemap: warm the persistent nodemap on disk with debugupdatecache
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44363
diff
changeset
|
119 setup (eg, `debugupdatecache`). |
336ec75ed1ac
nodemap: warm the persistent nodemap on disk with debugupdatecache
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44363
diff
changeset
|
120 """ |
44745
b4537125eb3c
nodemap: skip persistent nodemap warming for revlog not using it
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44697
diff
changeset
|
121 if revlog._inline: |
b4537125eb3c
nodemap: skip persistent nodemap warming for revlog not using it
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44697
diff
changeset
|
122 return # inlined revlog are too small for this to be relevant |
b4537125eb3c
nodemap: skip persistent nodemap warming for revlog not using it
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44697
diff
changeset
|
123 if revlog.nodemap_file is None: |
b4537125eb3c
nodemap: skip persistent nodemap warming for revlog not using it
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44697
diff
changeset
|
124 return # we do not use persistent_nodemap on this revlog |
b4537125eb3c
nodemap: skip persistent nodemap warming for revlog not using it
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44697
diff
changeset
|
125 |
44497
3265c92f7d13
nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44494
diff
changeset
|
126 notr = _NoTransaction() |
3265c92f7d13
nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44494
diff
changeset
|
127 _persist_nodemap(notr, revlog) |
3265c92f7d13
nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44494
diff
changeset
|
128 for k in sorted(notr._postclose): |
3265c92f7d13
nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44494
diff
changeset
|
129 notr._postclose[k](None) |
44445
336ec75ed1ac
nodemap: warm the persistent nodemap on disk with debugupdatecache
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44363
diff
changeset
|
130 |
336ec75ed1ac
nodemap: warm the persistent nodemap on disk with debugupdatecache
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44363
diff
changeset
|
131 |
44516
64e2f603de9d
nodemap: make sure hooks have access to an up-to-date version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44515
diff
changeset
|
132 def _persist_nodemap(tr, revlog, pending=False): |
45942
89a2afe31e82
formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents:
44792
diff
changeset
|
133 """Write nodemap data on disk for a given revlog""" |
44308
5962fd0d1045
nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44307
diff
changeset
|
134 if getattr(revlog, 'filteredrevs', ()): |
5962fd0d1045
nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44307
diff
changeset
|
135 raise error.ProgrammingError( |
5962fd0d1045
nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44307
diff
changeset
|
136 "cannot persist nodemap of a filtered changelog" |
5962fd0d1045
nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44307
diff
changeset
|
137 ) |
5962fd0d1045
nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44307
diff
changeset
|
138 if revlog.nodemap_file is None: |
5962fd0d1045
nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44307
diff
changeset
|
139 msg = "calling persist nodemap on a revlog without the feature enableb" |
5962fd0d1045
nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44307
diff
changeset
|
140 raise error.ProgrammingError(msg) |
44333
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
141 |
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
142 can_incremental = util.safehasattr(revlog.index, "nodemap_data_incremental") |
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
143 ondisk_docket = revlog._nodemap_docket |
44363
f7459da77f23
nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44340
diff
changeset
|
144 feed_data = util.safehasattr(revlog.index, "update_nodemap_data") |
44792
5e3c718692bb
nodemap: drop the 'exp-' prefix for internal opener option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44790
diff
changeset
|
145 use_mmap = revlog.opener.options.get(b"persistent-nodemap.mmap") |
5e3c718692bb
nodemap: drop the 'exp-' prefix for internal opener option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44790
diff
changeset
|
146 mode = revlog.opener.options.get(b"persistent-nodemap.mode") |
44788
6493f0a567c2
nodemap: add a new mode option, with an optional "warn" value
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44745
diff
changeset
|
147 if not can_incremental: |
6493f0a567c2
nodemap: add a new mode option, with an optional "warn" value
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44745
diff
changeset
|
148 msg = _(b"persistent nodemap in strict mode without efficient method") |
6493f0a567c2
nodemap: add a new mode option, with an optional "warn" value
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44745
diff
changeset
|
149 if mode == b'warn': |
6493f0a567c2
nodemap: add a new mode option, with an optional "warn" value
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44745
diff
changeset
|
150 tr._report(b"%s\n" % msg) |
44789
6b01799e9db0
nodemap: add a new mode value, "strict"
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44788
diff
changeset
|
151 elif mode == b'strict': |
6b01799e9db0
nodemap: add a new mode value, "strict"
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44788
diff
changeset
|
152 raise error.Abort(msg) |
44333
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
153 |
44337
1d2b37def017
nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44336
diff
changeset
|
154 data = None |
44333
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
155 # first attemp an incremental update of the data |
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
156 if can_incremental and ondisk_docket is not None: |
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
157 target_docket = revlog._nodemap_docket.copy() |
44337
1d2b37def017
nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44336
diff
changeset
|
158 ( |
1d2b37def017
nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44336
diff
changeset
|
159 src_docket, |
1d2b37def017
nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44336
diff
changeset
|
160 data_changed_count, |
1d2b37def017
nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44336
diff
changeset
|
161 data, |
1d2b37def017
nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44336
diff
changeset
|
162 ) = revlog.index.nodemap_data_incremental() |
44636
c70bcaf7927b
nodemap: automatically "vacuum" the persistent nodemap when too sparse
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44635
diff
changeset
|
163 new_length = target_docket.data_length + len(data) |
c70bcaf7927b
nodemap: automatically "vacuum" the persistent nodemap when too sparse
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44635
diff
changeset
|
164 new_unused = target_docket.data_unused + data_changed_count |
44337
1d2b37def017
nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44336
diff
changeset
|
165 if src_docket != target_docket: |
1d2b37def017
nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44336
diff
changeset
|
166 data = None |
44636
c70bcaf7927b
nodemap: automatically "vacuum" the persistent nodemap when too sparse
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44635
diff
changeset
|
167 elif new_length <= (new_unused * 10): # under 10% of unused data |
c70bcaf7927b
nodemap: automatically "vacuum" the persistent nodemap when too sparse
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44635
diff
changeset
|
168 data = None |
44337
1d2b37def017
nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44336
diff
changeset
|
169 else: |
1d2b37def017
nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44336
diff
changeset
|
170 datafile = _rawdata_filepath(revlog, target_docket) |
1d2b37def017
nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44336
diff
changeset
|
171 # EXP-TODO: if this is a cache, this should use a cache vfs, not a |
1d2b37def017
nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44336
diff
changeset
|
172 # store vfs |
44634
01b0805534bb
nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44517
diff
changeset
|
173 tr.add(datafile, target_docket.data_length) |
44338
2ea6a67ff502
nodemap: write new data from the expected current data length
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44337
diff
changeset
|
174 with revlog.opener(datafile, b'r+') as fd: |
2ea6a67ff502
nodemap: write new data from the expected current data length
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44337
diff
changeset
|
175 fd.seek(target_docket.data_length) |
44337
1d2b37def017
nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44336
diff
changeset
|
176 fd.write(data) |
44363
f7459da77f23
nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44340
diff
changeset
|
177 if feed_data: |
f7459da77f23
nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44340
diff
changeset
|
178 if use_mmap: |
f7459da77f23
nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44340
diff
changeset
|
179 fd.seek(0) |
f7459da77f23
nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44340
diff
changeset
|
180 new_data = fd.read(new_length) |
f7459da77f23
nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44340
diff
changeset
|
181 else: |
f7459da77f23
nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44340
diff
changeset
|
182 fd.flush() |
f7459da77f23
nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44340
diff
changeset
|
183 new_data = util.buffer(util.mmapread(fd, new_length)) |
f7459da77f23
nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44340
diff
changeset
|
184 target_docket.data_length = new_length |
44636
c70bcaf7927b
nodemap: automatically "vacuum" the persistent nodemap when too sparse
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44635
diff
changeset
|
185 target_docket.data_unused = new_unused |
44337
1d2b37def017
nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44336
diff
changeset
|
186 |
1d2b37def017
nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44336
diff
changeset
|
187 if data is None: |
44333
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
188 # otherwise fallback to a full new export |
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
189 target_docket = NodeMapDocket() |
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
190 datafile = _rawdata_filepath(revlog, target_docket) |
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
191 if util.safehasattr(revlog.index, "nodemap_data_all"): |
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
192 data = revlog.index.nodemap_data_all() |
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
193 else: |
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
194 data = persistent_data(revlog.index) |
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
195 # EXP-TODO: if this is a cache, this should use a cache vfs, not a |
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
196 # store vfs |
44635
99ea74cbed74
nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44634
diff
changeset
|
197 |
99ea74cbed74
nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44634
diff
changeset
|
198 tryunlink = revlog.opener.tryunlink |
99ea74cbed74
nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44634
diff
changeset
|
199 |
99ea74cbed74
nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44634
diff
changeset
|
200 def abortck(tr): |
99ea74cbed74
nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44634
diff
changeset
|
201 tryunlink(datafile) |
99ea74cbed74
nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44634
diff
changeset
|
202 |
99ea74cbed74
nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44634
diff
changeset
|
203 callback_id = b"delete-%s" % datafile |
99ea74cbed74
nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44634
diff
changeset
|
204 |
99ea74cbed74
nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44634
diff
changeset
|
205 # some flavor of the transaction abort does not cleanup new file, it |
99ea74cbed74
nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44634
diff
changeset
|
206 # simply empty them. |
99ea74cbed74
nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44634
diff
changeset
|
207 tr.addabort(callback_id, abortck) |
44363
f7459da77f23
nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44340
diff
changeset
|
208 with revlog.opener(datafile, b'w+') as fd: |
44333
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
209 fd.write(data) |
44363
f7459da77f23
nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44340
diff
changeset
|
210 if feed_data: |
f7459da77f23
nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44340
diff
changeset
|
211 if use_mmap: |
f7459da77f23
nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44340
diff
changeset
|
212 new_data = data |
f7459da77f23
nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44340
diff
changeset
|
213 else: |
f7459da77f23
nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44340
diff
changeset
|
214 fd.flush() |
f7459da77f23
nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44340
diff
changeset
|
215 new_data = util.buffer(util.mmapread(fd, len(data))) |
44336
8374b69aef75
nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44335
diff
changeset
|
216 target_docket.data_length = len(data) |
44335
e41a164db7a9
nodemap: track the maximum revision tracked in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44333
diff
changeset
|
217 target_docket.tip_rev = revlog.tiprev() |
44515
6c906eaedd0d
nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44497
diff
changeset
|
218 target_docket.tip_node = revlog.node(target_docket.tip_rev) |
44333
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
219 # EXP-TODO: if this is a cache, this should use a cache vfs, not a |
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
220 # store vfs |
44516
64e2f603de9d
nodemap: make sure hooks have access to an up-to-date version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44515
diff
changeset
|
221 file_path = revlog.nodemap_file |
64e2f603de9d
nodemap: make sure hooks have access to an up-to-date version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44515
diff
changeset
|
222 if pending: |
64e2f603de9d
nodemap: make sure hooks have access to an up-to-date version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44515
diff
changeset
|
223 file_path += b'.a' |
44634
01b0805534bb
nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44517
diff
changeset
|
224 tr.registertmp(file_path) |
01b0805534bb
nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44517
diff
changeset
|
225 else: |
01b0805534bb
nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44517
diff
changeset
|
226 tr.addbackup(file_path) |
01b0805534bb
nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44517
diff
changeset
|
227 |
44516
64e2f603de9d
nodemap: make sure hooks have access to an up-to-date version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44515
diff
changeset
|
228 with revlog.opener(file_path, b'w', atomictemp=True) as fp: |
44333
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
229 fp.write(target_docket.serialize()) |
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
230 revlog._nodemap_docket = target_docket |
44363
f7459da77f23
nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44340
diff
changeset
|
231 if feed_data: |
44340
6ecc34b31137
nodemap: update the index with the newly written data (when appropriate)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44339
diff
changeset
|
232 revlog.index.update_nodemap_data(target_docket, new_data) |
6ecc34b31137
nodemap: update the index with the newly written data (when appropriate)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44339
diff
changeset
|
233 |
44333
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
234 # search for old index file in all cases, some older process might have |
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
235 # left one behind. |
44322
72c15641c8b4
nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44321
diff
changeset
|
236 olds = _other_rawdata_filepath(revlog, target_docket) |
44312
563dfdfd01a4
nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44311
diff
changeset
|
237 if olds: |
563dfdfd01a4
nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44311
diff
changeset
|
238 realvfs = getattr(revlog, '_realopener', revlog.opener) |
563dfdfd01a4
nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44311
diff
changeset
|
239 |
563dfdfd01a4
nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44311
diff
changeset
|
240 def cleanup(tr): |
563dfdfd01a4
nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44311
diff
changeset
|
241 for oldfile in olds: |
563dfdfd01a4
nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44311
diff
changeset
|
242 realvfs.tryunlink(oldfile) |
563dfdfd01a4
nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44311
diff
changeset
|
243 |
563dfdfd01a4
nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44311
diff
changeset
|
244 callback_id = b"revlog-cleanup-nodemap-%s" % revlog.nodemap_file |
44497
3265c92f7d13
nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44494
diff
changeset
|
245 tr.addpostclose(callback_id, cleanup) |
44311
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
246 |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
247 |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
248 ### Nodemap docket file |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
249 # |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
250 # The nodemap data are stored on disk using 2 files: |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
251 # |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
252 # * a raw data files containing a persistent nodemap |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
253 # (see `Nodemap Trie` section) |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
254 # |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
255 # * a small "docket" file containing medatadata |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
256 # |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
257 # While the nodemap data can be multiple tens of megabytes, the "docket" is |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
258 # small, it is easy to update it automatically or to duplicated its content |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
259 # during a transaction. |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
260 # |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
261 # Multiple raw data can exist at the same time (The currently valid one and a |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
262 # new one beind used by an in progress transaction). To accomodate this, the |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
263 # filename hosting the raw data has a variable parts. The exact filename is |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
264 # specified inside the "docket" file. |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
265 # |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
266 # The docket file contains information to find, qualify and validate the raw |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
267 # data. Its content is currently very light, but it will expand as the on disk |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
268 # nodemap gains the necessary features to be used in production. |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
269 |
44790
261e71752d1f
nodemap: move on disk file to version 1
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44789
diff
changeset
|
270 ONDISK_VERSION = 1 |
44311
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
271 S_VERSION = struct.Struct(">B") |
44515
6c906eaedd0d
nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44497
diff
changeset
|
272 S_HEADER = struct.Struct(">BQQQQ") |
44311
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
273 |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
274 ID_SIZE = 8 |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
275 |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
276 |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
277 def _make_uid(): |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
278 """return a new unique identifier. |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
279 |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
280 The identifier is random and composed of ascii characters.""" |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
281 return nodemod.hex(os.urandom(ID_SIZE)) |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
282 |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
283 |
44322
72c15641c8b4
nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44321
diff
changeset
|
284 class NodeMapDocket(object): |
72c15641c8b4
nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44321
diff
changeset
|
285 """metadata associated with persistent nodemap data |
72c15641c8b4
nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44321
diff
changeset
|
286 |
72c15641c8b4
nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44321
diff
changeset
|
287 The persistent data may come from disk or be on their way to disk. |
72c15641c8b4
nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44321
diff
changeset
|
288 """ |
72c15641c8b4
nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44321
diff
changeset
|
289 |
72c15641c8b4
nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44321
diff
changeset
|
290 def __init__(self, uid=None): |
72c15641c8b4
nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44321
diff
changeset
|
291 if uid is None: |
72c15641c8b4
nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44321
diff
changeset
|
292 uid = _make_uid() |
44494
283fd803afa5
nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44462
diff
changeset
|
293 # a unique identifier for the data file: |
283fd803afa5
nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44462
diff
changeset
|
294 # - When new data are appended, it is preserved. |
283fd803afa5
nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44462
diff
changeset
|
295 # - When a new data file is created, a new identifier is generated. |
44322
72c15641c8b4
nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44321
diff
changeset
|
296 self.uid = uid |
44494
283fd803afa5
nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44462
diff
changeset
|
297 # the tipmost revision stored in the data file. This revision and all |
283fd803afa5
nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44462
diff
changeset
|
298 # revision before it are expected to be encoded in the data file. |
44335
e41a164db7a9
nodemap: track the maximum revision tracked in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44333
diff
changeset
|
299 self.tip_rev = None |
44515
6c906eaedd0d
nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44497
diff
changeset
|
300 # the node of that tipmost revision, if it mismatch the current index |
6c906eaedd0d
nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44497
diff
changeset
|
301 # data the docket is not valid for the current index and should be |
6c906eaedd0d
nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44497
diff
changeset
|
302 # discarded. |
6c906eaedd0d
nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44497
diff
changeset
|
303 # |
6c906eaedd0d
nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44497
diff
changeset
|
304 # note: this method is not perfect as some destructive operation could |
6c906eaedd0d
nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44497
diff
changeset
|
305 # preserve the same tip_rev + tip_node while altering lower revision. |
6c906eaedd0d
nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44497
diff
changeset
|
306 # However this multiple other caches have the same vulnerability (eg: |
6c906eaedd0d
nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44497
diff
changeset
|
307 # brancmap cache). |
6c906eaedd0d
nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44497
diff
changeset
|
308 self.tip_node = None |
44494
283fd803afa5
nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44462
diff
changeset
|
309 # the size (in bytes) of the persisted data to encode the nodemap valid |
283fd803afa5
nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44462
diff
changeset
|
310 # for `tip_rev`. |
283fd803afa5
nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44462
diff
changeset
|
311 # - data file shorter than this are corrupted, |
283fd803afa5
nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44462
diff
changeset
|
312 # - any extra data should be ignored. |
44336
8374b69aef75
nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44335
diff
changeset
|
313 self.data_length = None |
44494
283fd803afa5
nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44462
diff
changeset
|
314 # the amount (in bytes) of "dead" data, still in the data file but no |
283fd803afa5
nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44462
diff
changeset
|
315 # longer used for the nodemap. |
44336
8374b69aef75
nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44335
diff
changeset
|
316 self.data_unused = 0 |
44322
72c15641c8b4
nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44321
diff
changeset
|
317 |
72c15641c8b4
nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44321
diff
changeset
|
318 def copy(self): |
44335
e41a164db7a9
nodemap: track the maximum revision tracked in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44333
diff
changeset
|
319 new = NodeMapDocket(uid=self.uid) |
e41a164db7a9
nodemap: track the maximum revision tracked in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44333
diff
changeset
|
320 new.tip_rev = self.tip_rev |
44515
6c906eaedd0d
nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44497
diff
changeset
|
321 new.tip_node = self.tip_node |
44336
8374b69aef75
nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44335
diff
changeset
|
322 new.data_length = self.data_length |
8374b69aef75
nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44335
diff
changeset
|
323 new.data_unused = self.data_unused |
44335
e41a164db7a9
nodemap: track the maximum revision tracked in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44333
diff
changeset
|
324 return new |
44322
72c15641c8b4
nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44321
diff
changeset
|
325 |
44337
1d2b37def017
nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44336
diff
changeset
|
326 def __cmp__(self, other): |
1d2b37def017
nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44336
diff
changeset
|
327 if self.uid < other.uid: |
1d2b37def017
nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44336
diff
changeset
|
328 return -1 |
1d2b37def017
nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44336
diff
changeset
|
329 if self.uid > other.uid: |
1d2b37def017
nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44336
diff
changeset
|
330 return 1 |
1d2b37def017
nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44336
diff
changeset
|
331 elif self.data_length < other.data_length: |
1d2b37def017
nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44336
diff
changeset
|
332 return -1 |
1d2b37def017
nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44336
diff
changeset
|
333 elif self.data_length > other.data_length: |
1d2b37def017
nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44336
diff
changeset
|
334 return 1 |
1d2b37def017
nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44336
diff
changeset
|
335 return 0 |
1d2b37def017
nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44336
diff
changeset
|
336 |
1d2b37def017
nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44336
diff
changeset
|
337 def __eq__(self, other): |
1d2b37def017
nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44336
diff
changeset
|
338 return self.uid == other.uid and self.data_length == other.data_length |
1d2b37def017
nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44336
diff
changeset
|
339 |
44322
72c15641c8b4
nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44321
diff
changeset
|
340 def serialize(self): |
72c15641c8b4
nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44321
diff
changeset
|
341 """return serialized bytes for a docket using the passed uid""" |
72c15641c8b4
nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44321
diff
changeset
|
342 data = [] |
72c15641c8b4
nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44321
diff
changeset
|
343 data.append(S_VERSION.pack(ONDISK_VERSION)) |
44336
8374b69aef75
nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44335
diff
changeset
|
344 headers = ( |
8374b69aef75
nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44335
diff
changeset
|
345 len(self.uid), |
8374b69aef75
nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44335
diff
changeset
|
346 self.tip_rev, |
8374b69aef75
nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44335
diff
changeset
|
347 self.data_length, |
8374b69aef75
nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44335
diff
changeset
|
348 self.data_unused, |
44515
6c906eaedd0d
nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44497
diff
changeset
|
349 len(self.tip_node), |
44336
8374b69aef75
nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44335
diff
changeset
|
350 ) |
44335
e41a164db7a9
nodemap: track the maximum revision tracked in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44333
diff
changeset
|
351 data.append(S_HEADER.pack(*headers)) |
44322
72c15641c8b4
nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44321
diff
changeset
|
352 data.append(self.uid) |
44515
6c906eaedd0d
nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44497
diff
changeset
|
353 data.append(self.tip_node) |
44322
72c15641c8b4
nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44321
diff
changeset
|
354 return b''.join(data) |
44311
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
355 |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
356 |
44322
72c15641c8b4
nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44321
diff
changeset
|
357 def _rawdata_filepath(revlog, docket): |
44311
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
358 """The (vfs relative) nodemap's rawdata file for a given uid""" |
44516
64e2f603de9d
nodemap: make sure hooks have access to an up-to-date version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44515
diff
changeset
|
359 if revlog.nodemap_file.endswith(b'.n.a'): |
64e2f603de9d
nodemap: make sure hooks have access to an up-to-date version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44515
diff
changeset
|
360 prefix = revlog.nodemap_file[:-4] |
64e2f603de9d
nodemap: make sure hooks have access to an up-to-date version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44515
diff
changeset
|
361 else: |
64e2f603de9d
nodemap: make sure hooks have access to an up-to-date version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44515
diff
changeset
|
362 prefix = revlog.nodemap_file[:-2] |
44322
72c15641c8b4
nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44321
diff
changeset
|
363 return b"%s-%s.nd" % (prefix, docket.uid) |
44308
5962fd0d1045
nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44307
diff
changeset
|
364 |
5962fd0d1045
nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44307
diff
changeset
|
365 |
44322
72c15641c8b4
nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44321
diff
changeset
|
366 def _other_rawdata_filepath(revlog, docket): |
44312
563dfdfd01a4
nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44311
diff
changeset
|
367 prefix = revlog.nodemap_file[:-2] |
44462
6aee0647e026
nodemap: fix missing r-prefix on regular expression
Augie Fackler <augie@google.com>
parents:
44445
diff
changeset
|
368 pattern = re.compile(br"(^|/)%s-[0-9a-f]+\.nd$" % prefix) |
44322
72c15641c8b4
nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44321
diff
changeset
|
369 new_file_path = _rawdata_filepath(revlog, docket) |
44312
563dfdfd01a4
nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44311
diff
changeset
|
370 new_file_name = revlog.opener.basename(new_file_path) |
563dfdfd01a4
nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44311
diff
changeset
|
371 dirpath = revlog.opener.dirname(new_file_path) |
563dfdfd01a4
nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44311
diff
changeset
|
372 others = [] |
563dfdfd01a4
nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44311
diff
changeset
|
373 for f in revlog.opener.listdir(dirpath): |
563dfdfd01a4
nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44311
diff
changeset
|
374 if pattern.match(f) and f != new_file_name: |
563dfdfd01a4
nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44311
diff
changeset
|
375 others.append(f) |
563dfdfd01a4
nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44311
diff
changeset
|
376 return others |
563dfdfd01a4
nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44311
diff
changeset
|
377 |
563dfdfd01a4
nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44311
diff
changeset
|
378 |
44307
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
379 ### Nodemap Trie |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
380 # |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
381 # This is a simple reference implementation to compute and persist a nodemap |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
382 # trie. This reference implementation is write only. The python version of this |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
383 # is not expected to be actually used, since it wont provide performance |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
384 # improvement over existing non-persistent C implementation. |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
385 # |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
386 # The nodemap is persisted as Trie using 4bits-address/16-entries block. each |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
387 # revision can be adressed using its node shortest prefix. |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
388 # |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
389 # The trie is stored as a sequence of block. Each block contains 16 entries |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
390 # (signed 64bit integer, big endian). Each entry can be one of the following: |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
391 # |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
392 # * value >= 0 -> index of sub-block |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
393 # * value == -1 -> no value |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
394 # * value < -1 -> a revision value: rev = -(value+10) |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
395 # |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
396 # The implementation focus on simplicity, not on performance. A Rust |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
397 # implementation should provide a efficient version of the same binary |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
398 # persistence. This reference python implementation is never meant to be |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
399 # extensively use in production. |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
400 |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
401 |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
402 def persistent_data(index): |
45942
89a2afe31e82
formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents:
44792
diff
changeset
|
403 """return the persistent binary form for a nodemap for a given index""" |
44307
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
404 trie = _build_trie(index) |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
405 return _persist_trie(trie) |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
406 |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
407 |
44333
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
408 def update_persistent_data(index, root, max_idx, last_rev): |
45942
89a2afe31e82
formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents:
44792
diff
changeset
|
409 """return the incremental update for persistent nodemap from a given index""" |
44336
8374b69aef75
nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44335
diff
changeset
|
410 changed_block, trie = _update_trie(index, root, last_rev) |
8374b69aef75
nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44335
diff
changeset
|
411 return ( |
8374b69aef75
nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44335
diff
changeset
|
412 changed_block * S_BLOCK.size, |
8374b69aef75
nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44335
diff
changeset
|
413 _persist_trie(trie, existing_idx=max_idx), |
8374b69aef75
nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44335
diff
changeset
|
414 ) |
44333
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
415 |
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
416 |
44307
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
417 S_BLOCK = struct.Struct(">" + ("l" * 16)) |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
418 |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
419 NO_ENTRY = -1 |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
420 # rev 0 need to be -2 because 0 is used by block, -1 is a special value. |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
421 REV_OFFSET = 2 |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
422 |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
423 |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
424 def _transform_rev(rev): |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
425 """Return the number used to represent the rev in the tree. |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
426 |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
427 (or retrieve a rev number from such representation) |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
428 |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
429 Note that this is an involution, a function equal to its inverse (i.e. |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
430 which gives the identity when applied to itself). |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
431 """ |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
432 return -(rev + REV_OFFSET) |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
433 |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
434 |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
435 def _to_int(hex_digit): |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
436 """turn an hexadecimal digit into a proper integer""" |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
437 return int(hex_digit, 16) |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
438 |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
439 |
44315
7762a295fd4d
nodemap: use an explicit "Block" object in the reference implementation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44314
diff
changeset
|
440 class Block(dict): |
7762a295fd4d
nodemap: use an explicit "Block" object in the reference implementation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44314
diff
changeset
|
441 """represent a block of the Trie |
7762a295fd4d
nodemap: use an explicit "Block" object in the reference implementation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44314
diff
changeset
|
442 |
7762a295fd4d
nodemap: use an explicit "Block" object in the reference implementation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44314
diff
changeset
|
443 contains up to 16 entry indexed from 0 to 15""" |
7762a295fd4d
nodemap: use an explicit "Block" object in the reference implementation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44314
diff
changeset
|
444 |
44321
f0862ee1a31e
nodemap: keep track of the ondisk id of nodemap blocks
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44319
diff
changeset
|
445 def __init__(self): |
f0862ee1a31e
nodemap: keep track of the ondisk id of nodemap blocks
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44319
diff
changeset
|
446 super(Block, self).__init__() |
f0862ee1a31e
nodemap: keep track of the ondisk id of nodemap blocks
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44319
diff
changeset
|
447 # If this block exist on disk, here is its ID |
f0862ee1a31e
nodemap: keep track of the ondisk id of nodemap blocks
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44319
diff
changeset
|
448 self.ondisk_id = None |
f0862ee1a31e
nodemap: keep track of the ondisk id of nodemap blocks
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44319
diff
changeset
|
449 |
44316
55b12f2593c1
nodemap: move the iteratio inside the Block object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44315
diff
changeset
|
450 def __iter__(self): |
55b12f2593c1
nodemap: move the iteratio inside the Block object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44315
diff
changeset
|
451 return iter(self.get(i) for i in range(16)) |
55b12f2593c1
nodemap: move the iteratio inside the Block object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44315
diff
changeset
|
452 |
44315
7762a295fd4d
nodemap: use an explicit "Block" object in the reference implementation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44314
diff
changeset
|
453 |
44307
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
454 def _build_trie(index): |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
455 """build a nodemap trie |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
456 |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
457 The nodemap stores revision number for each unique prefix. |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
458 |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
459 Each block is a dictionary with keys in `[0, 15]`. Values are either |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
460 another block or a revision number. |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
461 """ |
44315
7762a295fd4d
nodemap: use an explicit "Block" object in the reference implementation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44314
diff
changeset
|
462 root = Block() |
44307
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
463 for rev in range(len(index)): |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
464 hex = nodemod.hex(index[rev][7]) |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
465 _insert_into_block(index, 0, root, rev, hex) |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
466 return root |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
467 |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
468 |
44333
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
469 def _update_trie(index, root, last_rev): |
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
470 """consume""" |
44336
8374b69aef75
nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44335
diff
changeset
|
471 changed = 0 |
44333
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
472 for rev in range(last_rev + 1, len(index)): |
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
473 hex = nodemod.hex(index[rev][7]) |
44336
8374b69aef75
nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44335
diff
changeset
|
474 changed += _insert_into_block(index, 0, root, rev, hex) |
8374b69aef75
nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44335
diff
changeset
|
475 return changed, root |
44333
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
476 |
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
477 |
44307
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
478 def _insert_into_block(index, level, block, current_rev, current_hex): |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
479 """insert a new revision in a block |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
480 |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
481 index: the index we are adding revision for |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
482 level: the depth of the current block in the trie |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
483 block: the block currently being considered |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
484 current_rev: the revision number we are adding |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
485 current_hex: the hexadecimal representation of the of that revision |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
486 """ |
44336
8374b69aef75
nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44335
diff
changeset
|
487 changed = 1 |
44333
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
488 if block.ondisk_id is not None: |
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
489 block.ondisk_id = None |
44307
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
490 hex_digit = _to_int(current_hex[level : level + 1]) |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
491 entry = block.get(hex_digit) |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
492 if entry is None: |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
493 # no entry, simply store the revision number |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
494 block[hex_digit] = current_rev |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
495 elif isinstance(entry, dict): |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
496 # need to recurse to an underlying block |
44336
8374b69aef75
nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44335
diff
changeset
|
497 changed += _insert_into_block( |
8374b69aef75
nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44335
diff
changeset
|
498 index, level + 1, entry, current_rev, current_hex |
8374b69aef75
nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44335
diff
changeset
|
499 ) |
44307
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
500 else: |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
501 # collision with a previously unique prefix, inserting new |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
502 # vertices to fit both entry. |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
503 other_hex = nodemod.hex(index[entry][7]) |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
504 other_rev = entry |
44315
7762a295fd4d
nodemap: use an explicit "Block" object in the reference implementation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44314
diff
changeset
|
505 new = Block() |
44307
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
506 block[hex_digit] = new |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
507 _insert_into_block(index, level + 1, new, other_rev, other_hex) |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
508 _insert_into_block(index, level + 1, new, current_rev, current_hex) |
44336
8374b69aef75
nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44335
diff
changeset
|
509 return changed |
44307
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
510 |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
511 |
44333
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
512 def _persist_trie(root, existing_idx=None): |
44307
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
513 """turn a nodemap trie into persistent binary data |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
514 |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
515 See `_build_trie` for nodemap trie structure""" |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
516 block_map = {} |
44333
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
517 if existing_idx is not None: |
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
518 base_idx = existing_idx + 1 |
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
519 else: |
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
520 base_idx = 0 |
44307
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
521 chunks = [] |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
522 for tn in _walk_trie(root): |
44333
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
523 if tn.ondisk_id is not None: |
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
524 block_map[id(tn)] = tn.ondisk_id |
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
525 else: |
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
526 block_map[id(tn)] = len(chunks) + base_idx |
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
527 chunks.append(_persist_block(tn, block_map)) |
44307
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
528 return b''.join(chunks) |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
529 |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
530 |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
531 def _walk_trie(block): |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
532 """yield all the block in a trie |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
533 |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
534 Children blocks are always yield before their parent block. |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
535 """ |
44696
ed475420e0af
nodemap: rename a variable to prevent shawoding "_"
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44636
diff
changeset
|
536 for (__, item) in sorted(block.items()): |
44307
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
537 if isinstance(item, dict): |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
538 for sub_block in _walk_trie(item): |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
539 yield sub_block |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
540 yield block |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
541 |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
542 |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
543 def _persist_block(block_node, block_map): |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
544 """produce persistent binary data for a single block |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
545 |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
546 Children block are assumed to be already persisted and present in |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
547 block_map. |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
548 """ |
44316
55b12f2593c1
nodemap: move the iteratio inside the Block object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44315
diff
changeset
|
549 data = tuple(_to_value(v, block_map) for v in block_node) |
44307
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
550 return S_BLOCK.pack(*data) |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
551 |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
552 |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
553 def _to_value(item, block_map): |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
554 """persist any value as an integer""" |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
555 if item is None: |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
556 return NO_ENTRY |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
557 elif isinstance(item, dict): |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
558 return block_map[id(item)] |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
559 else: |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
560 return _transform_rev(item) |
44317
78721bbdb2ab
nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44316
diff
changeset
|
561 |
78721bbdb2ab
nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44316
diff
changeset
|
562 |
78721bbdb2ab
nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44316
diff
changeset
|
563 def parse_data(data): |
78721bbdb2ab
nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44316
diff
changeset
|
564 """parse parse nodemap data into a nodemap Trie""" |
78721bbdb2ab
nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44316
diff
changeset
|
565 if (len(data) % S_BLOCK.size) != 0: |
78721bbdb2ab
nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44316
diff
changeset
|
566 msg = "nodemap data size is not a multiple of block size (%d): %d" |
78721bbdb2ab
nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44316
diff
changeset
|
567 raise error.Abort(msg % (S_BLOCK.size, len(data))) |
78721bbdb2ab
nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44316
diff
changeset
|
568 if not data: |
44333
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
569 return Block(), None |
44317
78721bbdb2ab
nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44316
diff
changeset
|
570 block_map = {} |
78721bbdb2ab
nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44316
diff
changeset
|
571 new_blocks = [] |
78721bbdb2ab
nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44316
diff
changeset
|
572 for i in range(0, len(data), S_BLOCK.size): |
78721bbdb2ab
nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44316
diff
changeset
|
573 block = Block() |
44321
f0862ee1a31e
nodemap: keep track of the ondisk id of nodemap blocks
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44319
diff
changeset
|
574 block.ondisk_id = len(block_map) |
f0862ee1a31e
nodemap: keep track of the ondisk id of nodemap blocks
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44319
diff
changeset
|
575 block_map[block.ondisk_id] = block |
44317
78721bbdb2ab
nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44316
diff
changeset
|
576 block_data = data[i : i + S_BLOCK.size] |
78721bbdb2ab
nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44316
diff
changeset
|
577 values = S_BLOCK.unpack(block_data) |
78721bbdb2ab
nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44316
diff
changeset
|
578 new_blocks.append((block, values)) |
78721bbdb2ab
nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44316
diff
changeset
|
579 for b, values in new_blocks: |
78721bbdb2ab
nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44316
diff
changeset
|
580 for idx, v in enumerate(values): |
78721bbdb2ab
nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44316
diff
changeset
|
581 if v == NO_ENTRY: |
78721bbdb2ab
nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44316
diff
changeset
|
582 continue |
78721bbdb2ab
nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44316
diff
changeset
|
583 elif v >= 0: |
78721bbdb2ab
nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44316
diff
changeset
|
584 b[idx] = block_map[v] |
78721bbdb2ab
nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44316
diff
changeset
|
585 else: |
78721bbdb2ab
nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44316
diff
changeset
|
586 b[idx] = _transform_rev(v) |
44333
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
587 return block, i // S_BLOCK.size |
44318
20e125cdd719
nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44317
diff
changeset
|
588 |
20e125cdd719
nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44317
diff
changeset
|
589 |
20e125cdd719
nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44317
diff
changeset
|
590 # debug utility |
20e125cdd719
nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44317
diff
changeset
|
591 |
20e125cdd719
nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44317
diff
changeset
|
592 |
20e125cdd719
nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44317
diff
changeset
|
593 def check_data(ui, index, data): |
20e125cdd719
nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44317
diff
changeset
|
594 """verify that the provided nodemap data are valid for the given idex""" |
20e125cdd719
nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44317
diff
changeset
|
595 ret = 0 |
20e125cdd719
nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44317
diff
changeset
|
596 ui.status((b"revision in index: %d\n") % len(index)) |
44333
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
597 root, __ = parse_data(data) |
44318
20e125cdd719
nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44317
diff
changeset
|
598 all_revs = set(_all_revisions(root)) |
20e125cdd719
nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44317
diff
changeset
|
599 ui.status((b"revision in nodemap: %d\n") % len(all_revs)) |
20e125cdd719
nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44317
diff
changeset
|
600 for r in range(len(index)): |
20e125cdd719
nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44317
diff
changeset
|
601 if r not in all_revs: |
20e125cdd719
nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44317
diff
changeset
|
602 msg = b" revision missing from nodemap: %d\n" % r |
20e125cdd719
nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44317
diff
changeset
|
603 ui.write_err(msg) |
20e125cdd719
nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44317
diff
changeset
|
604 ret = 1 |
20e125cdd719
nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44317
diff
changeset
|
605 else: |
20e125cdd719
nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44317
diff
changeset
|
606 all_revs.remove(r) |
44319
d58206b70199
nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44318
diff
changeset
|
607 nm_rev = _find_node(root, nodemod.hex(index[r][7])) |
d58206b70199
nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44318
diff
changeset
|
608 if nm_rev is None: |
d58206b70199
nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44318
diff
changeset
|
609 msg = b" revision node does not match any entries: %d\n" % r |
d58206b70199
nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44318
diff
changeset
|
610 ui.write_err(msg) |
d58206b70199
nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44318
diff
changeset
|
611 ret = 1 |
d58206b70199
nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44318
diff
changeset
|
612 elif nm_rev != r: |
d58206b70199
nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44318
diff
changeset
|
613 msg = ( |
d58206b70199
nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44318
diff
changeset
|
614 b" revision node does not match the expected revision: " |
d58206b70199
nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44318
diff
changeset
|
615 b"%d != %d\n" % (r, nm_rev) |
d58206b70199
nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44318
diff
changeset
|
616 ) |
d58206b70199
nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44318
diff
changeset
|
617 ui.write_err(msg) |
d58206b70199
nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44318
diff
changeset
|
618 ret = 1 |
d58206b70199
nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44318
diff
changeset
|
619 |
44318
20e125cdd719
nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44317
diff
changeset
|
620 if all_revs: |
20e125cdd719
nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44317
diff
changeset
|
621 for r in sorted(all_revs): |
20e125cdd719
nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44317
diff
changeset
|
622 msg = b" extra revision in nodemap: %d\n" % r |
20e125cdd719
nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44317
diff
changeset
|
623 ui.write_err(msg) |
20e125cdd719
nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44317
diff
changeset
|
624 ret = 1 |
20e125cdd719
nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44317
diff
changeset
|
625 return ret |
20e125cdd719
nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44317
diff
changeset
|
626 |
20e125cdd719
nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44317
diff
changeset
|
627 |
20e125cdd719
nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44317
diff
changeset
|
628 def _all_revisions(root): |
20e125cdd719
nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44317
diff
changeset
|
629 """return all revisions stored in a Trie""" |
20e125cdd719
nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44317
diff
changeset
|
630 for block in _walk_trie(root): |
20e125cdd719
nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44317
diff
changeset
|
631 for v in block: |
20e125cdd719
nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44317
diff
changeset
|
632 if v is None or isinstance(v, Block): |
20e125cdd719
nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44317
diff
changeset
|
633 continue |
20e125cdd719
nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44317
diff
changeset
|
634 yield v |
44319
d58206b70199
nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44318
diff
changeset
|
635 |
d58206b70199
nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44318
diff
changeset
|
636 |
d58206b70199
nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44318
diff
changeset
|
637 def _find_node(block, node): |
d58206b70199
nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44318
diff
changeset
|
638 """find the revision associated with a given node""" |
d58206b70199
nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44318
diff
changeset
|
639 entry = block.get(_to_int(node[0:1])) |
d58206b70199
nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44318
diff
changeset
|
640 if isinstance(entry, dict): |
d58206b70199
nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44318
diff
changeset
|
641 return _find_node(entry, node[1:]) |
d58206b70199
nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44318
diff
changeset
|
642 return entry |