Mercurial > hg
annotate mercurial/revlogutils/nodemap.py @ 46470:dadb4db55661
nodemap: fix a typo in error message
Differential Revision: https://phab.mercurial-scm.org/D9934
author | Pulkit Goyal <7895pulkit@gmail.com> |
---|---|
date | Sun, 31 Jan 2021 23:40:57 +0530 |
parents | 835aafb2543f |
children | 98e39f04d60e |
rev | line source |
---|---|
44034
ab595920de0e
revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
1 # nodemap.py - nodemap related code and utilities |
ab595920de0e
revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
2 # |
ab595920de0e
revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
3 # Copyright 2019 Pierre-Yves David <pierre-yves.david@octobus.net> |
ab595920de0e
revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
4 # Copyright 2019 George Racinet <georges.racinet@octobus.net> |
ab595920de0e
revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
5 # |
ab595920de0e
revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
6 # This software may be used and distributed according to the terms of the |
ab595920de0e
revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
7 # GNU General Public License version 2 or any later version. |
ab595920de0e
revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
8 |
ab595920de0e
revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
9 from __future__ import absolute_import |
44307
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
10 |
44363
f7459da77f23
nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44340
diff
changeset
|
11 import errno |
44311
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
12 import os |
44312
563dfdfd01a4
nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44311
diff
changeset
|
13 import re |
44307
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
14 import struct |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
15 |
46113
59fa3890d40a
node: import symbols explicitly
Joerg Sonnenberger <joerg@bec.de>
parents:
46089
diff
changeset
|
16 from ..node import hex |
44788
6493f0a567c2
nodemap: add a new mode option, with an optional "warn" value
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44745
diff
changeset
|
17 |
44307
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
18 from .. import ( |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
19 error, |
44314
7f4f7ef3133e
nodemap: add a optional `nodemap_add_full` method on indexes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44312
diff
changeset
|
20 util, |
44307
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
21 ) |
44034
ab595920de0e
revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
22 |
ab595920de0e
revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
23 |
ab595920de0e
revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
24 class NodeMap(dict): |
ab595920de0e
revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
25 def __missing__(self, x): |
ab595920de0e
revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
26 raise error.RevlogError(b'unknown node: %s' % x) |
44307
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
27 |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
28 |
44309
6c07480d6659
nodemap: add a function to read the data from disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44308
diff
changeset
|
29 def persisted_data(revlog): |
6c07480d6659
nodemap: add a function to read the data from disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44308
diff
changeset
|
30 """read the nodemap for a revlog from disk""" |
6c07480d6659
nodemap: add a function to read the data from disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44308
diff
changeset
|
31 if revlog.nodemap_file is None: |
6c07480d6659
nodemap: add a function to read the data from disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44308
diff
changeset
|
32 return None |
44311
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
33 pdata = revlog.opener.tryread(revlog.nodemap_file) |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
34 if not pdata: |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
35 return None |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
36 offset = 0 |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
37 (version,) = S_VERSION.unpack(pdata[offset : offset + S_VERSION.size]) |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
38 if version != ONDISK_VERSION: |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
39 return None |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
40 offset += S_VERSION.size |
44335
e41a164db7a9
nodemap: track the maximum revision tracked in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44333
diff
changeset
|
41 headers = S_HEADER.unpack(pdata[offset : offset + S_HEADER.size]) |
44515
6c906eaedd0d
nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44497
diff
changeset
|
42 uid_size, tip_rev, data_length, data_unused, tip_node_size = headers |
44311
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
43 offset += S_HEADER.size |
44322
72c15641c8b4
nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44321
diff
changeset
|
44 docket = NodeMapDocket(pdata[offset : offset + uid_size]) |
44515
6c906eaedd0d
nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44497
diff
changeset
|
45 offset += uid_size |
44335
e41a164db7a9
nodemap: track the maximum revision tracked in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44333
diff
changeset
|
46 docket.tip_rev = tip_rev |
44515
6c906eaedd0d
nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44497
diff
changeset
|
47 docket.tip_node = pdata[offset : offset + tip_node_size] |
44336
8374b69aef75
nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44335
diff
changeset
|
48 docket.data_length = data_length |
8374b69aef75
nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44335
diff
changeset
|
49 docket.data_unused = data_unused |
44311
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
50 |
44322
72c15641c8b4
nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44321
diff
changeset
|
51 filename = _rawdata_filepath(revlog, docket) |
44792
5e3c718692bb
nodemap: drop the 'exp-' prefix for internal opener option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44790
diff
changeset
|
52 use_mmap = revlog.opener.options.get(b"persistent-nodemap.mmap") |
44363
f7459da77f23
nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44340
diff
changeset
|
53 try: |
f7459da77f23
nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44340
diff
changeset
|
54 with revlog.opener(filename) as fd: |
f7459da77f23
nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44340
diff
changeset
|
55 if use_mmap: |
f7459da77f23
nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44340
diff
changeset
|
56 data = util.buffer(util.mmapread(fd, data_length)) |
f7459da77f23
nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44340
diff
changeset
|
57 else: |
f7459da77f23
nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44340
diff
changeset
|
58 data = fd.read(data_length) |
46321
d32e7ed81f4f
persistent-nodemap: catch the right exception on python
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
46312
diff
changeset
|
59 except (IOError, OSError) as e: |
46089
8ff2d8359d0f
persistent-nodemap: properly ignore non-existent `.nd` data file
Simon Sapin <simon-commits@exyr.org>
parents:
45942
diff
changeset
|
60 if e.errno == errno.ENOENT: |
8ff2d8359d0f
persistent-nodemap: properly ignore non-existent `.nd` data file
Simon Sapin <simon-commits@exyr.org>
parents:
45942
diff
changeset
|
61 return None |
8ff2d8359d0f
persistent-nodemap: properly ignore non-existent `.nd` data file
Simon Sapin <simon-commits@exyr.org>
parents:
45942
diff
changeset
|
62 else: |
44363
f7459da77f23
nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44340
diff
changeset
|
63 raise |
44339
c7eebdb15139
nodemap: never read more than the expected data amount
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44338
diff
changeset
|
64 if len(data) < data_length: |
c7eebdb15139
nodemap: never read more than the expected data amount
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44338
diff
changeset
|
65 return None |
c7eebdb15139
nodemap: never read more than the expected data amount
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44338
diff
changeset
|
66 return docket, data |
44309
6c07480d6659
nodemap: add a function to read the data from disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44308
diff
changeset
|
67 |
6c07480d6659
nodemap: add a function to read the data from disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44308
diff
changeset
|
68 |
44308
5962fd0d1045
nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44307
diff
changeset
|
69 def setup_persistent_nodemap(tr, revlog): |
5962fd0d1045
nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44307
diff
changeset
|
70 """Install whatever is needed transaction side to persist a nodemap on disk |
5962fd0d1045
nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44307
diff
changeset
|
71 |
5962fd0d1045
nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44307
diff
changeset
|
72 (only actually persist the nodemap if this is relevant for this revlog) |
5962fd0d1045
nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44307
diff
changeset
|
73 """ |
44310
daad3aace942
nodemap: only use persistent nodemap for non-inlined revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44309
diff
changeset
|
74 if revlog._inline: |
daad3aace942
nodemap: only use persistent nodemap for non-inlined revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44309
diff
changeset
|
75 return # inlined revlog are too small for this to be relevant |
44308
5962fd0d1045
nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44307
diff
changeset
|
76 if revlog.nodemap_file is None: |
5962fd0d1045
nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44307
diff
changeset
|
77 return # we do not use persistent_nodemap on this revlog |
44517
448d700e0d27
nodemap: make sure the nodemap docket is updated after the changelog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44516
diff
changeset
|
78 |
448d700e0d27
nodemap: make sure the nodemap docket is updated after the changelog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44516
diff
changeset
|
79 # we need to happen after the changelog finalization, in that use "cl-" |
448d700e0d27
nodemap: make sure the nodemap docket is updated after the changelog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44516
diff
changeset
|
80 callback_id = b"nm-revlog-persistent-nodemap-%s" % revlog.nodemap_file |
44308
5962fd0d1045
nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44307
diff
changeset
|
81 if tr.hasfinalize(callback_id): |
5962fd0d1045
nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44307
diff
changeset
|
82 return # no need to register again |
44516
64e2f603de9d
nodemap: make sure hooks have access to an up-to-date version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44515
diff
changeset
|
83 tr.addpending( |
46468
83f037acf71a
nodemap: make `_persist_nodemap` a public function
Pulkit Goyal <7895pulkit@gmail.com>
parents:
46321
diff
changeset
|
84 callback_id, lambda tr: persist_nodemap(tr, revlog, pending=True) |
44516
64e2f603de9d
nodemap: make sure hooks have access to an up-to-date version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44515
diff
changeset
|
85 ) |
46468
83f037acf71a
nodemap: make `_persist_nodemap` a public function
Pulkit Goyal <7895pulkit@gmail.com>
parents:
46321
diff
changeset
|
86 tr.addfinalize(callback_id, lambda tr: persist_nodemap(tr, revlog)) |
44497
3265c92f7d13
nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44494
diff
changeset
|
87 |
3265c92f7d13
nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44494
diff
changeset
|
88 |
3265c92f7d13
nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44494
diff
changeset
|
89 class _NoTransaction(object): |
45942
89a2afe31e82
formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents:
44792
diff
changeset
|
90 """transaction like object to update the nodemap outside a transaction""" |
44497
3265c92f7d13
nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44494
diff
changeset
|
91 |
3265c92f7d13
nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44494
diff
changeset
|
92 def __init__(self): |
3265c92f7d13
nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44494
diff
changeset
|
93 self._postclose = {} |
3265c92f7d13
nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44494
diff
changeset
|
94 |
3265c92f7d13
nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44494
diff
changeset
|
95 def addpostclose(self, callback_id, callback_func): |
3265c92f7d13
nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44494
diff
changeset
|
96 self._postclose[callback_id] = callback_func |
44308
5962fd0d1045
nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44307
diff
changeset
|
97 |
44634
01b0805534bb
nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44517
diff
changeset
|
98 def registertmp(self, *args, **kwargs): |
01b0805534bb
nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44517
diff
changeset
|
99 pass |
01b0805534bb
nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44517
diff
changeset
|
100 |
01b0805534bb
nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44517
diff
changeset
|
101 def addbackup(self, *args, **kwargs): |
01b0805534bb
nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44517
diff
changeset
|
102 pass |
01b0805534bb
nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44517
diff
changeset
|
103 |
01b0805534bb
nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44517
diff
changeset
|
104 def add(self, *args, **kwargs): |
01b0805534bb
nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44517
diff
changeset
|
105 pass |
01b0805534bb
nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44517
diff
changeset
|
106 |
44635
99ea74cbed74
nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44634
diff
changeset
|
107 def addabort(self, *args, **kwargs): |
99ea74cbed74
nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44634
diff
changeset
|
108 pass |
99ea74cbed74
nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44634
diff
changeset
|
109 |
44788
6493f0a567c2
nodemap: add a new mode option, with an optional "warn" value
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44745
diff
changeset
|
110 def _report(self, *args): |
6493f0a567c2
nodemap: add a new mode option, with an optional "warn" value
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44745
diff
changeset
|
111 pass |
6493f0a567c2
nodemap: add a new mode option, with an optional "warn" value
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44745
diff
changeset
|
112 |
44308
5962fd0d1045
nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44307
diff
changeset
|
113 |
44445
336ec75ed1ac
nodemap: warm the persistent nodemap on disk with debugupdatecache
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44363
diff
changeset
|
114 def update_persistent_nodemap(revlog): |
336ec75ed1ac
nodemap: warm the persistent nodemap on disk with debugupdatecache
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44363
diff
changeset
|
115 """update the persistent nodemap right now |
336ec75ed1ac
nodemap: warm the persistent nodemap on disk with debugupdatecache
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44363
diff
changeset
|
116 |
336ec75ed1ac
nodemap: warm the persistent nodemap on disk with debugupdatecache
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44363
diff
changeset
|
117 To be used for updating the nodemap on disk outside of a normal transaction |
336ec75ed1ac
nodemap: warm the persistent nodemap on disk with debugupdatecache
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44363
diff
changeset
|
118 setup (eg, `debugupdatecache`). |
336ec75ed1ac
nodemap: warm the persistent nodemap on disk with debugupdatecache
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44363
diff
changeset
|
119 """ |
44745
b4537125eb3c
nodemap: skip persistent nodemap warming for revlog not using it
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44697
diff
changeset
|
120 if revlog._inline: |
b4537125eb3c
nodemap: skip persistent nodemap warming for revlog not using it
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44697
diff
changeset
|
121 return # inlined revlog are too small for this to be relevant |
b4537125eb3c
nodemap: skip persistent nodemap warming for revlog not using it
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44697
diff
changeset
|
122 if revlog.nodemap_file is None: |
b4537125eb3c
nodemap: skip persistent nodemap warming for revlog not using it
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44697
diff
changeset
|
123 return # we do not use persistent_nodemap on this revlog |
b4537125eb3c
nodemap: skip persistent nodemap warming for revlog not using it
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44697
diff
changeset
|
124 |
44497
3265c92f7d13
nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44494
diff
changeset
|
125 notr = _NoTransaction() |
46468
83f037acf71a
nodemap: make `_persist_nodemap` a public function
Pulkit Goyal <7895pulkit@gmail.com>
parents:
46321
diff
changeset
|
126 persist_nodemap(notr, revlog) |
44497
3265c92f7d13
nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44494
diff
changeset
|
127 for k in sorted(notr._postclose): |
3265c92f7d13
nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44494
diff
changeset
|
128 notr._postclose[k](None) |
44445
336ec75ed1ac
nodemap: warm the persistent nodemap on disk with debugupdatecache
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44363
diff
changeset
|
129 |
336ec75ed1ac
nodemap: warm the persistent nodemap on disk with debugupdatecache
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44363
diff
changeset
|
130 |
46468
83f037acf71a
nodemap: make `_persist_nodemap` a public function
Pulkit Goyal <7895pulkit@gmail.com>
parents:
46321
diff
changeset
|
131 def persist_nodemap(tr, revlog, pending=False): |
45942
89a2afe31e82
formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents:
44792
diff
changeset
|
132 """Write nodemap data on disk for a given revlog""" |
44308
5962fd0d1045
nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44307
diff
changeset
|
133 if getattr(revlog, 'filteredrevs', ()): |
5962fd0d1045
nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44307
diff
changeset
|
134 raise error.ProgrammingError( |
5962fd0d1045
nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44307
diff
changeset
|
135 "cannot persist nodemap of a filtered changelog" |
5962fd0d1045
nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44307
diff
changeset
|
136 ) |
5962fd0d1045
nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44307
diff
changeset
|
137 if revlog.nodemap_file is None: |
46470
dadb4db55661
nodemap: fix a typo in error message
Pulkit Goyal <7895pulkit@gmail.com>
parents:
46469
diff
changeset
|
138 msg = "calling persist nodemap on a revlog without the feature enabled" |
44308
5962fd0d1045
nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44307
diff
changeset
|
139 raise error.ProgrammingError(msg) |
44333
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
140 |
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
141 can_incremental = util.safehasattr(revlog.index, "nodemap_data_incremental") |
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
142 ondisk_docket = revlog._nodemap_docket |
44363
f7459da77f23
nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44340
diff
changeset
|
143 feed_data = util.safehasattr(revlog.index, "update_nodemap_data") |
44792
5e3c718692bb
nodemap: drop the 'exp-' prefix for internal opener option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44790
diff
changeset
|
144 use_mmap = revlog.opener.options.get(b"persistent-nodemap.mmap") |
44333
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
145 |
44337
1d2b37def017
nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44336
diff
changeset
|
146 data = None |
44333
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
147 # first attemp an incremental update of the data |
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
148 if can_incremental and ondisk_docket is not None: |
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
149 target_docket = revlog._nodemap_docket.copy() |
44337
1d2b37def017
nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44336
diff
changeset
|
150 ( |
1d2b37def017
nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44336
diff
changeset
|
151 src_docket, |
1d2b37def017
nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44336
diff
changeset
|
152 data_changed_count, |
1d2b37def017
nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44336
diff
changeset
|
153 data, |
1d2b37def017
nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44336
diff
changeset
|
154 ) = revlog.index.nodemap_data_incremental() |
44636
c70bcaf7927b
nodemap: automatically "vacuum" the persistent nodemap when too sparse
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44635
diff
changeset
|
155 new_length = target_docket.data_length + len(data) |
c70bcaf7927b
nodemap: automatically "vacuum" the persistent nodemap when too sparse
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44635
diff
changeset
|
156 new_unused = target_docket.data_unused + data_changed_count |
44337
1d2b37def017
nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44336
diff
changeset
|
157 if src_docket != target_docket: |
1d2b37def017
nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44336
diff
changeset
|
158 data = None |
44636
c70bcaf7927b
nodemap: automatically "vacuum" the persistent nodemap when too sparse
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44635
diff
changeset
|
159 elif new_length <= (new_unused * 10): # under 10% of unused data |
c70bcaf7927b
nodemap: automatically "vacuum" the persistent nodemap when too sparse
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44635
diff
changeset
|
160 data = None |
44337
1d2b37def017
nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44336
diff
changeset
|
161 else: |
1d2b37def017
nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44336
diff
changeset
|
162 datafile = _rawdata_filepath(revlog, target_docket) |
1d2b37def017
nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44336
diff
changeset
|
163 # EXP-TODO: if this is a cache, this should use a cache vfs, not a |
1d2b37def017
nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44336
diff
changeset
|
164 # store vfs |
44634
01b0805534bb
nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44517
diff
changeset
|
165 tr.add(datafile, target_docket.data_length) |
44338
2ea6a67ff502
nodemap: write new data from the expected current data length
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44337
diff
changeset
|
166 with revlog.opener(datafile, b'r+') as fd: |
2ea6a67ff502
nodemap: write new data from the expected current data length
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44337
diff
changeset
|
167 fd.seek(target_docket.data_length) |
44337
1d2b37def017
nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44336
diff
changeset
|
168 fd.write(data) |
44363
f7459da77f23
nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44340
diff
changeset
|
169 if feed_data: |
f7459da77f23
nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44340
diff
changeset
|
170 if use_mmap: |
f7459da77f23
nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44340
diff
changeset
|
171 fd.seek(0) |
f7459da77f23
nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44340
diff
changeset
|
172 new_data = fd.read(new_length) |
f7459da77f23
nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44340
diff
changeset
|
173 else: |
f7459da77f23
nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44340
diff
changeset
|
174 fd.flush() |
f7459da77f23
nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44340
diff
changeset
|
175 new_data = util.buffer(util.mmapread(fd, new_length)) |
f7459da77f23
nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44340
diff
changeset
|
176 target_docket.data_length = new_length |
44636
c70bcaf7927b
nodemap: automatically "vacuum" the persistent nodemap when too sparse
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44635
diff
changeset
|
177 target_docket.data_unused = new_unused |
44337
1d2b37def017
nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44336
diff
changeset
|
178 |
1d2b37def017
nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44336
diff
changeset
|
179 if data is None: |
44333
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
180 # otherwise fallback to a full new export |
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
181 target_docket = NodeMapDocket() |
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
182 datafile = _rawdata_filepath(revlog, target_docket) |
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
183 if util.safehasattr(revlog.index, "nodemap_data_all"): |
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
184 data = revlog.index.nodemap_data_all() |
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
185 else: |
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
186 data = persistent_data(revlog.index) |
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
187 # EXP-TODO: if this is a cache, this should use a cache vfs, not a |
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
188 # store vfs |
44635
99ea74cbed74
nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44634
diff
changeset
|
189 |
99ea74cbed74
nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44634
diff
changeset
|
190 tryunlink = revlog.opener.tryunlink |
99ea74cbed74
nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44634
diff
changeset
|
191 |
99ea74cbed74
nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44634
diff
changeset
|
192 def abortck(tr): |
99ea74cbed74
nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44634
diff
changeset
|
193 tryunlink(datafile) |
99ea74cbed74
nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44634
diff
changeset
|
194 |
99ea74cbed74
nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44634
diff
changeset
|
195 callback_id = b"delete-%s" % datafile |
99ea74cbed74
nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44634
diff
changeset
|
196 |
99ea74cbed74
nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44634
diff
changeset
|
197 # some flavor of the transaction abort does not cleanup new file, it |
99ea74cbed74
nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44634
diff
changeset
|
198 # simply empty them. |
99ea74cbed74
nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44634
diff
changeset
|
199 tr.addabort(callback_id, abortck) |
44363
f7459da77f23
nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44340
diff
changeset
|
200 with revlog.opener(datafile, b'w+') as fd: |
44333
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
201 fd.write(data) |
44363
f7459da77f23
nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44340
diff
changeset
|
202 if feed_data: |
f7459da77f23
nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44340
diff
changeset
|
203 if use_mmap: |
f7459da77f23
nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44340
diff
changeset
|
204 new_data = data |
f7459da77f23
nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44340
diff
changeset
|
205 else: |
f7459da77f23
nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44340
diff
changeset
|
206 fd.flush() |
f7459da77f23
nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44340
diff
changeset
|
207 new_data = util.buffer(util.mmapread(fd, len(data))) |
44336
8374b69aef75
nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44335
diff
changeset
|
208 target_docket.data_length = len(data) |
44335
e41a164db7a9
nodemap: track the maximum revision tracked in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44333
diff
changeset
|
209 target_docket.tip_rev = revlog.tiprev() |
44515
6c906eaedd0d
nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44497
diff
changeset
|
210 target_docket.tip_node = revlog.node(target_docket.tip_rev) |
44333
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
211 # EXP-TODO: if this is a cache, this should use a cache vfs, not a |
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
212 # store vfs |
44516
64e2f603de9d
nodemap: make sure hooks have access to an up-to-date version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44515
diff
changeset
|
213 file_path = revlog.nodemap_file |
64e2f603de9d
nodemap: make sure hooks have access to an up-to-date version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44515
diff
changeset
|
214 if pending: |
64e2f603de9d
nodemap: make sure hooks have access to an up-to-date version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44515
diff
changeset
|
215 file_path += b'.a' |
44634
01b0805534bb
nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44517
diff
changeset
|
216 tr.registertmp(file_path) |
01b0805534bb
nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44517
diff
changeset
|
217 else: |
01b0805534bb
nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44517
diff
changeset
|
218 tr.addbackup(file_path) |
01b0805534bb
nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44517
diff
changeset
|
219 |
44516
64e2f603de9d
nodemap: make sure hooks have access to an up-to-date version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44515
diff
changeset
|
220 with revlog.opener(file_path, b'w', atomictemp=True) as fp: |
44333
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
221 fp.write(target_docket.serialize()) |
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
222 revlog._nodemap_docket = target_docket |
44363
f7459da77f23
nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44340
diff
changeset
|
223 if feed_data: |
44340
6ecc34b31137
nodemap: update the index with the newly written data (when appropriate)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44339
diff
changeset
|
224 revlog.index.update_nodemap_data(target_docket, new_data) |
6ecc34b31137
nodemap: update the index with the newly written data (when appropriate)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44339
diff
changeset
|
225 |
44333
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
226 # search for old index file in all cases, some older process might have |
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
227 # left one behind. |
44322
72c15641c8b4
nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44321
diff
changeset
|
228 olds = _other_rawdata_filepath(revlog, target_docket) |
44312
563dfdfd01a4
nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44311
diff
changeset
|
229 if olds: |
563dfdfd01a4
nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44311
diff
changeset
|
230 realvfs = getattr(revlog, '_realopener', revlog.opener) |
563dfdfd01a4
nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44311
diff
changeset
|
231 |
563dfdfd01a4
nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44311
diff
changeset
|
232 def cleanup(tr): |
563dfdfd01a4
nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44311
diff
changeset
|
233 for oldfile in olds: |
563dfdfd01a4
nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44311
diff
changeset
|
234 realvfs.tryunlink(oldfile) |
563dfdfd01a4
nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44311
diff
changeset
|
235 |
563dfdfd01a4
nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44311
diff
changeset
|
236 callback_id = b"revlog-cleanup-nodemap-%s" % revlog.nodemap_file |
44497
3265c92f7d13
nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44494
diff
changeset
|
237 tr.addpostclose(callback_id, cleanup) |
44311
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
238 |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
239 |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
240 ### Nodemap docket file |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
241 # |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
242 # The nodemap data are stored on disk using 2 files: |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
243 # |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
244 # * a raw data files containing a persistent nodemap |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
245 # (see `Nodemap Trie` section) |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
246 # |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
247 # * a small "docket" file containing medatadata |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
248 # |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
249 # While the nodemap data can be multiple tens of megabytes, the "docket" is |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
250 # small, it is easy to update it automatically or to duplicated its content |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
251 # during a transaction. |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
252 # |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
253 # Multiple raw data can exist at the same time (The currently valid one and a |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
254 # new one beind used by an in progress transaction). To accomodate this, the |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
255 # filename hosting the raw data has a variable parts. The exact filename is |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
256 # specified inside the "docket" file. |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
257 # |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
258 # The docket file contains information to find, qualify and validate the raw |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
259 # data. Its content is currently very light, but it will expand as the on disk |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
260 # nodemap gains the necessary features to be used in production. |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
261 |
44790
261e71752d1f
nodemap: move on disk file to version 1
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44789
diff
changeset
|
262 ONDISK_VERSION = 1 |
44311
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
263 S_VERSION = struct.Struct(">B") |
44515
6c906eaedd0d
nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44497
diff
changeset
|
264 S_HEADER = struct.Struct(">BQQQQ") |
44311
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
265 |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
266 ID_SIZE = 8 |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
267 |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
268 |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
269 def _make_uid(): |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
270 """return a new unique identifier. |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
271 |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
272 The identifier is random and composed of ascii characters.""" |
46113
59fa3890d40a
node: import symbols explicitly
Joerg Sonnenberger <joerg@bec.de>
parents:
46089
diff
changeset
|
273 return hex(os.urandom(ID_SIZE)) |
44311
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
274 |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
275 |
44322
72c15641c8b4
nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44321
diff
changeset
|
276 class NodeMapDocket(object): |
72c15641c8b4
nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44321
diff
changeset
|
277 """metadata associated with persistent nodemap data |
72c15641c8b4
nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44321
diff
changeset
|
278 |
72c15641c8b4
nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44321
diff
changeset
|
279 The persistent data may come from disk or be on their way to disk. |
72c15641c8b4
nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44321
diff
changeset
|
280 """ |
72c15641c8b4
nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44321
diff
changeset
|
281 |
72c15641c8b4
nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44321
diff
changeset
|
282 def __init__(self, uid=None): |
72c15641c8b4
nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44321
diff
changeset
|
283 if uid is None: |
72c15641c8b4
nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44321
diff
changeset
|
284 uid = _make_uid() |
44494
283fd803afa5
nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44462
diff
changeset
|
285 # a unique identifier for the data file: |
283fd803afa5
nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44462
diff
changeset
|
286 # - When new data are appended, it is preserved. |
283fd803afa5
nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44462
diff
changeset
|
287 # - When a new data file is created, a new identifier is generated. |
44322
72c15641c8b4
nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44321
diff
changeset
|
288 self.uid = uid |
44494
283fd803afa5
nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44462
diff
changeset
|
289 # the tipmost revision stored in the data file. This revision and all |
283fd803afa5
nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44462
diff
changeset
|
290 # revision before it are expected to be encoded in the data file. |
44335
e41a164db7a9
nodemap: track the maximum revision tracked in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44333
diff
changeset
|
291 self.tip_rev = None |
44515
6c906eaedd0d
nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44497
diff
changeset
|
292 # the node of that tipmost revision, if it mismatch the current index |
6c906eaedd0d
nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44497
diff
changeset
|
293 # data the docket is not valid for the current index and should be |
6c906eaedd0d
nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44497
diff
changeset
|
294 # discarded. |
6c906eaedd0d
nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44497
diff
changeset
|
295 # |
6c906eaedd0d
nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44497
diff
changeset
|
296 # note: this method is not perfect as some destructive operation could |
6c906eaedd0d
nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44497
diff
changeset
|
297 # preserve the same tip_rev + tip_node while altering lower revision. |
6c906eaedd0d
nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44497
diff
changeset
|
298 # However this multiple other caches have the same vulnerability (eg: |
6c906eaedd0d
nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44497
diff
changeset
|
299 # brancmap cache). |
6c906eaedd0d
nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44497
diff
changeset
|
300 self.tip_node = None |
44494
283fd803afa5
nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44462
diff
changeset
|
301 # the size (in bytes) of the persisted data to encode the nodemap valid |
283fd803afa5
nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44462
diff
changeset
|
302 # for `tip_rev`. |
283fd803afa5
nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44462
diff
changeset
|
303 # - data file shorter than this are corrupted, |
283fd803afa5
nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44462
diff
changeset
|
304 # - any extra data should be ignored. |
44336
8374b69aef75
nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44335
diff
changeset
|
305 self.data_length = None |
44494
283fd803afa5
nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44462
diff
changeset
|
306 # the amount (in bytes) of "dead" data, still in the data file but no |
283fd803afa5
nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44462
diff
changeset
|
307 # longer used for the nodemap. |
44336
8374b69aef75
nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44335
diff
changeset
|
308 self.data_unused = 0 |
44322
72c15641c8b4
nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44321
diff
changeset
|
309 |
72c15641c8b4
nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44321
diff
changeset
|
310 def copy(self): |
44335
e41a164db7a9
nodemap: track the maximum revision tracked in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44333
diff
changeset
|
311 new = NodeMapDocket(uid=self.uid) |
e41a164db7a9
nodemap: track the maximum revision tracked in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44333
diff
changeset
|
312 new.tip_rev = self.tip_rev |
44515
6c906eaedd0d
nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44497
diff
changeset
|
313 new.tip_node = self.tip_node |
44336
8374b69aef75
nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44335
diff
changeset
|
314 new.data_length = self.data_length |
8374b69aef75
nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44335
diff
changeset
|
315 new.data_unused = self.data_unused |
44335
e41a164db7a9
nodemap: track the maximum revision tracked in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44333
diff
changeset
|
316 return new |
44322
72c15641c8b4
nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44321
diff
changeset
|
317 |
44337
1d2b37def017
nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44336
diff
changeset
|
318 def __cmp__(self, other): |
1d2b37def017
nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44336
diff
changeset
|
319 if self.uid < other.uid: |
1d2b37def017
nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44336
diff
changeset
|
320 return -1 |
1d2b37def017
nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44336
diff
changeset
|
321 if self.uid > other.uid: |
1d2b37def017
nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44336
diff
changeset
|
322 return 1 |
1d2b37def017
nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44336
diff
changeset
|
323 elif self.data_length < other.data_length: |
1d2b37def017
nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44336
diff
changeset
|
324 return -1 |
1d2b37def017
nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44336
diff
changeset
|
325 elif self.data_length > other.data_length: |
1d2b37def017
nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44336
diff
changeset
|
326 return 1 |
1d2b37def017
nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44336
diff
changeset
|
327 return 0 |
1d2b37def017
nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44336
diff
changeset
|
328 |
1d2b37def017
nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44336
diff
changeset
|
329 def __eq__(self, other): |
1d2b37def017
nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44336
diff
changeset
|
330 return self.uid == other.uid and self.data_length == other.data_length |
1d2b37def017
nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44336
diff
changeset
|
331 |
44322
72c15641c8b4
nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44321
diff
changeset
|
332 def serialize(self): |
72c15641c8b4
nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44321
diff
changeset
|
333 """return serialized bytes for a docket using the passed uid""" |
72c15641c8b4
nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44321
diff
changeset
|
334 data = [] |
72c15641c8b4
nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44321
diff
changeset
|
335 data.append(S_VERSION.pack(ONDISK_VERSION)) |
44336
8374b69aef75
nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44335
diff
changeset
|
336 headers = ( |
8374b69aef75
nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44335
diff
changeset
|
337 len(self.uid), |
8374b69aef75
nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44335
diff
changeset
|
338 self.tip_rev, |
8374b69aef75
nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44335
diff
changeset
|
339 self.data_length, |
8374b69aef75
nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44335
diff
changeset
|
340 self.data_unused, |
44515
6c906eaedd0d
nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44497
diff
changeset
|
341 len(self.tip_node), |
44336
8374b69aef75
nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44335
diff
changeset
|
342 ) |
44335
e41a164db7a9
nodemap: track the maximum revision tracked in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44333
diff
changeset
|
343 data.append(S_HEADER.pack(*headers)) |
44322
72c15641c8b4
nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44321
diff
changeset
|
344 data.append(self.uid) |
44515
6c906eaedd0d
nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44497
diff
changeset
|
345 data.append(self.tip_node) |
44322
72c15641c8b4
nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44321
diff
changeset
|
346 return b''.join(data) |
44311
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
347 |
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
348 |
44322
72c15641c8b4
nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44321
diff
changeset
|
349 def _rawdata_filepath(revlog, docket): |
44311
2b72c4ff8ed1
nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44310
diff
changeset
|
350 """The (vfs relative) nodemap's rawdata file for a given uid""" |
44516
64e2f603de9d
nodemap: make sure hooks have access to an up-to-date version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44515
diff
changeset
|
351 if revlog.nodemap_file.endswith(b'.n.a'): |
64e2f603de9d
nodemap: make sure hooks have access to an up-to-date version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44515
diff
changeset
|
352 prefix = revlog.nodemap_file[:-4] |
64e2f603de9d
nodemap: make sure hooks have access to an up-to-date version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44515
diff
changeset
|
353 else: |
64e2f603de9d
nodemap: make sure hooks have access to an up-to-date version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44515
diff
changeset
|
354 prefix = revlog.nodemap_file[:-2] |
44322
72c15641c8b4
nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44321
diff
changeset
|
355 return b"%s-%s.nd" % (prefix, docket.uid) |
44308
5962fd0d1045
nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44307
diff
changeset
|
356 |
5962fd0d1045
nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44307
diff
changeset
|
357 |
44322
72c15641c8b4
nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44321
diff
changeset
|
358 def _other_rawdata_filepath(revlog, docket): |
44312
563dfdfd01a4
nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44311
diff
changeset
|
359 prefix = revlog.nodemap_file[:-2] |
44462
6aee0647e026
nodemap: fix missing r-prefix on regular expression
Augie Fackler <augie@google.com>
parents:
44445
diff
changeset
|
360 pattern = re.compile(br"(^|/)%s-[0-9a-f]+\.nd$" % prefix) |
44322
72c15641c8b4
nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44321
diff
changeset
|
361 new_file_path = _rawdata_filepath(revlog, docket) |
44312
563dfdfd01a4
nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44311
diff
changeset
|
362 new_file_name = revlog.opener.basename(new_file_path) |
563dfdfd01a4
nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44311
diff
changeset
|
363 dirpath = revlog.opener.dirname(new_file_path) |
563dfdfd01a4
nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44311
diff
changeset
|
364 others = [] |
563dfdfd01a4
nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44311
diff
changeset
|
365 for f in revlog.opener.listdir(dirpath): |
563dfdfd01a4
nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44311
diff
changeset
|
366 if pattern.match(f) and f != new_file_name: |
563dfdfd01a4
nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44311
diff
changeset
|
367 others.append(f) |
563dfdfd01a4
nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44311
diff
changeset
|
368 return others |
563dfdfd01a4
nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44311
diff
changeset
|
369 |
563dfdfd01a4
nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44311
diff
changeset
|
370 |
44307
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
371 ### Nodemap Trie |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
372 # |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
373 # This is a simple reference implementation to compute and persist a nodemap |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
374 # trie. This reference implementation is write only. The python version of this |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
375 # is not expected to be actually used, since it wont provide performance |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
376 # improvement over existing non-persistent C implementation. |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
377 # |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
378 # The nodemap is persisted as Trie using 4bits-address/16-entries block. each |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
379 # revision can be adressed using its node shortest prefix. |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
380 # |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
381 # The trie is stored as a sequence of block. Each block contains 16 entries |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
382 # (signed 64bit integer, big endian). Each entry can be one of the following: |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
383 # |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
384 # * value >= 0 -> index of sub-block |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
385 # * value == -1 -> no value |
46199
8e7ce6555ea7
nodemap: match comment to actual code
Joerg Sonnenberger <joerg@bec.de>
parents:
46113
diff
changeset
|
386 # * value < -1 -> encoded revision: rev = -(value+2) |
8e7ce6555ea7
nodemap: match comment to actual code
Joerg Sonnenberger <joerg@bec.de>
parents:
46113
diff
changeset
|
387 # |
8e7ce6555ea7
nodemap: match comment to actual code
Joerg Sonnenberger <joerg@bec.de>
parents:
46113
diff
changeset
|
388 # See REV_OFFSET and _transform_rev below. |
44307
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
389 # |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
390 # The implementation focus on simplicity, not on performance. A Rust |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
391 # implementation should provide a efficient version of the same binary |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
392 # persistence. This reference python implementation is never meant to be |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
393 # extensively use in production. |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
394 |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
395 |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
396 def persistent_data(index): |
45942
89a2afe31e82
formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents:
44792
diff
changeset
|
397 """return the persistent binary form for a nodemap for a given index""" |
44307
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
398 trie = _build_trie(index) |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
399 return _persist_trie(trie) |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
400 |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
401 |
44333
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
402 def update_persistent_data(index, root, max_idx, last_rev): |
45942
89a2afe31e82
formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents:
44792
diff
changeset
|
403 """return the incremental update for persistent nodemap from a given index""" |
44336
8374b69aef75
nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44335
diff
changeset
|
404 changed_block, trie = _update_trie(index, root, last_rev) |
8374b69aef75
nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44335
diff
changeset
|
405 return ( |
8374b69aef75
nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44335
diff
changeset
|
406 changed_block * S_BLOCK.size, |
8374b69aef75
nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44335
diff
changeset
|
407 _persist_trie(trie, existing_idx=max_idx), |
8374b69aef75
nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44335
diff
changeset
|
408 ) |
44333
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
409 |
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
410 |
44307
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
411 S_BLOCK = struct.Struct(">" + ("l" * 16)) |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
412 |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
413 NO_ENTRY = -1 |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
414 # rev 0 need to be -2 because 0 is used by block, -1 is a special value. |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
415 REV_OFFSET = 2 |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
416 |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
417 |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
418 def _transform_rev(rev): |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
419 """Return the number used to represent the rev in the tree. |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
420 |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
421 (or retrieve a rev number from such representation) |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
422 |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
423 Note that this is an involution, a function equal to its inverse (i.e. |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
424 which gives the identity when applied to itself). |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
425 """ |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
426 return -(rev + REV_OFFSET) |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
427 |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
428 |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
429 def _to_int(hex_digit): |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
430 """turn an hexadecimal digit into a proper integer""" |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
431 return int(hex_digit, 16) |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
432 |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
433 |
44315
7762a295fd4d
nodemap: use an explicit "Block" object in the reference implementation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44314
diff
changeset
|
434 class Block(dict): |
7762a295fd4d
nodemap: use an explicit "Block" object in the reference implementation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44314
diff
changeset
|
435 """represent a block of the Trie |
7762a295fd4d
nodemap: use an explicit "Block" object in the reference implementation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44314
diff
changeset
|
436 |
7762a295fd4d
nodemap: use an explicit "Block" object in the reference implementation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44314
diff
changeset
|
437 contains up to 16 entry indexed from 0 to 15""" |
7762a295fd4d
nodemap: use an explicit "Block" object in the reference implementation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44314
diff
changeset
|
438 |
44321
f0862ee1a31e
nodemap: keep track of the ondisk id of nodemap blocks
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44319
diff
changeset
|
439 def __init__(self): |
f0862ee1a31e
nodemap: keep track of the ondisk id of nodemap blocks
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44319
diff
changeset
|
440 super(Block, self).__init__() |
f0862ee1a31e
nodemap: keep track of the ondisk id of nodemap blocks
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44319
diff
changeset
|
441 # If this block exist on disk, here is its ID |
f0862ee1a31e
nodemap: keep track of the ondisk id of nodemap blocks
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44319
diff
changeset
|
442 self.ondisk_id = None |
f0862ee1a31e
nodemap: keep track of the ondisk id of nodemap blocks
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44319
diff
changeset
|
443 |
44316
55b12f2593c1
nodemap: move the iteratio inside the Block object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44315
diff
changeset
|
444 def __iter__(self): |
55b12f2593c1
nodemap: move the iteratio inside the Block object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44315
diff
changeset
|
445 return iter(self.get(i) for i in range(16)) |
55b12f2593c1
nodemap: move the iteratio inside the Block object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44315
diff
changeset
|
446 |
44315
7762a295fd4d
nodemap: use an explicit "Block" object in the reference implementation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44314
diff
changeset
|
447 |
44307
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
448 def _build_trie(index): |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
449 """build a nodemap trie |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
450 |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
451 The nodemap stores revision number for each unique prefix. |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
452 |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
453 Each block is a dictionary with keys in `[0, 15]`. Values are either |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
454 another block or a revision number. |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
455 """ |
44315
7762a295fd4d
nodemap: use an explicit "Block" object in the reference implementation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44314
diff
changeset
|
456 root = Block() |
44307
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
457 for rev in range(len(index)): |
46113
59fa3890d40a
node: import symbols explicitly
Joerg Sonnenberger <joerg@bec.de>
parents:
46089
diff
changeset
|
458 current_hex = hex(index[rev][7]) |
59fa3890d40a
node: import symbols explicitly
Joerg Sonnenberger <joerg@bec.de>
parents:
46089
diff
changeset
|
459 _insert_into_block(index, 0, root, rev, current_hex) |
44307
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
460 return root |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
461 |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
462 |
44333
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
463 def _update_trie(index, root, last_rev): |
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
464 """consume""" |
44336
8374b69aef75
nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44335
diff
changeset
|
465 changed = 0 |
44333
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
466 for rev in range(last_rev + 1, len(index)): |
46113
59fa3890d40a
node: import symbols explicitly
Joerg Sonnenberger <joerg@bec.de>
parents:
46089
diff
changeset
|
467 current_hex = hex(index[rev][7]) |
59fa3890d40a
node: import symbols explicitly
Joerg Sonnenberger <joerg@bec.de>
parents:
46089
diff
changeset
|
468 changed += _insert_into_block(index, 0, root, rev, current_hex) |
44336
8374b69aef75
nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44335
diff
changeset
|
469 return changed, root |
44333
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
470 |
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
471 |
44307
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
472 def _insert_into_block(index, level, block, current_rev, current_hex): |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
473 """insert a new revision in a block |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
474 |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
475 index: the index we are adding revision for |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
476 level: the depth of the current block in the trie |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
477 block: the block currently being considered |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
478 current_rev: the revision number we are adding |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
479 current_hex: the hexadecimal representation of the of that revision |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
480 """ |
44336
8374b69aef75
nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44335
diff
changeset
|
481 changed = 1 |
44333
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
482 if block.ondisk_id is not None: |
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
483 block.ondisk_id = None |
44307
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
484 hex_digit = _to_int(current_hex[level : level + 1]) |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
485 entry = block.get(hex_digit) |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
486 if entry is None: |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
487 # no entry, simply store the revision number |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
488 block[hex_digit] = current_rev |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
489 elif isinstance(entry, dict): |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
490 # need to recurse to an underlying block |
44336
8374b69aef75
nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44335
diff
changeset
|
491 changed += _insert_into_block( |
8374b69aef75
nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44335
diff
changeset
|
492 index, level + 1, entry, current_rev, current_hex |
8374b69aef75
nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44335
diff
changeset
|
493 ) |
44307
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
494 else: |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
495 # collision with a previously unique prefix, inserting new |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
496 # vertices to fit both entry. |
46113
59fa3890d40a
node: import symbols explicitly
Joerg Sonnenberger <joerg@bec.de>
parents:
46089
diff
changeset
|
497 other_hex = hex(index[entry][7]) |
44307
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
498 other_rev = entry |
44315
7762a295fd4d
nodemap: use an explicit "Block" object in the reference implementation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44314
diff
changeset
|
499 new = Block() |
44307
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
500 block[hex_digit] = new |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
501 _insert_into_block(index, level + 1, new, other_rev, other_hex) |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
502 _insert_into_block(index, level + 1, new, current_rev, current_hex) |
44336
8374b69aef75
nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44335
diff
changeset
|
503 return changed |
44307
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
504 |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
505 |
44333
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
506 def _persist_trie(root, existing_idx=None): |
44307
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
507 """turn a nodemap trie into persistent binary data |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
508 |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
509 See `_build_trie` for nodemap trie structure""" |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
510 block_map = {} |
44333
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
511 if existing_idx is not None: |
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
512 base_idx = existing_idx + 1 |
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
513 else: |
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
514 base_idx = 0 |
44307
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
515 chunks = [] |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
516 for tn in _walk_trie(root): |
44333
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
517 if tn.ondisk_id is not None: |
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
518 block_map[id(tn)] = tn.ondisk_id |
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
519 else: |
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
520 block_map[id(tn)] = len(chunks) + base_idx |
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
521 chunks.append(_persist_block(tn, block_map)) |
44307
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
522 return b''.join(chunks) |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
523 |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
524 |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
525 def _walk_trie(block): |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
526 """yield all the block in a trie |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
527 |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
528 Children blocks are always yield before their parent block. |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
529 """ |
44696
ed475420e0af
nodemap: rename a variable to prevent shawoding "_"
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44636
diff
changeset
|
530 for (__, item) in sorted(block.items()): |
44307
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
531 if isinstance(item, dict): |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
532 for sub_block in _walk_trie(item): |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
533 yield sub_block |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
534 yield block |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
535 |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
536 |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
537 def _persist_block(block_node, block_map): |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
538 """produce persistent binary data for a single block |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
539 |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
540 Children block are assumed to be already persisted and present in |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
541 block_map. |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
542 """ |
44316
55b12f2593c1
nodemap: move the iteratio inside the Block object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44315
diff
changeset
|
543 data = tuple(_to_value(v, block_map) for v in block_node) |
44307
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
544 return S_BLOCK.pack(*data) |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
545 |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
546 |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
547 def _to_value(item, block_map): |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
548 """persist any value as an integer""" |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
549 if item is None: |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
550 return NO_ENTRY |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
551 elif isinstance(item, dict): |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
552 return block_map[id(item)] |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
553 else: |
c577bb4a04d4
nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44034
diff
changeset
|
554 return _transform_rev(item) |
44317
78721bbdb2ab
nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44316
diff
changeset
|
555 |
78721bbdb2ab
nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44316
diff
changeset
|
556 |
78721bbdb2ab
nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44316
diff
changeset
|
557 def parse_data(data): |
78721bbdb2ab
nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44316
diff
changeset
|
558 """parse parse nodemap data into a nodemap Trie""" |
78721bbdb2ab
nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44316
diff
changeset
|
559 if (len(data) % S_BLOCK.size) != 0: |
78721bbdb2ab
nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44316
diff
changeset
|
560 msg = "nodemap data size is not a multiple of block size (%d): %d" |
78721bbdb2ab
nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44316
diff
changeset
|
561 raise error.Abort(msg % (S_BLOCK.size, len(data))) |
78721bbdb2ab
nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44316
diff
changeset
|
562 if not data: |
44333
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
563 return Block(), None |
44317
78721bbdb2ab
nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44316
diff
changeset
|
564 block_map = {} |
78721bbdb2ab
nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44316
diff
changeset
|
565 new_blocks = [] |
78721bbdb2ab
nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44316
diff
changeset
|
566 for i in range(0, len(data), S_BLOCK.size): |
78721bbdb2ab
nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44316
diff
changeset
|
567 block = Block() |
44321
f0862ee1a31e
nodemap: keep track of the ondisk id of nodemap blocks
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44319
diff
changeset
|
568 block.ondisk_id = len(block_map) |
f0862ee1a31e
nodemap: keep track of the ondisk id of nodemap blocks
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44319
diff
changeset
|
569 block_map[block.ondisk_id] = block |
44317
78721bbdb2ab
nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44316
diff
changeset
|
570 block_data = data[i : i + S_BLOCK.size] |
78721bbdb2ab
nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44316
diff
changeset
|
571 values = S_BLOCK.unpack(block_data) |
78721bbdb2ab
nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44316
diff
changeset
|
572 new_blocks.append((block, values)) |
78721bbdb2ab
nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44316
diff
changeset
|
573 for b, values in new_blocks: |
78721bbdb2ab
nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44316
diff
changeset
|
574 for idx, v in enumerate(values): |
78721bbdb2ab
nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44316
diff
changeset
|
575 if v == NO_ENTRY: |
78721bbdb2ab
nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44316
diff
changeset
|
576 continue |
78721bbdb2ab
nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44316
diff
changeset
|
577 elif v >= 0: |
78721bbdb2ab
nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44316
diff
changeset
|
578 b[idx] = block_map[v] |
78721bbdb2ab
nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44316
diff
changeset
|
579 else: |
78721bbdb2ab
nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44316
diff
changeset
|
580 b[idx] = _transform_rev(v) |
44333
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
581 return block, i // S_BLOCK.size |
44318
20e125cdd719
nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44317
diff
changeset
|
582 |
20e125cdd719
nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44317
diff
changeset
|
583 |
20e125cdd719
nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44317
diff
changeset
|
584 # debug utility |
20e125cdd719
nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44317
diff
changeset
|
585 |
20e125cdd719
nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44317
diff
changeset
|
586 |
20e125cdd719
nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44317
diff
changeset
|
587 def check_data(ui, index, data): |
20e125cdd719
nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44317
diff
changeset
|
588 """verify that the provided nodemap data are valid for the given idex""" |
20e125cdd719
nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44317
diff
changeset
|
589 ret = 0 |
20e125cdd719
nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44317
diff
changeset
|
590 ui.status((b"revision in index: %d\n") % len(index)) |
44333
50ad851efd9b
nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44323
diff
changeset
|
591 root, __ = parse_data(data) |
44318
20e125cdd719
nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44317
diff
changeset
|
592 all_revs = set(_all_revisions(root)) |
20e125cdd719
nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44317
diff
changeset
|
593 ui.status((b"revision in nodemap: %d\n") % len(all_revs)) |
20e125cdd719
nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44317
diff
changeset
|
594 for r in range(len(index)): |
20e125cdd719
nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44317
diff
changeset
|
595 if r not in all_revs: |
20e125cdd719
nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44317
diff
changeset
|
596 msg = b" revision missing from nodemap: %d\n" % r |
20e125cdd719
nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44317
diff
changeset
|
597 ui.write_err(msg) |
20e125cdd719
nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44317
diff
changeset
|
598 ret = 1 |
20e125cdd719
nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44317
diff
changeset
|
599 else: |
20e125cdd719
nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44317
diff
changeset
|
600 all_revs.remove(r) |
46113
59fa3890d40a
node: import symbols explicitly
Joerg Sonnenberger <joerg@bec.de>
parents:
46089
diff
changeset
|
601 nm_rev = _find_node(root, hex(index[r][7])) |
44319
d58206b70199
nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44318
diff
changeset
|
602 if nm_rev is None: |
d58206b70199
nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44318
diff
changeset
|
603 msg = b" revision node does not match any entries: %d\n" % r |
d58206b70199
nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44318
diff
changeset
|
604 ui.write_err(msg) |
d58206b70199
nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44318
diff
changeset
|
605 ret = 1 |
d58206b70199
nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44318
diff
changeset
|
606 elif nm_rev != r: |
d58206b70199
nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44318
diff
changeset
|
607 msg = ( |
d58206b70199
nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44318
diff
changeset
|
608 b" revision node does not match the expected revision: " |
d58206b70199
nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44318
diff
changeset
|
609 b"%d != %d\n" % (r, nm_rev) |
d58206b70199
nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44318
diff
changeset
|
610 ) |
d58206b70199
nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44318
diff
changeset
|
611 ui.write_err(msg) |
d58206b70199
nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44318
diff
changeset
|
612 ret = 1 |
d58206b70199
nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44318
diff
changeset
|
613 |
44318
20e125cdd719
nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44317
diff
changeset
|
614 if all_revs: |
20e125cdd719
nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44317
diff
changeset
|
615 for r in sorted(all_revs): |
20e125cdd719
nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44317
diff
changeset
|
616 msg = b" extra revision in nodemap: %d\n" % r |
20e125cdd719
nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44317
diff
changeset
|
617 ui.write_err(msg) |
20e125cdd719
nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44317
diff
changeset
|
618 ret = 1 |
20e125cdd719
nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44317
diff
changeset
|
619 return ret |
20e125cdd719
nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44317
diff
changeset
|
620 |
20e125cdd719
nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44317
diff
changeset
|
621 |
20e125cdd719
nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44317
diff
changeset
|
622 def _all_revisions(root): |
20e125cdd719
nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44317
diff
changeset
|
623 """return all revisions stored in a Trie""" |
20e125cdd719
nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44317
diff
changeset
|
624 for block in _walk_trie(root): |
20e125cdd719
nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44317
diff
changeset
|
625 for v in block: |
20e125cdd719
nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44317
diff
changeset
|
626 if v is None or isinstance(v, Block): |
20e125cdd719
nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44317
diff
changeset
|
627 continue |
20e125cdd719
nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44317
diff
changeset
|
628 yield v |
44319
d58206b70199
nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44318
diff
changeset
|
629 |
d58206b70199
nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44318
diff
changeset
|
630 |
d58206b70199
nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44318
diff
changeset
|
631 def _find_node(block, node): |
d58206b70199
nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44318
diff
changeset
|
632 """find the revision associated with a given node""" |
d58206b70199
nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44318
diff
changeset
|
633 entry = block.get(_to_int(node[0:1])) |
d58206b70199
nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44318
diff
changeset
|
634 if isinstance(entry, dict): |
d58206b70199
nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44318
diff
changeset
|
635 return _find_node(entry, node[1:]) |
d58206b70199
nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44318
diff
changeset
|
636 return entry |
46469
835aafb2543f
revlog: refactor logic to compute nodemap file in separate function
Pulkit Goyal <7895pulkit@gmail.com>
parents:
46468
diff
changeset
|
637 |
835aafb2543f
revlog: refactor logic to compute nodemap file in separate function
Pulkit Goyal <7895pulkit@gmail.com>
parents:
46468
diff
changeset
|
638 |
835aafb2543f
revlog: refactor logic to compute nodemap file in separate function
Pulkit Goyal <7895pulkit@gmail.com>
parents:
46468
diff
changeset
|
639 def get_nodemap_file(opener, indexfile): |
835aafb2543f
revlog: refactor logic to compute nodemap file in separate function
Pulkit Goyal <7895pulkit@gmail.com>
parents:
46468
diff
changeset
|
640 if indexfile.endswith(b'.a'): |
835aafb2543f
revlog: refactor logic to compute nodemap file in separate function
Pulkit Goyal <7895pulkit@gmail.com>
parents:
46468
diff
changeset
|
641 pending_path = indexfile[:-4] + b".n.a" |
835aafb2543f
revlog: refactor logic to compute nodemap file in separate function
Pulkit Goyal <7895pulkit@gmail.com>
parents:
46468
diff
changeset
|
642 if opener.exists(pending_path): |
835aafb2543f
revlog: refactor logic to compute nodemap file in separate function
Pulkit Goyal <7895pulkit@gmail.com>
parents:
46468
diff
changeset
|
643 return pending_path |
835aafb2543f
revlog: refactor logic to compute nodemap file in separate function
Pulkit Goyal <7895pulkit@gmail.com>
parents:
46468
diff
changeset
|
644 else: |
835aafb2543f
revlog: refactor logic to compute nodemap file in separate function
Pulkit Goyal <7895pulkit@gmail.com>
parents:
46468
diff
changeset
|
645 return indexfile[:-4] + b".n" |
835aafb2543f
revlog: refactor logic to compute nodemap file in separate function
Pulkit Goyal <7895pulkit@gmail.com>
parents:
46468
diff
changeset
|
646 else: |
835aafb2543f
revlog: refactor logic to compute nodemap file in separate function
Pulkit Goyal <7895pulkit@gmail.com>
parents:
46468
diff
changeset
|
647 return indexfile[:-2] + b".n" |