annotate mercurial/revlogutils/randomaccessfile.py @ 51700:7f0cb9ee0534

Backout accidental publication of a large range of revisions I accidentally published 25e7f9dcad0f::bd1483fd7088, this is the inverse.
author Raphaël Gomès <rgomes@octobus.net>
date Tue, 23 Jul 2024 10:02:46 +0200
parents a93e52f0b6ff
children eb9dea148233
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
47425
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
1 # Copyright Mercurial Contributors
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
2 #
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
3 # This software may be used and distributed according to the terms of the
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
4 # GNU General Public License version 2 or any later version.
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
5
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
6 import contextlib
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
7
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
8 from ..i18n import _
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
9 from .. import (
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
10 error,
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
11 util,
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
12 )
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
13
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
14
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
15 _MAX_CACHED_CHUNK_SIZE = 1048576 # 1 MiB
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
16
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
17 PARTIAL_READ_MSG = _(
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
18 b'partial read of revlog %s; expected %d bytes from offset %d, got %d'
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
19 )
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
20
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
21
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
22 def _is_power_of_two(n):
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
23 return (n & (n - 1) == 0) and n != 0
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
24
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
25
51103
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
26 class appender:
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
27 """the changelog index must be updated last on disk, so we use this class
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
28 to delay writes to it"""
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
29
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
30 def __init__(self, vfs, name, mode, buf):
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
31 self.data = buf
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
32 fp = vfs(name, mode)
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
33 self.fp = fp
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
34 self.offset = fp.tell()
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
35 self.size = vfs.fstat(fp).st_size
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
36 self._end = self.size
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
37
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
38 def end(self):
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
39 return self._end
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
40
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
41 def tell(self):
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
42 return self.offset
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
43
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
44 def flush(self):
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
45 pass
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
46
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
47 @property
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
48 def closed(self):
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
49 return self.fp.closed
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
50
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
51 def close(self):
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
52 self.fp.close()
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
53
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
54 def seek(self, offset, whence=0):
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
55 '''virtual file offset spans real file and data'''
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
56 if whence == 0:
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
57 self.offset = offset
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
58 elif whence == 1:
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
59 self.offset += offset
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
60 elif whence == 2:
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
61 self.offset = self.end() + offset
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
62 if self.offset < self.size:
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
63 self.fp.seek(self.offset)
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
64
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
65 def read(self, count=-1):
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
66 '''only trick here is reads that span real file and data'''
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
67 ret = b""
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
68 if self.offset < self.size:
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
69 s = self.fp.read(count)
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
70 ret = s
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
71 self.offset += len(s)
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
72 if count > 0:
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
73 count -= len(s)
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
74 if count != 0:
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
75 doff = self.offset - self.size
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
76 self.data.insert(0, b"".join(self.data))
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
77 del self.data[1:]
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
78 s = self.data[0][doff : doff + count]
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
79 self.offset += len(s)
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
80 ret += s
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
81 return ret
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
82
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
83 def write(self, s):
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
84 self.data.append(bytes(s))
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
85 self.offset += len(s)
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
86 self._end += len(s)
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
87
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
88 def __enter__(self):
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
89 self.fp.__enter__()
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
90 return self
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
91
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
92 def __exit__(self, *args):
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
93 return self.fp.__exit__(*args)
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
94
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51102
diff changeset
95
48946
642e31cb55f0 py3: use class X: instead of class X(object):
Gregory Szorc <gregory.szorc@gmail.com>
parents: 47463
diff changeset
96 class randomaccessfile:
47425
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
97 """Accessing arbitrary chuncks of data within a file, with some caching"""
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
98
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
99 def __init__(
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
100 self,
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
101 opener,
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
102 filename,
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
103 default_cached_chunk_size,
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
104 initial_cache=None,
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
105 ):
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
106 # Required by bitwise manipulation below
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
107 assert _is_power_of_two(default_cached_chunk_size)
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
108
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
109 self.opener = opener
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
110 self.filename = filename
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
111 self.default_cached_chunk_size = default_cached_chunk_size
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
112 self.writing_handle = None # This is set from revlog.py
47463
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
113 self.reading_handle = None
47425
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
114 self._cached_chunk = b''
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
115 self._cached_chunk_position = 0 # Offset from the start of the file
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
116 if initial_cache:
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
117 self._cached_chunk_position, self._cached_chunk = initial_cache
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
118
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
119 def clear_cache(self):
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
120 self._cached_chunk = b''
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
121 self._cached_chunk_position = 0
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
122
51102
594f912818ab changelog-delay: adds some check around delaying and diverting write
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51027
diff changeset
123 @property
594f912818ab changelog-delay: adds some check around delaying and diverting write
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51027
diff changeset
124 def is_open(self):
594f912818ab changelog-delay: adds some check around delaying and diverting write
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51027
diff changeset
125 """True if any file handle is being held
594f912818ab changelog-delay: adds some check around delaying and diverting write
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51027
diff changeset
126
594f912818ab changelog-delay: adds some check around delaying and diverting write
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51027
diff changeset
127 Used for assert and debug in the python code"""
594f912818ab changelog-delay: adds some check around delaying and diverting write
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51027
diff changeset
128 return (
594f912818ab changelog-delay: adds some check around delaying and diverting write
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51027
diff changeset
129 self.reading_handle is not None or self.writing_handle is not None
594f912818ab changelog-delay: adds some check around delaying and diverting write
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51027
diff changeset
130 )
594f912818ab changelog-delay: adds some check around delaying and diverting write
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51027
diff changeset
131
47425
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
132 def _open(self, mode=b'r'):
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
133 """Return a file object"""
51182
a93e52f0b6ff changelog: disallow delayed write on inline changesets
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
134 return self.opener(self.filename, mode=mode)
47425
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
135
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
136 @contextlib.contextmanager
51027
3314c41c3759 randomaccessfile: drop explicit passing of file description
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48946
diff changeset
137 def _read_handle(self):
47425
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
138 """File object suitable for reading data"""
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
139 # Use a file handle being actively used for writes, if available.
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
140 # There is some danger to doing this because reads will seek the
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
141 # file. However, revlog._writeentry performs a SEEK_END before all
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
142 # writes, so we should be safe.
51027
3314c41c3759 randomaccessfile: drop explicit passing of file description
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48946
diff changeset
143 if self.writing_handle:
47425
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
144 yield self.writing_handle
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
145
47463
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
146 elif self.reading_handle:
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
147 yield self.reading_handle
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
148
47425
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
149 # Otherwise open a new file handle.
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
150 else:
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
151 with self._open() as fp:
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
152 yield fp
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
153
47463
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
154 @contextlib.contextmanager
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
155 def reading(self):
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
156 """Context manager that keeps the file open for reading"""
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
157 if (
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
158 self.reading_handle is None
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
159 and self.writing_handle is None
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
160 and self.filename is not None
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
161 ):
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
162 with self._open() as fp:
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
163 self.reading_handle = fp
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
164 try:
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
165 yield
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
166 finally:
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
167 self.reading_handle = None
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
168 else:
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
169 yield
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
170
51027
3314c41c3759 randomaccessfile: drop explicit passing of file description
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48946
diff changeset
171 def read_chunk(self, offset, length):
47425
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
172 """Read a chunk of bytes from the file.
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
173
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
174 Accepts an absolute offset, length to read, and an optional existing
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
175 file handle to read from.
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
176
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
177 If an existing file handle is passed, it will be seeked and the
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
178 original seek position will NOT be restored.
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
179
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
180 Returns a str or buffer of raw byte data.
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
181
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
182 Raises if the requested number of bytes could not be read.
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
183 """
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
184 end = offset + length
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
185 cache_start = self._cached_chunk_position
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
186 cache_end = cache_start + len(self._cached_chunk)
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
187 # Is the requested chunk within the cache?
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
188 if cache_start <= offset and end <= cache_end:
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
189 if cache_start == offset and end == cache_end:
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
190 return self._cached_chunk # avoid a copy
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
191 relative_start = offset - cache_start
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
192 return util.buffer(self._cached_chunk, relative_start, length)
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
193
51027
3314c41c3759 randomaccessfile: drop explicit passing of file description
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48946
diff changeset
194 return self._read_and_update_cache(offset, length)
47425
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
195
51027
3314c41c3759 randomaccessfile: drop explicit passing of file description
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48946
diff changeset
196 def _read_and_update_cache(self, offset, length):
47425
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
197 # Cache data both forward and backward around the requested
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
198 # data, in a fixed size window. This helps speed up operations
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
199 # involving reading the revlog backwards.
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
200 real_offset = offset & ~(self.default_cached_chunk_size - 1)
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
201 real_length = (
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
202 (offset + length + self.default_cached_chunk_size)
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
203 & ~(self.default_cached_chunk_size - 1)
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
204 ) - real_offset
51027
3314c41c3759 randomaccessfile: drop explicit passing of file description
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48946
diff changeset
205 with self._read_handle() as file_obj:
47425
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
206 file_obj.seek(real_offset)
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
207 data = file_obj.read(real_length)
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
208
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
209 self._add_cached_chunk(real_offset, data)
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
210
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
211 relative_offset = offset - real_offset
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
212 got = len(data) - relative_offset
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
213 if got < length:
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
214 message = PARTIAL_READ_MSG % (self.filename, length, offset, got)
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
215 raise error.RevlogError(message)
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
216
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
217 if offset != real_offset or real_length != length:
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
218 return util.buffer(data, relative_offset, length)
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
219 return data
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
220
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
221 def _add_cached_chunk(self, offset, data):
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
222 """Add to or replace the cached data chunk.
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
223
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
224 Accepts an absolute offset and the data that is at that location.
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
225 """
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
226 if (
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
227 self._cached_chunk_position + len(self._cached_chunk) == offset
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
228 and len(self._cached_chunk) + len(data) < _MAX_CACHED_CHUNK_SIZE
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
229 ):
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
230 # add to existing cache
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
231 self._cached_chunk += data
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
232 else:
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
233 self._cached_chunk = data
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
234 self._cached_chunk_position = offset