annotate mercurial/store.py @ 50641:5460424092e2

stream-clone: smoothly detect and handle a case were a revlog is split This detect and handle the most common case for a race condition around stream and revlog splitting. The one were the revlog is split between the initial collection of data and the time were we start considering stream that data. In such case, we repatch an inlined version of that revlog together when this happens. This is necessary as stream-v2 promised a specific number of bytes and a specific number of files to the client. In stream-v3, we will have the opportunity to just send a split revlog instead. Getting a better version of the protocol for stream-v3 is still useful, but it is no longer a blocket to fix that race condition. Note that another, rarer race condition exist, were the revlog is split while we creating the revlog and extracing content from it. This can be dealt with later.
author Pierre-Yves David <pierre-yves.david@octobus.net>
date Mon, 29 May 2023 18:41:58 +0200
parents 9caa860dcbec
children 6b522a9e7451
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
6839
01db3e101362 move filename encoding functions from util.py to new store.py
Adrian Buehlmann <adrian@cadifra.com>
parents:
diff changeset
1 # store.py - repository store handling for Mercurial
01db3e101362 move filename encoding functions from util.py to new store.py
Adrian Buehlmann <adrian@cadifra.com>
parents:
diff changeset
2 #
46819
d4ba4d51f85f contributor: change mentions of mpm to olivia
Raphaël Gomès <rgomes@octobus.net>
parents: 46780
diff changeset
3 # Copyright 2008 Olivia Mackall <olivia@selenic.com>
6839
01db3e101362 move filename encoding functions from util.py to new store.py
Adrian Buehlmann <adrian@cadifra.com>
parents:
diff changeset
4 #
8225
46293a0c7e9f updated license to be explicit about GPL version 2
Martin Geisler <mg@lazybytes.net>
parents: 8210
diff changeset
5 # This software may be used and distributed according to the terms of the
10263
25e572394f5c Update license to GPLv2+
Matt Mackall <mpm@selenic.com>
parents: 9133
diff changeset
6 # GNU General Public License version 2 or any later version.
6839
01db3e101362 move filename encoding functions from util.py to new store.py
Adrian Buehlmann <adrian@cadifra.com>
parents:
diff changeset
7
50479
5217e36356bb store: add logic to group revlog file together
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50478
diff changeset
8 import collections
41978
a56487081109 store: don't read the whole fncache in memory
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 41973
diff changeset
9 import functools
27480
509159675cdb store: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26778
diff changeset
10 import os
46990
0b569c75d180 store: exclude `undo.` nodemap's file from `walk`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
11 import re
27480
509159675cdb store: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26778
diff changeset
12 import stat
50638
8fc10bfd9887 store: declare a `files` method on BaseStoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50632
diff changeset
13 from typing import Generator, List
27480
509159675cdb store: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26778
diff changeset
14
509159675cdb store: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26778
diff changeset
15 from .i18n import _
43089
c59eb1560c44 py3: manually import getattr where it is needed
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43077
diff changeset
16 from .pycompat import getattr
50471
521fec115dad store: use a StoreEntry object instead of tuple for store files
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50469
diff changeset
17 from .thirdparty import attr
46113
59fa3890d40a node: import symbols explicitly
Joerg Sonnenberger <joerg@bec.de>
parents: 45942
diff changeset
18 from .node import hex
27480
509159675cdb store: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26778
diff changeset
19 from . import (
42905
3df3b139a43d localrepo: push manifestlog and changelog construction code into store
Augie Fackler <augie@google.com>
parents: 42715
diff changeset
20 changelog,
27480
509159675cdb store: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26778
diff changeset
21 error,
50627
e1ee6910f6bc store: add a `get_revlog_instance` method on revlog entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50514
diff changeset
22 filelog,
42905
3df3b139a43d localrepo: push manifestlog and changelog construction code into store
Augie Fackler <augie@google.com>
parents: 42715
diff changeset
23 manifest,
32372
df448de7cf3b parsers: switch to policy importer
Yuya Nishihara <yuya@tcha.org>
parents: 31362
diff changeset
24 policy,
30077
8f42d8c412c8 py3: make encodefun in store.py compatible with py3k
Mateusz Kwapich <mitrandir@fb.com>
parents: 30076
diff changeset
25 pycompat,
27480
509159675cdb store: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26778
diff changeset
26 util,
31234
9b7a2ef4f27c vfs: use 'vfs' module directly in 'mercurial.store'
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31219
diff changeset
27 vfs as vfsmod,
27480
509159675cdb store: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26778
diff changeset
28 )
44060
a61287a95dc3 core: migrate uses of hashlib.sha1 to hashutil.sha1
Augie Fackler <augie@google.com>
parents: 43506
diff changeset
29 from .utils import hashutil
6840
80e51429cb9a introduce store classes
Adrian Buehlmann <adrian@cadifra.com>
parents: 6839
diff changeset
30
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43106
diff changeset
31 parsers = policy.importmod('parsers')
41978
a56487081109 store: don't read the whole fncache in memory
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 41973
diff changeset
32 # how much bytes should be read from fncache in one read
a56487081109 store: don't read the whole fncache in memory
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 41973
diff changeset
33 # It is done to prevent loading large fncache files into memory
a56487081109 store: don't read the whole fncache in memory
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 41973
diff changeset
34 fncache_chunksize = 10 ** 6
32372
df448de7cf3b parsers: switch to policy importer
Yuya Nishihara <yuya@tcha.org>
parents: 31362
diff changeset
35
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
36
50494
b4953fad744e store: do the revlog matching on entry directly
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50493
diff changeset
37 def _match_tracked_entry(entry, matcher):
40494
9aeb9e2d28a7 store: introduce _matchtrackedpath() and use it to filter store files
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 40340
diff changeset
38 """parses a fncache entry and returns whether the entry is tracking a path
9aeb9e2d28a7 store: introduce _matchtrackedpath() and use it to filter store files
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 40340
diff changeset
39 matched by matcher or not.
9aeb9e2d28a7 store: introduce _matchtrackedpath() and use it to filter store files
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 40340
diff changeset
40
9aeb9e2d28a7 store: introduce _matchtrackedpath() and use it to filter store files
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 40340
diff changeset
41 If matcher is None, returns True"""
9aeb9e2d28a7 store: introduce _matchtrackedpath() and use it to filter store files
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 40340
diff changeset
42
9aeb9e2d28a7 store: introduce _matchtrackedpath() and use it to filter store files
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 40340
diff changeset
43 if matcher is None:
9aeb9e2d28a7 store: introduce _matchtrackedpath() and use it to filter store files
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 40340
diff changeset
44 return True
50498
1b776f25302f store: use the boolean property in `store`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50497
diff changeset
45 if entry.is_filelog:
50494
b4953fad744e store: do the revlog matching on entry directly
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50493
diff changeset
46 return matcher(entry.target_id)
50498
1b776f25302f store: use the boolean property in `store`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50497
diff changeset
47 elif entry.is_manifestlog:
50494
b4953fad744e store: do the revlog matching on entry directly
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50493
diff changeset
48 return matcher.visitdir(entry.target_id.rstrip(b'/'))
b4953fad744e store: do the revlog matching on entry directly
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50493
diff changeset
49 raise error.ProgrammingError(b"cannot process entry %r" % entry)
40624
66adfd58cb77 store: raise ProgrammingError if unable to decode a storage path
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 40584
diff changeset
50
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
51
8531
810387f59696 filelog encoding: move the encoding/decoding into store
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 8530
diff changeset
52 # This avoids a collision between a file named foo and a dir named
810387f59696 filelog encoding: move the encoding/decoding into store
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 8530
diff changeset
53 # foo.i or foo.d
17607
cc58dc47cb5e store: use fast C implementation of encodedir() if it's available
Adrian Buehlmann <adrian@cadifra.com>
parents: 17605
diff changeset
54 def _encodedir(path):
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45483
diff changeset
55 """
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 33412
diff changeset
56 >>> _encodedir(b'data/foo.i')
13949
ba43aa1e173c store: add some doctests
Adrian Buehlmann <adrian@cadifra.com>
parents: 13426
diff changeset
57 'data/foo.i'
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 33412
diff changeset
58 >>> _encodedir(b'data/foo.i/bla.i')
13949
ba43aa1e173c store: add some doctests
Adrian Buehlmann <adrian@cadifra.com>
parents: 13426
diff changeset
59 'data/foo.i.hg/bla.i'
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 33412
diff changeset
60 >>> _encodedir(b'data/foo.i.hg/bla.i')
13949
ba43aa1e173c store: add some doctests
Adrian Buehlmann <adrian@cadifra.com>
parents: 13426
diff changeset
61 'data/foo.i.hg.hg/bla.i'
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 33412
diff changeset
62 >>> _encodedir(b'data/foo.i\\ndata/foo.i/bla.i\\ndata/foo.i.hg/bla.i\\n')
17605
e9cc29be3305 store: add multiline doctest case for encodedir()
Adrian Buehlmann <adrian@cadifra.com>
parents: 17604
diff changeset
63 'data/foo.i\\ndata/foo.i.hg/bla.i\\ndata/foo.i.hg.hg/bla.i\\n'
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45483
diff changeset
64 """
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
65 return (
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
66 path.replace(b".hg/", b".hg.hg/")
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
67 .replace(b".i/", b".i.hg/")
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
68 .replace(b".d/", b".d.hg/")
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
69 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
70
8531
810387f59696 filelog encoding: move the encoding/decoding into store
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 8530
diff changeset
71
17607
cc58dc47cb5e store: use fast C implementation of encodedir() if it's available
Adrian Buehlmann <adrian@cadifra.com>
parents: 17605
diff changeset
72 encodedir = getattr(parsers, 'encodedir', _encodedir)
cc58dc47cb5e store: use fast C implementation of encodedir() if it's available
Adrian Buehlmann <adrian@cadifra.com>
parents: 17605
diff changeset
73
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
74
8531
810387f59696 filelog encoding: move the encoding/decoding into store
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 8530
diff changeset
75 def decodedir(path):
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45483
diff changeset
76 """
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 33412
diff changeset
77 >>> decodedir(b'data/foo.i')
13949
ba43aa1e173c store: add some doctests
Adrian Buehlmann <adrian@cadifra.com>
parents: 13426
diff changeset
78 'data/foo.i'
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 33412
diff changeset
79 >>> decodedir(b'data/foo.i.hg/bla.i')
13949
ba43aa1e173c store: add some doctests
Adrian Buehlmann <adrian@cadifra.com>
parents: 13426
diff changeset
80 'data/foo.i/bla.i'
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 33412
diff changeset
81 >>> decodedir(b'data/foo.i.hg.hg/bla.i')
13949
ba43aa1e173c store: add some doctests
Adrian Buehlmann <adrian@cadifra.com>
parents: 13426
diff changeset
82 'data/foo.i.hg/bla.i'
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45483
diff changeset
83 """
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
84 if b".hg/" not in path:
8531
810387f59696 filelog encoding: move the encoding/decoding into store
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 8530
diff changeset
85 return path
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
86 return (
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
87 path.replace(b".d.hg/", b".d/")
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
88 .replace(b".i.hg/", b".i/")
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
89 .replace(b".hg.hg/", b".hg/")
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
90 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
91
8531
810387f59696 filelog encoding: move the encoding/decoding into store
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 8530
diff changeset
92
29071
2f58975eb2cb store: treat range as a generator instead of a list for py3 compat
timeless <timeless@mozdev.org>
parents: 28007
diff changeset
93 def _reserved():
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45483
diff changeset
94 """characters that are problematic for filesystems
29071
2f58975eb2cb store: treat range as a generator instead of a list for py3 compat
timeless <timeless@mozdev.org>
parents: 28007
diff changeset
95
2f58975eb2cb store: treat range as a generator instead of a list for py3 compat
timeless <timeless@mozdev.org>
parents: 28007
diff changeset
96 * ascii escapes (0..31)
2f58975eb2cb store: treat range as a generator instead of a list for py3 compat
timeless <timeless@mozdev.org>
parents: 28007
diff changeset
97 * ascii hi (126..255)
2f58975eb2cb store: treat range as a generator instead of a list for py3 compat
timeless <timeless@mozdev.org>
parents: 28007
diff changeset
98 * windows specials
2f58975eb2cb store: treat range as a generator instead of a list for py3 compat
timeless <timeless@mozdev.org>
parents: 28007
diff changeset
99
2f58975eb2cb store: treat range as a generator instead of a list for py3 compat
timeless <timeless@mozdev.org>
parents: 28007
diff changeset
100 these characters will be escaped by encodefunctions
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45483
diff changeset
101 """
30076
400dfded8a29 py3: make the string unicode so its iterable in py3k
Mateusz Kwapich <mitrandir@fb.com>
parents: 29338
diff changeset
102 winreserved = [ord(x) for x in u'\\:*?"<>|']
29071
2f58975eb2cb store: treat range as a generator instead of a list for py3 compat
timeless <timeless@mozdev.org>
parents: 28007
diff changeset
103 for x in range(32):
2f58975eb2cb store: treat range as a generator instead of a list for py3 compat
timeless <timeless@mozdev.org>
parents: 28007
diff changeset
104 yield x
2f58975eb2cb store: treat range as a generator instead of a list for py3 compat
timeless <timeless@mozdev.org>
parents: 28007
diff changeset
105 for x in range(126, 256):
2f58975eb2cb store: treat range as a generator instead of a list for py3 compat
timeless <timeless@mozdev.org>
parents: 28007
diff changeset
106 yield x
2f58975eb2cb store: treat range as a generator instead of a list for py3 compat
timeless <timeless@mozdev.org>
parents: 28007
diff changeset
107 for x in winreserved:
2f58975eb2cb store: treat range as a generator instead of a list for py3 compat
timeless <timeless@mozdev.org>
parents: 28007
diff changeset
108 yield x
2f58975eb2cb store: treat range as a generator instead of a list for py3 compat
timeless <timeless@mozdev.org>
parents: 28007
diff changeset
109
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
110
6839
01db3e101362 move filename encoding functions from util.py to new store.py
Adrian Buehlmann <adrian@cadifra.com>
parents:
diff changeset
111 def _buildencodefun():
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45483
diff changeset
112 """
13949
ba43aa1e173c store: add some doctests
Adrian Buehlmann <adrian@cadifra.com>
parents: 13426
diff changeset
113 >>> enc, dec = _buildencodefun()
ba43aa1e173c store: add some doctests
Adrian Buehlmann <adrian@cadifra.com>
parents: 13426
diff changeset
114
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 33412
diff changeset
115 >>> enc(b'nothing/special.txt')
13949
ba43aa1e173c store: add some doctests
Adrian Buehlmann <adrian@cadifra.com>
parents: 13426
diff changeset
116 'nothing/special.txt'
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 33412
diff changeset
117 >>> dec(b'nothing/special.txt')
13949
ba43aa1e173c store: add some doctests
Adrian Buehlmann <adrian@cadifra.com>
parents: 13426
diff changeset
118 'nothing/special.txt'
ba43aa1e173c store: add some doctests
Adrian Buehlmann <adrian@cadifra.com>
parents: 13426
diff changeset
119
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 33412
diff changeset
120 >>> enc(b'HELLO')
13949
ba43aa1e173c store: add some doctests
Adrian Buehlmann <adrian@cadifra.com>
parents: 13426
diff changeset
121 '_h_e_l_l_o'
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 33412
diff changeset
122 >>> dec(b'_h_e_l_l_o')
13949
ba43aa1e173c store: add some doctests
Adrian Buehlmann <adrian@cadifra.com>
parents: 13426
diff changeset
123 'HELLO'
ba43aa1e173c store: add some doctests
Adrian Buehlmann <adrian@cadifra.com>
parents: 13426
diff changeset
124
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 33412
diff changeset
125 >>> enc(b'hello:world?')
13949
ba43aa1e173c store: add some doctests
Adrian Buehlmann <adrian@cadifra.com>
parents: 13426
diff changeset
126 'hello~3aworld~3f'
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 33412
diff changeset
127 >>> dec(b'hello~3aworld~3f')
13949
ba43aa1e173c store: add some doctests
Adrian Buehlmann <adrian@cadifra.com>
parents: 13426
diff changeset
128 'hello:world?'
ba43aa1e173c store: add some doctests
Adrian Buehlmann <adrian@cadifra.com>
parents: 13426
diff changeset
129
34136
414a3513c2bd doctest: do not embed non-ascii characters in docstring
Yuya Nishihara <yuya@tcha.org>
parents: 34131
diff changeset
130 >>> enc(b'the\\x07quick\\xADshot')
13949
ba43aa1e173c store: add some doctests
Adrian Buehlmann <adrian@cadifra.com>
parents: 13426
diff changeset
131 'the~07quick~adshot'
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 33412
diff changeset
132 >>> dec(b'the~07quick~adshot')
13949
ba43aa1e173c store: add some doctests
Adrian Buehlmann <adrian@cadifra.com>
parents: 13426
diff changeset
133 'the\\x07quick\\xadshot'
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45483
diff changeset
134 """
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
135 e = b'_'
31253
64596338ba10 py3: factor out bytechr() function
Yuya Nishihara <yuya@tcha.org>
parents: 31234
diff changeset
136 xchr = pycompat.bytechr
64596338ba10 py3: factor out bytechr() function
Yuya Nishihara <yuya@tcha.org>
parents: 31234
diff changeset
137 asciistr = list(map(xchr, range(127)))
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
138 capitals = list(range(ord(b"A"), ord(b"Z") + 1))
30077
8f42d8c412c8 py3: make encodefun in store.py compatible with py3k
Mateusz Kwapich <mitrandir@fb.com>
parents: 30076
diff changeset
139
44452
9d2b2df2c2ba cleanup: run pyupgrade on our source tree to clean up varying things
Augie Fackler <augie@google.com>
parents: 44060
diff changeset
140 cmap = {x: x for x in asciistr}
29071
2f58975eb2cb store: treat range as a generator instead of a list for py3 compat
timeless <timeless@mozdev.org>
parents: 28007
diff changeset
141 for x in _reserved():
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
142 cmap[xchr(x)] = b"~%02x" % x
30077
8f42d8c412c8 py3: make encodefun in store.py compatible with py3k
Mateusz Kwapich <mitrandir@fb.com>
parents: 30076
diff changeset
143 for x in capitals + [ord(e)]:
8f42d8c412c8 py3: make encodefun in store.py compatible with py3k
Mateusz Kwapich <mitrandir@fb.com>
parents: 30076
diff changeset
144 cmap[xchr(x)] = e + xchr(x).lower()
8f42d8c412c8 py3: make encodefun in store.py compatible with py3k
Mateusz Kwapich <mitrandir@fb.com>
parents: 30076
diff changeset
145
6839
01db3e101362 move filename encoding functions from util.py to new store.py
Adrian Buehlmann <adrian@cadifra.com>
parents:
diff changeset
146 dmap = {}
48913
f254fc73d956 global: bulk replace simple pycompat.iteritems(x) with x.items()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 48875
diff changeset
147 for k, v in cmap.items():
6839
01db3e101362 move filename encoding functions from util.py to new store.py
Adrian Buehlmann <adrian@cadifra.com>
parents:
diff changeset
148 dmap[v] = k
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
149
6839
01db3e101362 move filename encoding functions from util.py to new store.py
Adrian Buehlmann <adrian@cadifra.com>
parents:
diff changeset
150 def decode(s):
01db3e101362 move filename encoding functions from util.py to new store.py
Adrian Buehlmann <adrian@cadifra.com>
parents:
diff changeset
151 i = 0
01db3e101362 move filename encoding functions from util.py to new store.py
Adrian Buehlmann <adrian@cadifra.com>
parents:
diff changeset
152 while i < len(s):
49284
d44e3c45f0e4 py3: replace `pycompat.xrange` by `range`
Manuel Jacob <me@manueljacob.de>
parents: 48946
diff changeset
153 for l in range(1, 4):
6839
01db3e101362 move filename encoding functions from util.py to new store.py
Adrian Buehlmann <adrian@cadifra.com>
parents:
diff changeset
154 try:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
155 yield dmap[s[i : i + l]]
6839
01db3e101362 move filename encoding functions from util.py to new store.py
Adrian Buehlmann <adrian@cadifra.com>
parents:
diff changeset
156 i += l
01db3e101362 move filename encoding functions from util.py to new store.py
Adrian Buehlmann <adrian@cadifra.com>
parents:
diff changeset
157 break
01db3e101362 move filename encoding functions from util.py to new store.py
Adrian Buehlmann <adrian@cadifra.com>
parents:
diff changeset
158 except KeyError:
01db3e101362 move filename encoding functions from util.py to new store.py
Adrian Buehlmann <adrian@cadifra.com>
parents:
diff changeset
159 pass
01db3e101362 move filename encoding functions from util.py to new store.py
Adrian Buehlmann <adrian@cadifra.com>
parents:
diff changeset
160 else:
01db3e101362 move filename encoding functions from util.py to new store.py
Adrian Buehlmann <adrian@cadifra.com>
parents:
diff changeset
161 raise KeyError
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
162
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
163 return (
49284
d44e3c45f0e4 py3: replace `pycompat.xrange` by `range`
Manuel Jacob <me@manueljacob.de>
parents: 48946
diff changeset
164 lambda s: b''.join([cmap[s[c : c + 1]] for c in range(len(s))]),
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
165 lambda s: b''.join(list(decode(s))),
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
166 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
167
17608
776240123525 store: extract functions _encodefname and _decodefname
Adrian Buehlmann <adrian@cadifra.com>
parents: 17607
diff changeset
168
776240123525 store: extract functions _encodefname and _decodefname
Adrian Buehlmann <adrian@cadifra.com>
parents: 17607
diff changeset
169 _encodefname, _decodefname = _buildencodefun()
6839
01db3e101362 move filename encoding functions from util.py to new store.py
Adrian Buehlmann <adrian@cadifra.com>
parents:
diff changeset
170
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
171
17608
776240123525 store: extract functions _encodefname and _decodefname
Adrian Buehlmann <adrian@cadifra.com>
parents: 17607
diff changeset
172 def encodefilename(s):
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45483
diff changeset
173 """
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 33412
diff changeset
174 >>> encodefilename(b'foo.i/bar.d/bla.hg/hi:world?/HELLO')
17608
776240123525 store: extract functions _encodefname and _decodefname
Adrian Buehlmann <adrian@cadifra.com>
parents: 17607
diff changeset
175 'foo.i.hg/bar.d.hg/bla.hg.hg/hi~3aworld~3f/_h_e_l_l_o'
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45483
diff changeset
176 """
17608
776240123525 store: extract functions _encodefname and _decodefname
Adrian Buehlmann <adrian@cadifra.com>
parents: 17607
diff changeset
177 return _encodefname(encodedir(s))
776240123525 store: extract functions _encodefname and _decodefname
Adrian Buehlmann <adrian@cadifra.com>
parents: 17607
diff changeset
178
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
179
17608
776240123525 store: extract functions _encodefname and _decodefname
Adrian Buehlmann <adrian@cadifra.com>
parents: 17607
diff changeset
180 def decodefilename(s):
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45483
diff changeset
181 """
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 33412
diff changeset
182 >>> decodefilename(b'foo.i.hg/bar.d.hg/bla.hg.hg/hi~3aworld~3f/_h_e_l_l_o')
17608
776240123525 store: extract functions _encodefname and _decodefname
Adrian Buehlmann <adrian@cadifra.com>
parents: 17607
diff changeset
183 'foo.i/bar.d/bla.hg/hi:world?/HELLO'
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45483
diff changeset
184 """
17608
776240123525 store: extract functions _encodefname and _decodefname
Adrian Buehlmann <adrian@cadifra.com>
parents: 17607
diff changeset
185 return decodedir(_decodefname(s))
6839
01db3e101362 move filename encoding functions from util.py to new store.py
Adrian Buehlmann <adrian@cadifra.com>
parents:
diff changeset
186
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
187
14288
00a0ab08f986 store: change names to comply with project coding standards
Adrian Buehlmann <adrian@cadifra.com>
parents: 14194
diff changeset
188 def _buildlowerencodefun():
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45483
diff changeset
189 """
14288
00a0ab08f986 store: change names to comply with project coding standards
Adrian Buehlmann <adrian@cadifra.com>
parents: 14194
diff changeset
190 >>> f = _buildlowerencodefun()
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 33412
diff changeset
191 >>> f(b'nothing/special.txt')
13949
ba43aa1e173c store: add some doctests
Adrian Buehlmann <adrian@cadifra.com>
parents: 13426
diff changeset
192 'nothing/special.txt'
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 33412
diff changeset
193 >>> f(b'HELLO')
13949
ba43aa1e173c store: add some doctests
Adrian Buehlmann <adrian@cadifra.com>
parents: 13426
diff changeset
194 'hello'
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 33412
diff changeset
195 >>> f(b'hello:world?')
13949
ba43aa1e173c store: add some doctests
Adrian Buehlmann <adrian@cadifra.com>
parents: 13426
diff changeset
196 'hello~3aworld~3f'
34136
414a3513c2bd doctest: do not embed non-ascii characters in docstring
Yuya Nishihara <yuya@tcha.org>
parents: 34131
diff changeset
197 >>> f(b'the\\x07quick\\xADshot')
13949
ba43aa1e173c store: add some doctests
Adrian Buehlmann <adrian@cadifra.com>
parents: 13426
diff changeset
198 'the~07quick~adshot'
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45483
diff changeset
199 """
34214
7e3f078b6f31 py3: use bytechr() in store._buildlowerencodefun()
Yuya Nishihara <yuya@tcha.org>
parents: 34213
diff changeset
200 xchr = pycompat.bytechr
49284
d44e3c45f0e4 py3: replace `pycompat.xrange` by `range`
Manuel Jacob <me@manueljacob.de>
parents: 48946
diff changeset
201 cmap = {xchr(x): xchr(x) for x in range(127)}
29071
2f58975eb2cb store: treat range as a generator instead of a list for py3 compat
timeless <timeless@mozdev.org>
parents: 28007
diff changeset
202 for x in _reserved():
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
203 cmap[xchr(x)] = b"~%02x" % x
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
204 for x in range(ord(b"A"), ord(b"Z") + 1):
34214
7e3f078b6f31 py3: use bytechr() in store._buildlowerencodefun()
Yuya Nishihara <yuya@tcha.org>
parents: 34213
diff changeset
205 cmap[xchr(x)] = xchr(x).lower()
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
206
34213
96808804b68f store: give name to lowerencode function
Yuya Nishihara <yuya@tcha.org>
parents: 34136
diff changeset
207 def lowerencode(s):
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
208 return b"".join([cmap[c] for c in pycompat.iterbytestr(s)])
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
209
34213
96808804b68f store: give name to lowerencode function
Yuya Nishihara <yuya@tcha.org>
parents: 34136
diff changeset
210 return lowerencode
7229
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
211
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
212
18430
0459c6555f69 store: implement lowerencode in C
Bryan O'Sullivan <bryano@fb.com>
parents: 18054
diff changeset
213 lowerencode = getattr(parsers, 'lowerencode', None) or _buildlowerencodefun()
7229
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
214
17570
f53a7b256ca6 store: optimze _auxencode() a bit by grouping the reserved names by length
Adrian Buehlmann <adrian@cadifra.com>
parents: 17569
diff changeset
215 # Windows reserved names: con, prn, aux, nul, com1..com9, lpt1..lpt9
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
216 _winres3 = (b'aux', b'con', b'prn', b'nul') # length 3
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
217 _winres4 = (b'com', b'lpt') # length 4 (with trailing 1..9)
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
218
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
219
12687
34d8247a4595 store: encode first period or space in filenames (issue1713)
Adrian Buehlmann <adrian@cadifra.com>
parents: 12171
diff changeset
220 def _auxencode(path, dotencode):
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45483
diff changeset
221 """
13949
ba43aa1e173c store: add some doctests
Adrian Buehlmann <adrian@cadifra.com>
parents: 13426
diff changeset
222 Encodes filenames containing names reserved by Windows or which end in
ba43aa1e173c store: add some doctests
Adrian Buehlmann <adrian@cadifra.com>
parents: 13426
diff changeset
223 period or space. Does not touch other single reserved characters c.
ba43aa1e173c store: add some doctests
Adrian Buehlmann <adrian@cadifra.com>
parents: 13426
diff changeset
224 Specifically, c in '\\:*?"<>|' or ord(c) <= 31 are *not* encoded here.
ba43aa1e173c store: add some doctests
Adrian Buehlmann <adrian@cadifra.com>
parents: 13426
diff changeset
225 Additionally encodes space or period at the beginning, if dotencode is
17569
e9af2134825c store: explain "aux.foo" versus "foo.aux" in doc of _auxencode()
Adrian Buehlmann <adrian@cadifra.com>
parents: 17568
diff changeset
226 True. Parameter path is assumed to be all lowercase.
e9af2134825c store: explain "aux.foo" versus "foo.aux" in doc of _auxencode()
Adrian Buehlmann <adrian@cadifra.com>
parents: 17568
diff changeset
227 A segment only needs encoding if a reserved name appears as a
e9af2134825c store: explain "aux.foo" versus "foo.aux" in doc of _auxencode()
Adrian Buehlmann <adrian@cadifra.com>
parents: 17568
diff changeset
228 basename (e.g. "aux", "aux.foo"). A directory or file named "foo.aux"
e9af2134825c store: explain "aux.foo" versus "foo.aux" in doc of _auxencode()
Adrian Buehlmann <adrian@cadifra.com>
parents: 17568
diff changeset
229 doesn't need encoding.
13949
ba43aa1e173c store: add some doctests
Adrian Buehlmann <adrian@cadifra.com>
parents: 13426
diff changeset
230
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 33412
diff changeset
231 >>> s = b'.foo/aux.txt/txt.aux/con/prn/nul/foo.'
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 33412
diff changeset
232 >>> _auxencode(s.split(b'/'), True)
17574
81a033bb29bc store: let _auxencode() return the list of path segments
Adrian Buehlmann <adrian@cadifra.com>
parents: 17573
diff changeset
233 ['~2efoo', 'au~78.txt', 'txt.aux', 'co~6e', 'pr~6e', 'nu~6c', 'foo~2e']
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 33412
diff changeset
234 >>> s = b'.com1com2/lpt9.lpt4.lpt1/conprn/com0/lpt0/foo.'
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 33412
diff changeset
235 >>> _auxencode(s.split(b'/'), False)
17574
81a033bb29bc store: let _auxencode() return the list of path segments
Adrian Buehlmann <adrian@cadifra.com>
parents: 17573
diff changeset
236 ['.com1com2', 'lp~749.lpt4.lpt1', 'conprn', 'com0', 'lpt0', 'foo~2e']
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 33412
diff changeset
237 >>> _auxencode([b'foo. '], True)
17574
81a033bb29bc store: let _auxencode() return the list of path segments
Adrian Buehlmann <adrian@cadifra.com>
parents: 17573
diff changeset
238 ['foo.~20']
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 33412
diff changeset
239 >>> _auxencode([b' .foo'], True)
17574
81a033bb29bc store: let _auxencode() return the list of path segments
Adrian Buehlmann <adrian@cadifra.com>
parents: 17573
diff changeset
240 ['~20.foo']
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45483
diff changeset
241 """
17589
b11024849db6 store: parameter path of _auxencode is now a list of strings
Adrian Buehlmann <adrian@cadifra.com>
parents: 17588
diff changeset
242 for i, n in enumerate(path):
17572
b644287e79a8 store: unindent most of the contents of the for loop in _auxencode()
Adrian Buehlmann <adrian@cadifra.com>
parents: 17571
diff changeset
243 if not n:
b644287e79a8 store: unindent most of the contents of the for loop in _auxencode()
Adrian Buehlmann <adrian@cadifra.com>
parents: 17571
diff changeset
244 continue
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
245 if dotencode and n[0] in b'. ':
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
246 n = b"~%02x" % ord(n[0:1]) + n[1:]
17589
b11024849db6 store: parameter path of _auxencode is now a list of strings
Adrian Buehlmann <adrian@cadifra.com>
parents: 17588
diff changeset
247 path[i] = n
17572
b644287e79a8 store: unindent most of the contents of the for loop in _auxencode()
Adrian Buehlmann <adrian@cadifra.com>
parents: 17571
diff changeset
248 else:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
249 l = n.find(b'.')
17572
b644287e79a8 store: unindent most of the contents of the for loop in _auxencode()
Adrian Buehlmann <adrian@cadifra.com>
parents: 17571
diff changeset
250 if l == -1:
b644287e79a8 store: unindent most of the contents of the for loop in _auxencode()
Adrian Buehlmann <adrian@cadifra.com>
parents: 17571
diff changeset
251 l = len(n)
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
252 if (l == 3 and n[:3] in _winres3) or (
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
253 l == 4
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
254 and n[3:4] <= b'9'
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
255 and n[3:4] >= b'1'
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
256 and n[:3] in _winres4
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
257 ):
17572
b644287e79a8 store: unindent most of the contents of the for loop in _auxencode()
Adrian Buehlmann <adrian@cadifra.com>
parents: 17571
diff changeset
258 # encode third letter ('aux' -> 'au~78')
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
259 ec = b"~%02x" % ord(n[2:3])
17572
b644287e79a8 store: unindent most of the contents of the for loop in _auxencode()
Adrian Buehlmann <adrian@cadifra.com>
parents: 17571
diff changeset
260 n = n[0:2] + ec + n[3:]
17589
b11024849db6 store: parameter path of _auxencode is now a list of strings
Adrian Buehlmann <adrian@cadifra.com>
parents: 17588
diff changeset
261 path[i] = n
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
262 if n[-1] in b'. ':
17572
b644287e79a8 store: unindent most of the contents of the for loop in _auxencode()
Adrian Buehlmann <adrian@cadifra.com>
parents: 17571
diff changeset
263 # encode last period or space ('foo...' -> 'foo..~2e')
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
264 path[i] = n[:-1] + b"~%02x" % ord(n[-1:])
17589
b11024849db6 store: parameter path of _auxencode is now a list of strings
Adrian Buehlmann <adrian@cadifra.com>
parents: 17588
diff changeset
265 return path
7229
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
266
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
267
14288
00a0ab08f986 store: change names to comply with project coding standards
Adrian Buehlmann <adrian@cadifra.com>
parents: 14194
diff changeset
268 _maxstorepathlen = 120
00a0ab08f986 store: change names to comply with project coding standards
Adrian Buehlmann <adrian@cadifra.com>
parents: 14194
diff changeset
269 _dirprefixlen = 8
00a0ab08f986 store: change names to comply with project coding standards
Adrian Buehlmann <adrian@cadifra.com>
parents: 14194
diff changeset
270 _maxshortdirslen = 8 * (_dirprefixlen + 1) - 4
17610
d0afa149e059 store: refactor hashed encoding into its own function
Bryan O'Sullivan <bryano@fb.com>
parents: 17609
diff changeset
271
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
272
17610
d0afa149e059 store: refactor hashed encoding into its own function
Bryan O'Sullivan <bryano@fb.com>
parents: 17609
diff changeset
273 def _hashencode(path, dotencode):
46113
59fa3890d40a node: import symbols explicitly
Joerg Sonnenberger <joerg@bec.de>
parents: 45942
diff changeset
274 digest = hex(hashutil.sha1(path).digest())
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
275 le = lowerencode(path[5:]).split(b'/') # skips prefix 'data/' or 'meta/'
17610
d0afa149e059 store: refactor hashed encoding into its own function
Bryan O'Sullivan <bryano@fb.com>
parents: 17609
diff changeset
276 parts = _auxencode(le, dotencode)
d0afa149e059 store: refactor hashed encoding into its own function
Bryan O'Sullivan <bryano@fb.com>
parents: 17609
diff changeset
277 basename = parts[-1]
d0afa149e059 store: refactor hashed encoding into its own function
Bryan O'Sullivan <bryano@fb.com>
parents: 17609
diff changeset
278 _root, ext = os.path.splitext(basename)
d0afa149e059 store: refactor hashed encoding into its own function
Bryan O'Sullivan <bryano@fb.com>
parents: 17609
diff changeset
279 sdirs = []
d0afa149e059 store: refactor hashed encoding into its own function
Bryan O'Sullivan <bryano@fb.com>
parents: 17609
diff changeset
280 sdirslen = 0
d0afa149e059 store: refactor hashed encoding into its own function
Bryan O'Sullivan <bryano@fb.com>
parents: 17609
diff changeset
281 for p in parts[:-1]:
d0afa149e059 store: refactor hashed encoding into its own function
Bryan O'Sullivan <bryano@fb.com>
parents: 17609
diff changeset
282 d = p[:_dirprefixlen]
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
283 if d[-1] in b'. ':
17610
d0afa149e059 store: refactor hashed encoding into its own function
Bryan O'Sullivan <bryano@fb.com>
parents: 17609
diff changeset
284 # Windows can't access dirs ending in period or space
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
285 d = d[:-1] + b'_'
17610
d0afa149e059 store: refactor hashed encoding into its own function
Bryan O'Sullivan <bryano@fb.com>
parents: 17609
diff changeset
286 if sdirslen == 0:
d0afa149e059 store: refactor hashed encoding into its own function
Bryan O'Sullivan <bryano@fb.com>
parents: 17609
diff changeset
287 t = len(d)
d0afa149e059 store: refactor hashed encoding into its own function
Bryan O'Sullivan <bryano@fb.com>
parents: 17609
diff changeset
288 else:
d0afa149e059 store: refactor hashed encoding into its own function
Bryan O'Sullivan <bryano@fb.com>
parents: 17609
diff changeset
289 t = sdirslen + 1 + len(d)
d0afa149e059 store: refactor hashed encoding into its own function
Bryan O'Sullivan <bryano@fb.com>
parents: 17609
diff changeset
290 if t > _maxshortdirslen:
d0afa149e059 store: refactor hashed encoding into its own function
Bryan O'Sullivan <bryano@fb.com>
parents: 17609
diff changeset
291 break
d0afa149e059 store: refactor hashed encoding into its own function
Bryan O'Sullivan <bryano@fb.com>
parents: 17609
diff changeset
292 sdirs.append(d)
d0afa149e059 store: refactor hashed encoding into its own function
Bryan O'Sullivan <bryano@fb.com>
parents: 17609
diff changeset
293 sdirslen = t
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
294 dirs = b'/'.join(sdirs)
17610
d0afa149e059 store: refactor hashed encoding into its own function
Bryan O'Sullivan <bryano@fb.com>
parents: 17609
diff changeset
295 if len(dirs) > 0:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
296 dirs += b'/'
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
297 res = b'dh/' + dirs + digest + ext
17610
d0afa149e059 store: refactor hashed encoding into its own function
Bryan O'Sullivan <bryano@fb.com>
parents: 17609
diff changeset
298 spaceleft = _maxstorepathlen - len(res)
d0afa149e059 store: refactor hashed encoding into its own function
Bryan O'Sullivan <bryano@fb.com>
parents: 17609
diff changeset
299 if spaceleft > 0:
d0afa149e059 store: refactor hashed encoding into its own function
Bryan O'Sullivan <bryano@fb.com>
parents: 17609
diff changeset
300 filler = basename[:spaceleft]
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
301 res = b'dh/' + dirs + filler + digest + ext
17610
d0afa149e059 store: refactor hashed encoding into its own function
Bryan O'Sullivan <bryano@fb.com>
parents: 17609
diff changeset
302 return res
d0afa149e059 store: refactor hashed encoding into its own function
Bryan O'Sullivan <bryano@fb.com>
parents: 17609
diff changeset
303
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
304
17590
eb0884680f5c store: eliminate one level of lambda functions on _hybridencode
Adrian Buehlmann <adrian@cadifra.com>
parents: 17589
diff changeset
305 def _hybridencode(path, dotencode):
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45483
diff changeset
306 """encodes path with a length limit
7229
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
307
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
308 Encodes all paths that begin with 'data/', according to the following.
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
309
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
310 Default encoding (reversible):
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
311
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
312 Encodes all uppercase letters 'X' as '_x'. All reserved or illegal
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
313 characters are encoded as '~xx', where xx is the two digit hex code
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
314 of the character (see encodefilename).
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
315 Relevant path components consisting of Windows reserved filenames are
17738
b8424c92ba2b spelling: fix minor spell checker issues
Mads Kiilerich <mads@kiilerich.com>
parents: 17731
diff changeset
316 masked by encoding the third character ('aux' -> 'au~78', see _auxencode).
7229
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
317
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
318 Hashed encoding (not reversible):
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
319
14288
00a0ab08f986 store: change names to comply with project coding standards
Adrian Buehlmann <adrian@cadifra.com>
parents: 14194
diff changeset
320 If the default-encoded path is longer than _maxstorepathlen, a
7229
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
321 non-reversible hybrid hashing of the path is done instead.
14288
00a0ab08f986 store: change names to comply with project coding standards
Adrian Buehlmann <adrian@cadifra.com>
parents: 14194
diff changeset
322 This encoding uses up to _dirprefixlen characters of all directory
7229
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
323 levels of the lowerencoded path, but not more levels than can fit into
14288
00a0ab08f986 store: change names to comply with project coding standards
Adrian Buehlmann <adrian@cadifra.com>
parents: 14194
diff changeset
324 _maxshortdirslen.
7229
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
325 Then follows the filler followed by the sha digest of the full path.
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
326 The filler is the beginning of the basename of the lowerencoded path
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
327 (the basename is everything after the last path separator). The filler
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
328 is as long as possible, filling in characters from the basename until
14288
00a0ab08f986 store: change names to comply with project coding standards
Adrian Buehlmann <adrian@cadifra.com>
parents: 14194
diff changeset
329 the encoded path has _maxstorepathlen characters (or all chars of the
00a0ab08f986 store: change names to comply with project coding standards
Adrian Buehlmann <adrian@cadifra.com>
parents: 14194
diff changeset
330 basename have been taken).
7229
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
331 The extension (e.g. '.i' or '.d') is preserved.
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
332
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
333 The string 'data/' at the beginning is replaced with 'dh/', if the hashed
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
334 encoding was used.
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45483
diff changeset
335 """
17609
cbc180cfd60b store: reuse direncoded path in _hybridencode
Adrian Buehlmann <adrian@cadifra.com>
parents: 17608
diff changeset
336 path = encodedir(path)
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
337 ef = _encodefname(path).split(b'/')
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
338 res = b'/'.join(_auxencode(ef, dotencode))
14288
00a0ab08f986 store: change names to comply with project coding standards
Adrian Buehlmann <adrian@cadifra.com>
parents: 14194
diff changeset
339 if len(res) > _maxstorepathlen:
17610
d0afa149e059 store: refactor hashed encoding into its own function
Bryan O'Sullivan <bryano@fb.com>
parents: 17609
diff changeset
340 res = _hashencode(path, dotencode)
7229
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
341 return res
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
342
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
343
17624
ae103510f6aa store: add a fallback _pathencode Python function
Adrian Buehlmann <adrian@cadifra.com>
parents: 17623
diff changeset
344 def _pathencode(path):
18435
8c019d2fd7c0 store: switch to C-based hashed path encoding
Bryan O'Sullivan <bryano@fb.com>
parents: 18430
diff changeset
345 de = encodedir(path)
17693
0c6de45e1212 store: optimize _pathencode by checking the length of the unencoded path
Adrian Buehlmann <adrian@cadifra.com>
parents: 17653
diff changeset
346 if len(path) > _maxstorepathlen:
18435
8c019d2fd7c0 store: switch to C-based hashed path encoding
Bryan O'Sullivan <bryano@fb.com>
parents: 18430
diff changeset
347 return _hashencode(de, True)
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
348 ef = _encodefname(de).split(b'/')
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
349 res = b'/'.join(_auxencode(ef, True))
17624
ae103510f6aa store: add a fallback _pathencode Python function
Adrian Buehlmann <adrian@cadifra.com>
parents: 17623
diff changeset
350 if len(res) > _maxstorepathlen:
18435
8c019d2fd7c0 store: switch to C-based hashed path encoding
Bryan O'Sullivan <bryano@fb.com>
parents: 18430
diff changeset
351 return _hashencode(de, True)
17624
ae103510f6aa store: add a fallback _pathencode Python function
Adrian Buehlmann <adrian@cadifra.com>
parents: 17623
diff changeset
352 return res
ae103510f6aa store: add a fallback _pathencode Python function
Adrian Buehlmann <adrian@cadifra.com>
parents: 17623
diff changeset
353
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
354
17624
ae103510f6aa store: add a fallback _pathencode Python function
Adrian Buehlmann <adrian@cadifra.com>
parents: 17623
diff changeset
355 _pathencode = getattr(parsers, 'pathencode', _pathencode)
ae103510f6aa store: add a fallback _pathencode Python function
Adrian Buehlmann <adrian@cadifra.com>
parents: 17623
diff changeset
356
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
357
17623
448e6ed7c557 store: move _plainhybridencode and _dothybridencode higher up in the file
Adrian Buehlmann <adrian@cadifra.com>
parents: 17621
diff changeset
358 def _plainhybridencode(f):
448e6ed7c557 store: move _plainhybridencode and _dothybridencode higher up in the file
Adrian Buehlmann <adrian@cadifra.com>
parents: 17621
diff changeset
359 return _hybridencode(f, False)
448e6ed7c557 store: move _plainhybridencode and _dothybridencode higher up in the file
Adrian Buehlmann <adrian@cadifra.com>
parents: 17621
diff changeset
360
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
361
17726
7cb7e17c23b2 store: invoke "os.stat()" for "createmode" initialization via vfs
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17725
diff changeset
362 def _calcmode(vfs):
6898
69aeaaaf6e07 store: simplify class hierarchy
Matt Mackall <mpm@selenic.com>
parents: 6897
diff changeset
363 try:
69aeaaaf6e07 store: simplify class hierarchy
Matt Mackall <mpm@selenic.com>
parents: 6897
diff changeset
364 # files in .hg/ will be created using this mode
17726
7cb7e17c23b2 store: invoke "os.stat()" for "createmode" initialization via vfs
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17725
diff changeset
365 mode = vfs.stat().st_mode
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
366 # avoid some useless chmods
25658
e93036747902 global: mass rewrite to use modern octal syntax
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25091
diff changeset
367 if (0o777 & ~util.umask) == (0o777 & mode):
6898
69aeaaaf6e07 store: simplify class hierarchy
Matt Mackall <mpm@selenic.com>
parents: 6897
diff changeset
368 mode = None
69aeaaaf6e07 store: simplify class hierarchy
Matt Mackall <mpm@selenic.com>
parents: 6897
diff changeset
369 except OSError:
69aeaaaf6e07 store: simplify class hierarchy
Matt Mackall <mpm@selenic.com>
parents: 6897
diff changeset
370 mode = None
69aeaaaf6e07 store: simplify class hierarchy
Matt Mackall <mpm@selenic.com>
parents: 6897
diff changeset
371 return mode
69aeaaaf6e07 store: simplify class hierarchy
Matt Mackall <mpm@selenic.com>
parents: 6897
diff changeset
372
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
373
45351
909dafff6a78 store: refactor space delimited list to proper data structure
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
374 _data = [
909dafff6a78 store: refactor space delimited list to proper data structure
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
375 b'bookmarks',
909dafff6a78 store: refactor space delimited list to proper data structure
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
376 b'narrowspec',
909dafff6a78 store: refactor space delimited list to proper data structure
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
377 b'data',
909dafff6a78 store: refactor space delimited list to proper data structure
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
378 b'meta',
909dafff6a78 store: refactor space delimited list to proper data structure
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
379 b'00manifest.d',
909dafff6a78 store: refactor space delimited list to proper data structure
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
380 b'00manifest.i',
909dafff6a78 store: refactor space delimited list to proper data structure
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
381 b'00changelog.d',
909dafff6a78 store: refactor space delimited list to proper data structure
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
382 b'00changelog.i',
909dafff6a78 store: refactor space delimited list to proper data structure
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
383 b'phaseroots',
909dafff6a78 store: refactor space delimited list to proper data structure
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
384 b'obsstore',
45483
d252f51ab032 share: introduce config option to store requires in .hg/store
Pulkit Goyal <7895pulkit@gmail.com>
parents: 45351
diff changeset
385 b'requires',
45351
909dafff6a78 store: refactor space delimited list to proper data structure
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
386 ]
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
387
50469
53af67c70af0 store: cleanup what is recognized as a revlog file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50468
diff changeset
388 REVLOG_FILES_MAIN_EXT = (b'.i',)
47324
0a3fa41fa719 revlogv2: use a unique filename for data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47234
diff changeset
389 REVLOG_FILES_OTHER_EXT = (
0a3fa41fa719 revlogv2: use a unique filename for data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47234
diff changeset
390 b'.idx',
0a3fa41fa719 revlogv2: use a unique filename for data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47234
diff changeset
391 b'.d',
0a3fa41fa719 revlogv2: use a unique filename for data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47234
diff changeset
392 b'.dat',
0a3fa41fa719 revlogv2: use a unique filename for data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47234
diff changeset
393 b'.n',
0a3fa41fa719 revlogv2: use a unique filename for data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47234
diff changeset
394 b'.nd',
47389
e6292eb33384 revlog: store sidedata in their own file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47324
diff changeset
395 b'.sda',
47324
0a3fa41fa719 revlogv2: use a unique filename for data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47234
diff changeset
396 )
50479
5217e36356bb store: add logic to group revlog file together
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50478
diff changeset
397 # file extension that also use a `-SOMELONGIDHASH.ext` form
5217e36356bb store: add logic to group revlog file together
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50478
diff changeset
398 REVLOG_FILES_LONG_EXT = (
5217e36356bb store: add logic to group revlog file together
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50478
diff changeset
399 b'.nd',
5217e36356bb store: add logic to group revlog file together
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50478
diff changeset
400 b'.idx',
5217e36356bb store: add logic to group revlog file together
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50478
diff changeset
401 b'.dat',
5217e36356bb store: add logic to group revlog file together
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50478
diff changeset
402 b'.sda',
5217e36356bb store: add logic to group revlog file together
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50478
diff changeset
403 )
46989
aed6ceaad6d7 streamclone: treat volatile file as "fullfile"
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46895
diff changeset
404 # files that are "volatile" and might change between listing and streaming
aed6ceaad6d7 streamclone: treat volatile file as "fullfile"
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46895
diff changeset
405 #
aed6ceaad6d7 streamclone: treat volatile file as "fullfile"
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46895
diff changeset
406 # note: the ".nd" file are nodemap data and won't "change" but they might be
aed6ceaad6d7 streamclone: treat volatile file as "fullfile"
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46895
diff changeset
407 # deleted.
aed6ceaad6d7 streamclone: treat volatile file as "fullfile"
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46895
diff changeset
408 REVLOG_FILES_VOLATILE_EXT = (b'.n', b'.nd')
46895
6085b7f1536d store: also return some information about the type of file `walk` found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46854
diff changeset
409
46990
0b569c75d180 store: exclude `undo.` nodemap's file from `walk`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
410 # some exception to the above matching
47659
f030c7d22032 walk: no longer ignore revlogs of files starting with `undo.` (issue6542)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47443
diff changeset
411 #
f030c7d22032 walk: no longer ignore revlogs of files starting with `undo.` (issue6542)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47443
diff changeset
412 # XXX This is currently not in use because of issue6542
50319
4be9ecc982e1 py3: fix for Python 3.12 emitting SyntaxWarning on invalid escape sequences
Mads Kiilerich <mads@kiilerich.com>
parents: 50315
diff changeset
413 EXCLUDED = re.compile(br'.*undo\.[^/]+\.(nd?|i)$')
46990
0b569c75d180 store: exclude `undo.` nodemap's file from `walk`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
414
46895
6085b7f1536d store: also return some information about the type of file `walk` found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46854
diff changeset
415
6085b7f1536d store: also return some information about the type of file `walk` found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46854
diff changeset
416 def is_revlog(f, kind, st):
6085b7f1536d store: also return some information about the type of file `walk` found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46854
diff changeset
417 if kind != stat.S_IFREG:
6085b7f1536d store: also return some information about the type of file `walk` found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46854
diff changeset
418 return None
6085b7f1536d store: also return some information about the type of file `walk` found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46854
diff changeset
419 return revlog_type(f)
6085b7f1536d store: also return some information about the type of file `walk` found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46854
diff changeset
420
6085b7f1536d store: also return some information about the type of file `walk` found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46854
diff changeset
421
6085b7f1536d store: also return some information about the type of file `walk` found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46854
diff changeset
422 def revlog_type(f):
47659
f030c7d22032 walk: no longer ignore revlogs of files starting with `undo.` (issue6542)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47443
diff changeset
423 # XXX we need to filter `undo.` created by the transaction here, however
f030c7d22032 walk: no longer ignore revlogs of files starting with `undo.` (issue6542)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47443
diff changeset
424 # being naive about it also filter revlog for `undo.*` files, leading to
f030c7d22032 walk: no longer ignore revlogs of files starting with `undo.` (issue6542)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47443
diff changeset
425 # issue6542. So we no longer use EXCLUDED.
f030c7d22032 walk: no longer ignore revlogs of files starting with `undo.` (issue6542)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47443
diff changeset
426 if f.endswith(REVLOG_FILES_MAIN_EXT):
46895
6085b7f1536d store: also return some information about the type of file `walk` found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46854
diff changeset
427 return FILEFLAGS_REVLOG_MAIN
47659
f030c7d22032 walk: no longer ignore revlogs of files starting with `undo.` (issue6542)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47443
diff changeset
428 elif f.endswith(REVLOG_FILES_OTHER_EXT):
46989
aed6ceaad6d7 streamclone: treat volatile file as "fullfile"
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46895
diff changeset
429 t = FILETYPE_FILELOG_OTHER
aed6ceaad6d7 streamclone: treat volatile file as "fullfile"
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46895
diff changeset
430 if f.endswith(REVLOG_FILES_VOLATILE_EXT):
aed6ceaad6d7 streamclone: treat volatile file as "fullfile"
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46895
diff changeset
431 t |= FILEFLAGS_VOLATILE
aed6ceaad6d7 streamclone: treat volatile file as "fullfile"
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46895
diff changeset
432 return t
47324
0a3fa41fa719 revlogv2: use a unique filename for data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47234
diff changeset
433 return None
46383
374d7fff7cb5 store: use `endswith` to detect revlog extension
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46322
diff changeset
434
6903
0642d9d7ec80 clone: get a list of files to clone from store
Matt Mackall <mpm@selenic.com>
parents: 6902
diff changeset
435
46895
6085b7f1536d store: also return some information about the type of file `walk` found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46854
diff changeset
436 # the file is part of changelog data
6085b7f1536d store: also return some information about the type of file `walk` found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46854
diff changeset
437 FILEFLAGS_CHANGELOG = 1 << 13
6085b7f1536d store: also return some information about the type of file `walk` found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46854
diff changeset
438 # the file is part of manifest data
6085b7f1536d store: also return some information about the type of file `walk` found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46854
diff changeset
439 FILEFLAGS_MANIFESTLOG = 1 << 12
6085b7f1536d store: also return some information about the type of file `walk` found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46854
diff changeset
440 # the file is part of filelog data
6085b7f1536d store: also return some information about the type of file `walk` found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46854
diff changeset
441 FILEFLAGS_FILELOG = 1 << 11
6085b7f1536d store: also return some information about the type of file `walk` found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46854
diff changeset
442 # file that are not directly part of a revlog
6085b7f1536d store: also return some information about the type of file `walk` found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46854
diff changeset
443 FILEFLAGS_OTHER = 1 << 10
6085b7f1536d store: also return some information about the type of file `walk` found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46854
diff changeset
444
6085b7f1536d store: also return some information about the type of file `walk` found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46854
diff changeset
445 # the main entry point for a revlog
6085b7f1536d store: also return some information about the type of file `walk` found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46854
diff changeset
446 FILEFLAGS_REVLOG_MAIN = 1 << 1
6085b7f1536d store: also return some information about the type of file `walk` found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46854
diff changeset
447 # a secondary file for a revlog
6085b7f1536d store: also return some information about the type of file `walk` found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46854
diff changeset
448 FILEFLAGS_REVLOG_OTHER = 1 << 0
6085b7f1536d store: also return some information about the type of file `walk` found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46854
diff changeset
449
46989
aed6ceaad6d7 streamclone: treat volatile file as "fullfile"
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46895
diff changeset
450 # files that are "volatile" and might change between listing and streaming
aed6ceaad6d7 streamclone: treat volatile file as "fullfile"
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46895
diff changeset
451 FILEFLAGS_VOLATILE = 1 << 20
aed6ceaad6d7 streamclone: treat volatile file as "fullfile"
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46895
diff changeset
452
46895
6085b7f1536d store: also return some information about the type of file `walk` found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46854
diff changeset
453 FILETYPE_CHANGELOG_MAIN = FILEFLAGS_CHANGELOG | FILEFLAGS_REVLOG_MAIN
6085b7f1536d store: also return some information about the type of file `walk` found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46854
diff changeset
454 FILETYPE_CHANGELOG_OTHER = FILEFLAGS_CHANGELOG | FILEFLAGS_REVLOG_OTHER
6085b7f1536d store: also return some information about the type of file `walk` found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46854
diff changeset
455 FILETYPE_MANIFESTLOG_MAIN = FILEFLAGS_MANIFESTLOG | FILEFLAGS_REVLOG_MAIN
6085b7f1536d store: also return some information about the type of file `walk` found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46854
diff changeset
456 FILETYPE_MANIFESTLOG_OTHER = FILEFLAGS_MANIFESTLOG | FILEFLAGS_REVLOG_OTHER
6085b7f1536d store: also return some information about the type of file `walk` found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46854
diff changeset
457 FILETYPE_FILELOG_MAIN = FILEFLAGS_FILELOG | FILEFLAGS_REVLOG_MAIN
6085b7f1536d store: also return some information about the type of file `walk` found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46854
diff changeset
458 FILETYPE_FILELOG_OTHER = FILEFLAGS_FILELOG | FILEFLAGS_REVLOG_OTHER
6085b7f1536d store: also return some information about the type of file `walk` found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46854
diff changeset
459 FILETYPE_OTHER = FILEFLAGS_OTHER
37409
4c15bee42e9c store: make file filtering during walk configurable
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35582
diff changeset
460
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
461
50638
8fc10bfd9887 store: declare a `files` method on BaseStoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50632
diff changeset
462 @attr.s(slots=True)
8fc10bfd9887 store: declare a `files` method on BaseStoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50632
diff changeset
463 class StoreFile:
8fc10bfd9887 store: declare a `files` method on BaseStoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50632
diff changeset
464 """a file matching a store entry"""
8fc10bfd9887 store: declare a `files` method on BaseStoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50632
diff changeset
465
8fc10bfd9887 store: declare a `files` method on BaseStoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50632
diff changeset
466 unencoded_path = attr.ib()
8fc10bfd9887 store: declare a `files` method on BaseStoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50632
diff changeset
467 _file_size = attr.ib(default=None)
8fc10bfd9887 store: declare a `files` method on BaseStoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50632
diff changeset
468 is_volatile = attr.ib(default=False)
8fc10bfd9887 store: declare a `files` method on BaseStoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50632
diff changeset
469
8fc10bfd9887 store: declare a `files` method on BaseStoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50632
diff changeset
470 def file_size(self, vfs):
8fc10bfd9887 store: declare a `files` method on BaseStoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50632
diff changeset
471 if self._file_size is None:
50639
5e60abf811f3 stream-clone: make it the responsability of the store entry to stream content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50638
diff changeset
472 if vfs is None:
5e60abf811f3 stream-clone: make it the responsability of the store entry to stream content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50638
diff changeset
473 msg = b"calling vfs-less file_size without prior call: %s"
5e60abf811f3 stream-clone: make it the responsability of the store entry to stream content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50638
diff changeset
474 msg %= self.unencoded_path
5e60abf811f3 stream-clone: make it the responsability of the store entry to stream content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50638
diff changeset
475 raise error.ProgrammingError(msg)
50638
8fc10bfd9887 store: declare a `files` method on BaseStoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50632
diff changeset
476 try:
8fc10bfd9887 store: declare a `files` method on BaseStoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50632
diff changeset
477 self._file_size = vfs.stat(self.unencoded_path).st_size
8fc10bfd9887 store: declare a `files` method on BaseStoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50632
diff changeset
478 except FileNotFoundError:
8fc10bfd9887 store: declare a `files` method on BaseStoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50632
diff changeset
479 self._file_size = 0
8fc10bfd9887 store: declare a `files` method on BaseStoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50632
diff changeset
480 return self._file_size
8fc10bfd9887 store: declare a `files` method on BaseStoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50632
diff changeset
481
50639
5e60abf811f3 stream-clone: make it the responsability of the store entry to stream content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50638
diff changeset
482 def get_stream(self, vfs, copies):
5e60abf811f3 stream-clone: make it the responsability of the store entry to stream content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50638
diff changeset
483 """return data "stream" information for this file
5e60abf811f3 stream-clone: make it the responsability of the store entry to stream content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50638
diff changeset
484
5e60abf811f3 stream-clone: make it the responsability of the store entry to stream content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50638
diff changeset
485 (unencoded_file_path, content_iterator, content_size)
5e60abf811f3 stream-clone: make it the responsability of the store entry to stream content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50638
diff changeset
486 """
5e60abf811f3 stream-clone: make it the responsability of the store entry to stream content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50638
diff changeset
487 size = self.file_size(None)
5e60abf811f3 stream-clone: make it the responsability of the store entry to stream content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50638
diff changeset
488
5e60abf811f3 stream-clone: make it the responsability of the store entry to stream content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50638
diff changeset
489 def get_stream():
5e60abf811f3 stream-clone: make it the responsability of the store entry to stream content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50638
diff changeset
490 actual_path = copies[vfs.join(self.unencoded_path)]
5e60abf811f3 stream-clone: make it the responsability of the store entry to stream content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50638
diff changeset
491 with open(actual_path, 'rb') as fp:
5e60abf811f3 stream-clone: make it the responsability of the store entry to stream content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50638
diff changeset
492 yield None # ready to stream
5e60abf811f3 stream-clone: make it the responsability of the store entry to stream content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50638
diff changeset
493 if size <= 65536:
5e60abf811f3 stream-clone: make it the responsability of the store entry to stream content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50638
diff changeset
494 yield fp.read(size)
5e60abf811f3 stream-clone: make it the responsability of the store entry to stream content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50638
diff changeset
495 else:
5e60abf811f3 stream-clone: make it the responsability of the store entry to stream content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50638
diff changeset
496 yield from util.filechunkiter(fp, limit=size)
5e60abf811f3 stream-clone: make it the responsability of the store entry to stream content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50638
diff changeset
497
5e60abf811f3 stream-clone: make it the responsability of the store entry to stream content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50638
diff changeset
498 s = get_stream()
5e60abf811f3 stream-clone: make it the responsability of the store entry to stream content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50638
diff changeset
499 next(s)
5e60abf811f3 stream-clone: make it the responsability of the store entry to stream content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50638
diff changeset
500 return (self.unencoded_path, s, size)
5e60abf811f3 stream-clone: make it the responsability of the store entry to stream content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50638
diff changeset
501
50638
8fc10bfd9887 store: declare a `files` method on BaseStoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50632
diff changeset
502
50474
c37450a5f1dc store: have custom init for entries class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50473
diff changeset
503 @attr.s(slots=True, init=False)
50473
5a2fb64d38b2 store: use specialized class for store entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50472
diff changeset
504 class BaseStoreEntry:
50471
521fec115dad store: use a StoreEntry object instead of tuple for store files
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50469
diff changeset
505 """An entry in the store
521fec115dad store: use a StoreEntry object instead of tuple for store files
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50469
diff changeset
506
521fec115dad store: use a StoreEntry object instead of tuple for store files
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50469
diff changeset
507 This is returned by `store.walk` and represent some data in the store."""
521fec115dad store: use a StoreEntry object instead of tuple for store files
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50469
diff changeset
508
50638
8fc10bfd9887 store: declare a `files` method on BaseStoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50632
diff changeset
509 def files(self) -> List[StoreFile]:
8fc10bfd9887 store: declare a `files` method on BaseStoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50632
diff changeset
510 raise NotImplementedError
8fc10bfd9887 store: declare a `files` method on BaseStoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50632
diff changeset
511
50640
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50639
diff changeset
512 def get_streams(
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50639
diff changeset
513 self,
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50639
diff changeset
514 repo=None,
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50639
diff changeset
515 vfs=None,
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50639
diff changeset
516 copies=None,
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50639
diff changeset
517 max_changeset=None,
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50639
diff changeset
518 ):
50639
5e60abf811f3 stream-clone: make it the responsability of the store entry to stream content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50638
diff changeset
519 """return a list of data stream associated to files for this entry
5e60abf811f3 stream-clone: make it the responsability of the store entry to stream content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50638
diff changeset
520
5e60abf811f3 stream-clone: make it the responsability of the store entry to stream content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50638
diff changeset
521 return [(unencoded_file_path, content_iterator, content_size), …]
5e60abf811f3 stream-clone: make it the responsability of the store entry to stream content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50638
diff changeset
522 """
5e60abf811f3 stream-clone: make it the responsability of the store entry to stream content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50638
diff changeset
523 assert vfs is not None
5e60abf811f3 stream-clone: make it the responsability of the store entry to stream content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50638
diff changeset
524 return [f.get_stream(vfs, copies) for f in self.files()]
5e60abf811f3 stream-clone: make it the responsability of the store entry to stream content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50638
diff changeset
525
50496
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50495
diff changeset
526
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50495
diff changeset
527 @attr.s(slots=True, init=False)
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50495
diff changeset
528 class SimpleStoreEntry(BaseStoreEntry):
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50495
diff changeset
529 """A generic entry in the store"""
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50495
diff changeset
530
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50495
diff changeset
531 is_revlog = False
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50495
diff changeset
532
50495
ed8cda1c18e1 store: rename `unencoded_path` to `entry_path` for StoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50494
diff changeset
533 _entry_path = attr.ib()
50476
2b2284cf949b store: only access is_volatile information through the file object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50475
diff changeset
534 _is_volatile = attr.ib(default=False)
50475
7d4d2a160cf5 store: only access file_size information through the file object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50474
diff changeset
535 _file_size = attr.ib(default=None)
50631
b59e0a4f692f store: cache the `files()` return for store entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50629
diff changeset
536 _files = attr.ib(default=None)
50471
521fec115dad store: use a StoreEntry object instead of tuple for store files
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50469
diff changeset
537
50474
c37450a5f1dc store: have custom init for entries class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50473
diff changeset
538 def __init__(
c37450a5f1dc store: have custom init for entries class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50473
diff changeset
539 self,
50495
ed8cda1c18e1 store: rename `unencoded_path` to `entry_path` for StoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50494
diff changeset
540 entry_path,
50474
c37450a5f1dc store: have custom init for entries class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50473
diff changeset
541 is_volatile=False,
c37450a5f1dc store: have custom init for entries class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50473
diff changeset
542 file_size=None,
c37450a5f1dc store: have custom init for entries class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50473
diff changeset
543 ):
50496
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50495
diff changeset
544 super().__init__()
50495
ed8cda1c18e1 store: rename `unencoded_path` to `entry_path` for StoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50494
diff changeset
545 self._entry_path = entry_path
50476
2b2284cf949b store: only access is_volatile information through the file object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50475
diff changeset
546 self._is_volatile = is_volatile
50475
7d4d2a160cf5 store: only access file_size information through the file object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50474
diff changeset
547 self._file_size = file_size
50631
b59e0a4f692f store: cache the `files()` return for store entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50629
diff changeset
548 self._files = None
50474
c37450a5f1dc store: have custom init for entries class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50473
diff changeset
549
50638
8fc10bfd9887 store: declare a `files` method on BaseStoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50632
diff changeset
550 def files(self) -> List[StoreFile]:
50631
b59e0a4f692f store: cache the `files()` return for store entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50629
diff changeset
551 if self._files is None:
b59e0a4f692f store: cache the `files()` return for store entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50629
diff changeset
552 self._files = [
b59e0a4f692f store: cache the `files()` return for store entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50629
diff changeset
553 StoreFile(
b59e0a4f692f store: cache the `files()` return for store entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50629
diff changeset
554 unencoded_path=self._entry_path,
b59e0a4f692f store: cache the `files()` return for store entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50629
diff changeset
555 file_size=self._file_size,
b59e0a4f692f store: cache the `files()` return for store entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50629
diff changeset
556 is_volatile=self._is_volatile,
b59e0a4f692f store: cache the `files()` return for store entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50629
diff changeset
557 )
b59e0a4f692f store: cache the `files()` return for store entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50629
diff changeset
558 ]
b59e0a4f692f store: cache the `files()` return for store entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50629
diff changeset
559 return self._files
50472
9fdc28e21b68 store: introduce a EntryFile object to actually access file info
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50471
diff changeset
560
9fdc28e21b68 store: introduce a EntryFile object to actually access file info
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50471
diff changeset
561
50474
c37450a5f1dc store: have custom init for entries class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50473
diff changeset
562 @attr.s(slots=True, init=False)
50473
5a2fb64d38b2 store: use specialized class for store entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50472
diff changeset
563 class RevlogStoreEntry(BaseStoreEntry):
5a2fb64d38b2 store: use specialized class for store entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50472
diff changeset
564 """A revlog entry in the store"""
5a2fb64d38b2 store: use specialized class for store entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50472
diff changeset
565
5a2fb64d38b2 store: use specialized class for store entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50472
diff changeset
566 is_revlog = True
50496
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50495
diff changeset
567
50473
5a2fb64d38b2 store: use specialized class for store entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50472
diff changeset
568 revlog_type = attr.ib(default=None)
50483
60e613f6a229 store: add a `target_id` attribute on RevlogStoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50482
diff changeset
569 target_id = attr.ib(default=None)
50496
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50495
diff changeset
570 _path_prefix = attr.ib(default=None)
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50495
diff changeset
571 _details = attr.ib(default=None)
50631
b59e0a4f692f store: cache the `files()` return for store entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50629
diff changeset
572 _files = attr.ib(default=None)
50473
5a2fb64d38b2 store: use specialized class for store entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50472
diff changeset
573
50474
c37450a5f1dc store: have custom init for entries class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50473
diff changeset
574 def __init__(
c37450a5f1dc store: have custom init for entries class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50473
diff changeset
575 self,
c37450a5f1dc store: have custom init for entries class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50473
diff changeset
576 revlog_type,
50496
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50495
diff changeset
577 path_prefix,
50483
60e613f6a229 store: add a `target_id` attribute on RevlogStoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50482
diff changeset
578 target_id,
50496
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50495
diff changeset
579 details,
50474
c37450a5f1dc store: have custom init for entries class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50473
diff changeset
580 ):
50496
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50495
diff changeset
581 super().__init__()
50474
c37450a5f1dc store: have custom init for entries class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50473
diff changeset
582 self.revlog_type = revlog_type
50483
60e613f6a229 store: add a `target_id` attribute on RevlogStoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50482
diff changeset
583 self.target_id = target_id
50496
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50495
diff changeset
584 self._path_prefix = path_prefix
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50495
diff changeset
585 assert b'.i' in details, (path_prefix, details)
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50495
diff changeset
586 self._details = details
50631
b59e0a4f692f store: cache the `files()` return for store entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50629
diff changeset
587 self._files = None
50474
c37450a5f1dc store: have custom init for entries class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50473
diff changeset
588
50497
66c556968222 store: introduce boolean property for revlog type
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50496
diff changeset
589 @property
66c556968222 store: introduce boolean property for revlog type
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50496
diff changeset
590 def is_changelog(self):
66c556968222 store: introduce boolean property for revlog type
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50496
diff changeset
591 return self.revlog_type & FILEFLAGS_CHANGELOG
66c556968222 store: introduce boolean property for revlog type
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50496
diff changeset
592
66c556968222 store: introduce boolean property for revlog type
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50496
diff changeset
593 @property
66c556968222 store: introduce boolean property for revlog type
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50496
diff changeset
594 def is_manifestlog(self):
66c556968222 store: introduce boolean property for revlog type
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50496
diff changeset
595 return self.revlog_type & FILEFLAGS_MANIFESTLOG
66c556968222 store: introduce boolean property for revlog type
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50496
diff changeset
596
66c556968222 store: introduce boolean property for revlog type
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50496
diff changeset
597 @property
66c556968222 store: introduce boolean property for revlog type
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50496
diff changeset
598 def is_filelog(self):
66c556968222 store: introduce boolean property for revlog type
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50496
diff changeset
599 return self.revlog_type & FILEFLAGS_FILELOG
66c556968222 store: introduce boolean property for revlog type
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50496
diff changeset
600
50492
3473d18c029a store: introduce a main_file_path method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50483
diff changeset
601 def main_file_path(self):
3473d18c029a store: introduce a main_file_path method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50483
diff changeset
602 """unencoded path of the main revlog file"""
50496
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50495
diff changeset
603 return self._path_prefix + b'.i'
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50495
diff changeset
604
50638
8fc10bfd9887 store: declare a `files` method on BaseStoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50632
diff changeset
605 def files(self) -> List[StoreFile]:
50631
b59e0a4f692f store: cache the `files()` return for store entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50629
diff changeset
606 if self._files is None:
b59e0a4f692f store: cache the `files()` return for store entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50629
diff changeset
607 self._files = []
b59e0a4f692f store: cache the `files()` return for store entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50629
diff changeset
608 for ext in sorted(self._details, key=_ext_key):
b59e0a4f692f store: cache the `files()` return for store entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50629
diff changeset
609 path = self._path_prefix + ext
b59e0a4f692f store: cache the `files()` return for store entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50629
diff changeset
610 data = self._details[ext]
b59e0a4f692f store: cache the `files()` return for store entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50629
diff changeset
611 self._files.append(StoreFile(unencoded_path=path, **data))
b59e0a4f692f store: cache the `files()` return for store entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50629
diff changeset
612 return self._files
50492
3473d18c029a store: introduce a main_file_path method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50483
diff changeset
613
50640
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50639
diff changeset
614 def get_streams(
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50639
diff changeset
615 self,
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50639
diff changeset
616 repo=None,
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50639
diff changeset
617 vfs=None,
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50639
diff changeset
618 copies=None,
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50639
diff changeset
619 max_changeset=None,
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50639
diff changeset
620 ):
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50639
diff changeset
621 if repo is None or max_changeset is None:
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50639
diff changeset
622 return super().get_streams(
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50639
diff changeset
623 repo=repo,
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50639
diff changeset
624 vfs=vfs,
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50639
diff changeset
625 copies=copies,
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50639
diff changeset
626 max_changeset=max_changeset,
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50639
diff changeset
627 )
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50639
diff changeset
628 if any(k.endswith(b'.idx') for k in self._details.keys()):
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50639
diff changeset
629 # This use revlog-v2, ignore for now
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50639
diff changeset
630 return super().get_streams(
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50639
diff changeset
631 repo=repo,
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50639
diff changeset
632 vfs=vfs,
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50639
diff changeset
633 copies=copies,
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50639
diff changeset
634 max_changeset=max_changeset,
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50639
diff changeset
635 )
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50639
diff changeset
636 name_to_ext = {}
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50639
diff changeset
637 for ext in self._details.keys():
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50639
diff changeset
638 name_to_ext[self._path_prefix + ext] = ext
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50639
diff changeset
639 name_to_size = {}
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50639
diff changeset
640 for f in self.files():
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50639
diff changeset
641 name_to_size[f.unencoded_path] = f.file_size(None)
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50639
diff changeset
642 stream = [
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50639
diff changeset
643 f.get_stream(vfs, copies)
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50639
diff changeset
644 for f in self.files()
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50639
diff changeset
645 if name_to_ext[f.unencoded_path] not in (b'.d', b'.i')
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50639
diff changeset
646 ]
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50639
diff changeset
647
50641
5460424092e2 stream-clone: smoothly detect and handle a case were a revlog is split
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50640
diff changeset
648 is_inline = b'.d' not in self._details
5460424092e2 stream-clone: smoothly detect and handle a case were a revlog is split
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50640
diff changeset
649
50640
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50639
diff changeset
650 rl = self.get_revlog_instance(repo).get_revlog()
50641
5460424092e2 stream-clone: smoothly detect and handle a case were a revlog is split
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50640
diff changeset
651 rl_stream = rl.get_streams(max_changeset, force_inline=is_inline)
5460424092e2 stream-clone: smoothly detect and handle a case were a revlog is split
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50640
diff changeset
652
50640
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50639
diff changeset
653 for name, s, size in rl_stream:
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50639
diff changeset
654 if name_to_size.get(name, 0) != size:
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50639
diff changeset
655 msg = _(b"expected %d bytes but %d provided for %s")
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50639
diff changeset
656 msg %= name_to_size.get(name, 0), size, name
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50639
diff changeset
657 raise error.Abort(msg)
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50639
diff changeset
658 stream.extend(rl_stream)
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50639
diff changeset
659 files = self.files()
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50639
diff changeset
660 assert len(stream) == len(files), (
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50639
diff changeset
661 stream,
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50639
diff changeset
662 files,
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50639
diff changeset
663 self._path_prefix,
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50639
diff changeset
664 self.target_id,
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50639
diff changeset
665 )
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50639
diff changeset
666 return stream
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50639
diff changeset
667
50627
e1ee6910f6bc store: add a `get_revlog_instance` method on revlog entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50514
diff changeset
668 def get_revlog_instance(self, repo):
e1ee6910f6bc store: add a `get_revlog_instance` method on revlog entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50514
diff changeset
669 """Obtain a revlog instance from this store entry
e1ee6910f6bc store: add a `get_revlog_instance` method on revlog entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50514
diff changeset
670
e1ee6910f6bc store: add a `get_revlog_instance` method on revlog entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50514
diff changeset
671 An instance of the appropriate class is returned.
e1ee6910f6bc store: add a `get_revlog_instance` method on revlog entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50514
diff changeset
672 """
e1ee6910f6bc store: add a `get_revlog_instance` method on revlog entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50514
diff changeset
673 if self.is_changelog:
e1ee6910f6bc store: add a `get_revlog_instance` method on revlog entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50514
diff changeset
674 return changelog.changelog(repo.svfs)
e1ee6910f6bc store: add a `get_revlog_instance` method on revlog entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50514
diff changeset
675 elif self.is_manifestlog:
50628
3ea3767c23a4 store: do not drop the final `/` when creating manifestlog instance
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50627
diff changeset
676 mandir = self.target_id
50627
e1ee6910f6bc store: add a `get_revlog_instance` method on revlog entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50514
diff changeset
677 return manifest.manifestrevlog(
e1ee6910f6bc store: add a `get_revlog_instance` method on revlog entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50514
diff changeset
678 repo.nodeconstants, repo.svfs, tree=mandir
e1ee6910f6bc store: add a `get_revlog_instance` method on revlog entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50514
diff changeset
679 )
e1ee6910f6bc store: add a `get_revlog_instance` method on revlog entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50514
diff changeset
680 else:
e1ee6910f6bc store: add a `get_revlog_instance` method on revlog entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50514
diff changeset
681 return filelog.filelog(repo.svfs, self.target_id)
e1ee6910f6bc store: add a `get_revlog_instance` method on revlog entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50514
diff changeset
682
50473
5a2fb64d38b2 store: use specialized class for store entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50472
diff changeset
683
50479
5217e36356bb store: add logic to group revlog file together
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50478
diff changeset
684 def _gather_revlog(files_data):
5217e36356bb store: add logic to group revlog file together
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50478
diff changeset
685 """group files per revlog prefix
5217e36356bb store: add logic to group revlog file together
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50478
diff changeset
686
5217e36356bb store: add logic to group revlog file together
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50478
diff changeset
687 The returns a two level nested dict. The top level key is the revlog prefix
5217e36356bb store: add logic to group revlog file together
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50478
diff changeset
688 without extension, the second level is all the file "suffix" that were
5217e36356bb store: add logic to group revlog file together
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50478
diff changeset
689 seen for this revlog and arbitrary file data as value.
5217e36356bb store: add logic to group revlog file together
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50478
diff changeset
690 """
5217e36356bb store: add logic to group revlog file together
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50478
diff changeset
691 revlogs = collections.defaultdict(dict)
5217e36356bb store: add logic to group revlog file together
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50478
diff changeset
692 for u, value in files_data:
5217e36356bb store: add logic to group revlog file together
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50478
diff changeset
693 name, ext = _split_revlog_ext(u)
5217e36356bb store: add logic to group revlog file together
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50478
diff changeset
694 revlogs[name][ext] = value
5217e36356bb store: add logic to group revlog file together
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50478
diff changeset
695 return sorted(revlogs.items())
5217e36356bb store: add logic to group revlog file together
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50478
diff changeset
696
5217e36356bb store: add logic to group revlog file together
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50478
diff changeset
697
5217e36356bb store: add logic to group revlog file together
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50478
diff changeset
698 def _split_revlog_ext(filename):
5217e36356bb store: add logic to group revlog file together
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50478
diff changeset
699 """split the revlog file prefix from the variable extension"""
5217e36356bb store: add logic to group revlog file together
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50478
diff changeset
700 if filename.endswith(REVLOG_FILES_LONG_EXT):
5217e36356bb store: add logic to group revlog file together
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50478
diff changeset
701 char = b'-'
5217e36356bb store: add logic to group revlog file together
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50478
diff changeset
702 else:
5217e36356bb store: add logic to group revlog file together
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50478
diff changeset
703 char = b'.'
5217e36356bb store: add logic to group revlog file together
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50478
diff changeset
704 idx = filename.rfind(char)
5217e36356bb store: add logic to group revlog file together
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50478
diff changeset
705 return filename[:idx], filename[idx:]
5217e36356bb store: add logic to group revlog file together
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50478
diff changeset
706
5217e36356bb store: add logic to group revlog file together
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50478
diff changeset
707
50481
b08243dbc2e3 store: also gather files per revlog in `topfiles`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50480
diff changeset
708 def _ext_key(ext):
b08243dbc2e3 store: also gather files per revlog in `topfiles`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50480
diff changeset
709 """a key to order revlog suffix
b08243dbc2e3 store: also gather files per revlog in `topfiles`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50480
diff changeset
710
b08243dbc2e3 store: also gather files per revlog in `topfiles`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50480
diff changeset
711 important to issue .i after other entry."""
b08243dbc2e3 store: also gather files per revlog in `topfiles`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50480
diff changeset
712 # the only important part of this order is to keep the `.i` last.
b08243dbc2e3 store: also gather files per revlog in `topfiles`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50480
diff changeset
713 if ext.endswith(b'.n'):
b08243dbc2e3 store: also gather files per revlog in `topfiles`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50480
diff changeset
714 return (0, ext)
b08243dbc2e3 store: also gather files per revlog in `topfiles`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50480
diff changeset
715 elif ext.endswith(b'.nd'):
b08243dbc2e3 store: also gather files per revlog in `topfiles`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50480
diff changeset
716 return (10, ext)
b08243dbc2e3 store: also gather files per revlog in `topfiles`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50480
diff changeset
717 elif ext.endswith(b'.d'):
b08243dbc2e3 store: also gather files per revlog in `topfiles`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50480
diff changeset
718 return (20, ext)
b08243dbc2e3 store: also gather files per revlog in `topfiles`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50480
diff changeset
719 elif ext.endswith(b'.i'):
b08243dbc2e3 store: also gather files per revlog in `topfiles`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50480
diff changeset
720 return (50, ext)
b08243dbc2e3 store: also gather files per revlog in `topfiles`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50480
diff changeset
721 else:
b08243dbc2e3 store: also gather files per revlog in `topfiles`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50480
diff changeset
722 return (40, ext)
b08243dbc2e3 store: also gather files per revlog in `topfiles`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50480
diff changeset
723
b08243dbc2e3 store: also gather files per revlog in `topfiles`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50480
diff changeset
724
48946
642e31cb55f0 py3: use class X: instead of class X(object):
Gregory Szorc <gregory.szorc@gmail.com>
parents: 48941
diff changeset
725 class basicstore:
6840
80e51429cb9a introduce store classes
Adrian Buehlmann <adrian@cadifra.com>
parents: 6839
diff changeset
726 '''base class for local repository stores'''
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
727
17651
3b49c28658f6 store: rename "openertype" argument to "vfstype"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17649
diff changeset
728 def __init__(self, path, vfstype):
17724
bf4b72d8dd4d store: initialize vfs field first to use it for initialization of others
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17722
diff changeset
729 vfs = vfstype(path)
bf4b72d8dd4d store: initialize vfs field first to use it for initialization of others
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17722
diff changeset
730 self.path = vfs.base
17726
7cb7e17c23b2 store: invoke "os.stat()" for "createmode" initialization via vfs
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17725
diff changeset
731 self.createmode = _calcmode(vfs)
17652
2c6f7231becc store: rename "op" variables to "vfs"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17651
diff changeset
732 vfs.createmode = self.createmode
17728
004bd533880d store: invoke "os.path.isdir()" via vfs
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17727
diff changeset
733 self.rawvfs = vfs
31234
9b7a2ef4f27c vfs: use 'vfs' module directly in 'mercurial.store'
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31219
diff changeset
734 self.vfs = vfsmod.filtervfs(vfs, encodedir)
17653
dacb50696b75 store: initialize "vfs" fields by "vfs" constructors
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17652
diff changeset
735 self.opener = self.vfs
6840
80e51429cb9a introduce store classes
Adrian Buehlmann <adrian@cadifra.com>
parents: 6839
diff changeset
736
80e51429cb9a introduce store classes
Adrian Buehlmann <adrian@cadifra.com>
parents: 6839
diff changeset
737 def join(self, f):
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
738 return self.path + b'/' + encodedir(f)
6840
80e51429cb9a introduce store classes
Adrian Buehlmann <adrian@cadifra.com>
parents: 6839
diff changeset
739
50493
816e8bc6e066 store: split the wrapping of encodedstore between _wrap and datafiles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50492
diff changeset
740 def _walk(self, relpath, recurse, undecodable=None):
47877
2174f54aab18 store: return just one filename in walk functions
Valentin Gatien-Baron <valentin.gatienbaron@gmail.com>
parents: 47785
diff changeset
741 '''yields (revlog_type, unencoded, size)'''
13426
643b8212813e store: remove pointless pathjoiner parameter
Adrian Buehlmann <adrian@cadifra.com>
parents: 13391
diff changeset
742 path = self.path
643b8212813e store: remove pointless pathjoiner parameter
Adrian Buehlmann <adrian@cadifra.com>
parents: 13391
diff changeset
743 if relpath:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
744 path += b'/' + relpath
13426
643b8212813e store: remove pointless pathjoiner parameter
Adrian Buehlmann <adrian@cadifra.com>
parents: 13391
diff changeset
745 striplen = len(self.path) + 1
6899
56a7a54e074f store: simplify walking
Matt Mackall <mpm@selenic.com>
parents: 6898
diff changeset
746 l = []
17728
004bd533880d store: invoke "os.path.isdir()" via vfs
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17727
diff changeset
747 if self.rawvfs.isdir(path):
6899
56a7a54e074f store: simplify walking
Matt Mackall <mpm@selenic.com>
parents: 6898
diff changeset
748 visit = [path]
17747
aad3bce98f76 store: invoke "osutil.listdir()" via vfs
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17745
diff changeset
749 readdir = self.rawvfs.readdir
6899
56a7a54e074f store: simplify walking
Matt Mackall <mpm@selenic.com>
parents: 6898
diff changeset
750 while visit:
56a7a54e074f store: simplify walking
Matt Mackall <mpm@selenic.com>
parents: 6898
diff changeset
751 p = visit.pop()
17747
aad3bce98f76 store: invoke "osutil.listdir()" via vfs
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17745
diff changeset
752 for f, kind, st in readdir(p, stat=True):
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
753 fp = p + b'/' + f
46895
6085b7f1536d store: also return some information about the type of file `walk` found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46854
diff changeset
754 rl_type = is_revlog(f, kind, st)
6085b7f1536d store: also return some information about the type of file `walk` found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46854
diff changeset
755 if rl_type is not None:
6900
def492d1b592 store: change handling of decoding errors
Matt Mackall <mpm@selenic.com>
parents: 6899
diff changeset
756 n = util.pconvert(fp[striplen:])
50478
1c0244a8cdaf store: change `_walk` return to `(filename, (type, size))`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50477
diff changeset
757 l.append((decodedir(n), (rl_type, st.st_size)))
6899
56a7a54e074f store: simplify walking
Matt Mackall <mpm@selenic.com>
parents: 6898
diff changeset
758 elif kind == stat.S_IFDIR and recurse:
56a7a54e074f store: simplify walking
Matt Mackall <mpm@selenic.com>
parents: 6898
diff changeset
759 visit.append(fp)
50478
1c0244a8cdaf store: change `_walk` return to `(filename, (type, size))`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50477
diff changeset
760
17054
125ff5654b72 store: sort filenames in place
Bryan O'Sullivan <bryano@fb.com>
parents: 16404
diff changeset
761 l.sort()
125ff5654b72 store: sort filenames in place
Bryan O'Sullivan <bryano@fb.com>
parents: 16404
diff changeset
762 return l
6840
80e51429cb9a introduce store classes
Adrian Buehlmann <adrian@cadifra.com>
parents: 6839
diff changeset
763
46607
e9901d01d135 revlog: add a mechanism to verify expected file position before appending
Kyle Lippincott <spectral@google.com>
parents: 46383
diff changeset
764 def changelog(self, trypending, concurrencychecker=None):
e9901d01d135 revlog: add a mechanism to verify expected file position before appending
Kyle Lippincott <spectral@google.com>
parents: 46383
diff changeset
765 return changelog.changelog(
e9901d01d135 revlog: add a mechanism to verify expected file position before appending
Kyle Lippincott <spectral@google.com>
parents: 46383
diff changeset
766 self.vfs,
e9901d01d135 revlog: add a mechanism to verify expected file position before appending
Kyle Lippincott <spectral@google.com>
parents: 46383
diff changeset
767 trypending=trypending,
e9901d01d135 revlog: add a mechanism to verify expected file position before appending
Kyle Lippincott <spectral@google.com>
parents: 46383
diff changeset
768 concurrencychecker=concurrencychecker,
e9901d01d135 revlog: add a mechanism to verify expected file position before appending
Kyle Lippincott <spectral@google.com>
parents: 46383
diff changeset
769 )
42905
3df3b139a43d localrepo: push manifestlog and changelog construction code into store
Augie Fackler <augie@google.com>
parents: 42715
diff changeset
770
3df3b139a43d localrepo: push manifestlog and changelog construction code into store
Augie Fackler <augie@google.com>
parents: 42715
diff changeset
771 def manifestlog(self, repo, storenarrowmatch):
46780
6266d19556ad node: introduce nodeconstants class
Joerg Sonnenberger <joerg@bec.de>
parents: 46607
diff changeset
772 rootstore = manifest.manifestrevlog(repo.nodeconstants, self.vfs)
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
773 return manifest.manifestlog(self.vfs, repo, rootstore, storenarrowmatch)
42905
3df3b139a43d localrepo: push manifestlog and changelog construction code into store
Augie Fackler <augie@google.com>
parents: 42715
diff changeset
774
50504
862e3a13da44 store: rename `datafiles` to `data_entries`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50498
diff changeset
775 def data_entries(
50471
521fec115dad store: use a StoreEntry object instead of tuple for store files
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50469
diff changeset
776 self, matcher=None, undecodable=None
50473
5a2fb64d38b2 store: use specialized class for store entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50472
diff changeset
777 ) -> Generator[BaseStoreEntry, None, None]:
47877
2174f54aab18 store: return just one filename in walk functions
Valentin Gatien-Baron <valentin.gatienbaron@gmail.com>
parents: 47785
diff changeset
778 """Like walk, but excluding the changelog and root manifest.
2174f54aab18 store: return just one filename in walk functions
Valentin Gatien-Baron <valentin.gatienbaron@gmail.com>
parents: 47785
diff changeset
779
2174f54aab18 store: return just one filename in walk functions
Valentin Gatien-Baron <valentin.gatienbaron@gmail.com>
parents: 47785
diff changeset
780 When [undecodable] is None, revlogs names that can't be
2174f54aab18 store: return just one filename in walk functions
Valentin Gatien-Baron <valentin.gatienbaron@gmail.com>
parents: 47785
diff changeset
781 decoded cause an exception. When it is provided, it should
2174f54aab18 store: return just one filename in walk functions
Valentin Gatien-Baron <valentin.gatienbaron@gmail.com>
parents: 47785
diff changeset
782 be a list and the filenames that can't be decoded are added
2174f54aab18 store: return just one filename in walk functions
Valentin Gatien-Baron <valentin.gatienbaron@gmail.com>
parents: 47785
diff changeset
783 to it instead. This is very rarely needed."""
50482
1fc25227b068 store: actually tag tree manifest revlogs as manifest revlogs
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50481
diff changeset
784 dirs = [
50629
309cbd8400ae store: properly compute the targer_id of manifestlog in no-fncache walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50628
diff changeset
785 (b'data', FILEFLAGS_FILELOG, False),
309cbd8400ae store: properly compute the targer_id of manifestlog in no-fncache walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50628
diff changeset
786 (b'meta', FILEFLAGS_MANIFESTLOG, True),
50482
1fc25227b068 store: actually tag tree manifest revlogs as manifest revlogs
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50481
diff changeset
787 ]
50629
309cbd8400ae store: properly compute the targer_id of manifestlog in no-fncache walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50628
diff changeset
788 for base_dir, rl_type, strip_filename in dirs:
50493
816e8bc6e066 store: split the wrapping of encodedstore between _wrap and datafiles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50492
diff changeset
789 files = self._walk(base_dir, True, undecodable=undecodable)
50482
1fc25227b068 store: actually tag tree manifest revlogs as manifest revlogs
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50481
diff changeset
790 files = (f for f in files if f[1][0] is not None)
1fc25227b068 store: actually tag tree manifest revlogs as manifest revlogs
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50481
diff changeset
791 for revlog, details in _gather_revlog(files):
50496
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50495
diff changeset
792 file_details = {}
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50495
diff changeset
793 revlog_target_id = revlog.split(b'/', 1)[1]
50629
309cbd8400ae store: properly compute the targer_id of manifestlog in no-fncache walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50628
diff changeset
794 if strip_filename and b'/' in revlog:
309cbd8400ae store: properly compute the targer_id of manifestlog in no-fncache walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50628
diff changeset
795 revlog_target_id = revlog_target_id.rsplit(b'/', 1)[0]
309cbd8400ae store: properly compute the targer_id of manifestlog in no-fncache walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50628
diff changeset
796 revlog_target_id += b'/'
50482
1fc25227b068 store: actually tag tree manifest revlogs as manifest revlogs
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50481
diff changeset
797 for ext, (t, s) in sorted(details.items()):
50496
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50495
diff changeset
798 file_details[ext] = {
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50495
diff changeset
799 'is_volatile': bool(t & FILEFLAGS_VOLATILE),
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50495
diff changeset
800 'file_size': s,
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50495
diff changeset
801 }
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50495
diff changeset
802 yield RevlogStoreEntry(
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50495
diff changeset
803 path_prefix=revlog,
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50495
diff changeset
804 revlog_type=rl_type,
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50495
diff changeset
805 target_id=revlog_target_id,
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50495
diff changeset
806 details=file_details,
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50495
diff changeset
807 )
6840
80e51429cb9a introduce store classes
Adrian Buehlmann <adrian@cadifra.com>
parents: 6839
diff changeset
808
50514
0925eaf09c8b store: make `walk` return an entry for obsolescence if requested so
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50513
diff changeset
809 def top_entries(
0925eaf09c8b store: make `walk` return an entry for obsolescence if requested so
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50513
diff changeset
810 self, phase=False, obsolescence=False
0925eaf09c8b store: make `walk` return an entry for obsolescence if requested so
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50513
diff changeset
811 ) -> Generator[BaseStoreEntry, None, None]:
50513
5a62d56e3955 store: yield phases before changelog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50512
diff changeset
812 if phase and self.vfs.exists(b'phaseroots'):
5a62d56e3955 store: yield phases before changelog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50512
diff changeset
813 yield SimpleStoreEntry(
5a62d56e3955 store: yield phases before changelog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50512
diff changeset
814 entry_path=b'phaseroots',
5a62d56e3955 store: yield phases before changelog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50512
diff changeset
815 is_volatile=True,
5a62d56e3955 store: yield phases before changelog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50512
diff changeset
816 )
5a62d56e3955 store: yield phases before changelog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50512
diff changeset
817
50514
0925eaf09c8b store: make `walk` return an entry for obsolescence if requested so
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50513
diff changeset
818 if obsolescence and self.vfs.exists(b'obsstore'):
0925eaf09c8b store: make `walk` return an entry for obsolescence if requested so
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50513
diff changeset
819 # XXX if we had the file size it could be non-volatile
0925eaf09c8b store: make `walk` return an entry for obsolescence if requested so
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50513
diff changeset
820 yield SimpleStoreEntry(
0925eaf09c8b store: make `walk` return an entry for obsolescence if requested so
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50513
diff changeset
821 entry_path=b'obsstore',
0925eaf09c8b store: make `walk` return an entry for obsolescence if requested so
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50513
diff changeset
822 is_volatile=True,
0925eaf09c8b store: make `walk` return an entry for obsolescence if requested so
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50513
diff changeset
823 )
0925eaf09c8b store: make `walk` return an entry for obsolescence if requested so
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50513
diff changeset
824
50481
b08243dbc2e3 store: also gather files per revlog in `topfiles`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50480
diff changeset
825 files = reversed(self._walk(b'', False))
b08243dbc2e3 store: also gather files per revlog in `topfiles`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50480
diff changeset
826
b08243dbc2e3 store: also gather files per revlog in `topfiles`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50480
diff changeset
827 changelogs = collections.defaultdict(dict)
b08243dbc2e3 store: also gather files per revlog in `topfiles`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50480
diff changeset
828 manifestlogs = collections.defaultdict(dict)
b08243dbc2e3 store: also gather files per revlog in `topfiles`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50480
diff changeset
829
50478
1c0244a8cdaf store: change `_walk` return to `(filename, (type, size))`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50477
diff changeset
830 for u, (t, s) in files:
46895
6085b7f1536d store: also return some information about the type of file `walk` found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46854
diff changeset
831 if u.startswith(b'00changelog'):
50481
b08243dbc2e3 store: also gather files per revlog in `topfiles`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50480
diff changeset
832 name, ext = _split_revlog_ext(u)
b08243dbc2e3 store: also gather files per revlog in `topfiles`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50480
diff changeset
833 changelogs[name][ext] = (t, s)
46895
6085b7f1536d store: also return some information about the type of file `walk` found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46854
diff changeset
834 elif u.startswith(b'00manifest'):
50481
b08243dbc2e3 store: also gather files per revlog in `topfiles`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50480
diff changeset
835 name, ext = _split_revlog_ext(u)
b08243dbc2e3 store: also gather files per revlog in `topfiles`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50480
diff changeset
836 manifestlogs[name][ext] = (t, s)
46895
6085b7f1536d store: also return some information about the type of file `walk` found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46854
diff changeset
837 else:
50473
5a2fb64d38b2 store: use specialized class for store entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50472
diff changeset
838 yield SimpleStoreEntry(
50495
ed8cda1c18e1 store: rename `unencoded_path` to `entry_path` for StoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50494
diff changeset
839 entry_path=u,
50473
5a2fb64d38b2 store: use specialized class for store entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50472
diff changeset
840 is_volatile=bool(t & FILEFLAGS_VOLATILE),
5a2fb64d38b2 store: use specialized class for store entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50472
diff changeset
841 file_size=s,
5a2fb64d38b2 store: use specialized class for store entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50472
diff changeset
842 )
50481
b08243dbc2e3 store: also gather files per revlog in `topfiles`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50480
diff changeset
843 # yield manifest before changelog
b08243dbc2e3 store: also gather files per revlog in `topfiles`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50480
diff changeset
844 top_rl = [
b08243dbc2e3 store: also gather files per revlog in `topfiles`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50480
diff changeset
845 (manifestlogs, FILEFLAGS_MANIFESTLOG),
b08243dbc2e3 store: also gather files per revlog in `topfiles`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50480
diff changeset
846 (changelogs, FILEFLAGS_CHANGELOG),
b08243dbc2e3 store: also gather files per revlog in `topfiles`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50480
diff changeset
847 ]
b08243dbc2e3 store: also gather files per revlog in `topfiles`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50480
diff changeset
848 assert len(manifestlogs) <= 1
b08243dbc2e3 store: also gather files per revlog in `topfiles`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50480
diff changeset
849 assert len(changelogs) <= 1
b08243dbc2e3 store: also gather files per revlog in `topfiles`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50480
diff changeset
850 for data, revlog_type in top_rl:
b08243dbc2e3 store: also gather files per revlog in `topfiles`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50480
diff changeset
851 for revlog, details in sorted(data.items()):
50496
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50495
diff changeset
852 file_details = {}
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50495
diff changeset
853 for ext, (t, s) in details.items():
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50495
diff changeset
854 file_details[ext] = {
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50495
diff changeset
855 'is_volatile': bool(t & FILEFLAGS_VOLATILE),
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50495
diff changeset
856 'file_size': s,
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50495
diff changeset
857 }
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50495
diff changeset
858 yield RevlogStoreEntry(
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50495
diff changeset
859 path_prefix=revlog,
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50495
diff changeset
860 revlog_type=revlog_type,
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50495
diff changeset
861 target_id=b'',
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50495
diff changeset
862 details=file_details,
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50495
diff changeset
863 )
19177
1e104aaa4c44 store: move top file walk to a separate function
Durham Goode <durham@fb.com>
parents: 18435
diff changeset
864
50512
a32d739b0ffb store: make `walk` return an entry for phase if requested so
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50505
diff changeset
865 def walk(
50514
0925eaf09c8b store: make `walk` return an entry for obsolescence if requested so
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50513
diff changeset
866 self, matcher=None, phase=False, obsolescence=False
50512
a32d739b0ffb store: make `walk` return an entry for phase if requested so
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50505
diff changeset
867 ) -> Generator[BaseStoreEntry, None, None]:
50451
584ff1f97201 comments: fix spelling
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50319
diff changeset
868 """return files related to data storage (ie: revlogs)
46853
eed3e2b79b48 store: document the `walk` method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46819
diff changeset
869
50512
a32d739b0ffb store: make `walk` return an entry for phase if requested so
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50505
diff changeset
870 yields instance from BaseStoreEntry subclasses
40340
2d45b549392f store: pass matcher to store.datafiles()
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 39698
diff changeset
871
2d45b549392f store: pass matcher to store.datafiles()
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 39698
diff changeset
872 if a matcher is passed, storage files of only those tracked paths
2d45b549392f store: pass matcher to store.datafiles()
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 39698
diff changeset
873 are passed with matches the matcher
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45483
diff changeset
874 """
6840
80e51429cb9a introduce store classes
Adrian Buehlmann <adrian@cadifra.com>
parents: 6839
diff changeset
875 # yield data files first
50504
862e3a13da44 store: rename `datafiles` to `data_entries`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50498
diff changeset
876 for x in self.data_entries(matcher):
6840
80e51429cb9a introduce store classes
Adrian Buehlmann <adrian@cadifra.com>
parents: 6839
diff changeset
877 yield x
50514
0925eaf09c8b store: make `walk` return an entry for obsolescence if requested so
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50513
diff changeset
878 for x in self.top_entries(phase=phase, obsolescence=obsolescence):
6840
80e51429cb9a introduce store classes
Adrian Buehlmann <adrian@cadifra.com>
parents: 6839
diff changeset
879 yield x
80e51429cb9a introduce store classes
Adrian Buehlmann <adrian@cadifra.com>
parents: 6839
diff changeset
880
6903
0642d9d7ec80 clone: get a list of files to clone from store
Matt Mackall <mpm@selenic.com>
parents: 6902
diff changeset
881 def copylist(self):
45483
d252f51ab032 share: introduce config option to store requires in .hg/store
Pulkit Goyal <7895pulkit@gmail.com>
parents: 45351
diff changeset
882 return _data
6903
0642d9d7ec80 clone: get a list of files to clone from store
Matt Mackall <mpm@selenic.com>
parents: 6902
diff changeset
883
20883
cd443c7589cc fncache: move fncache writing to be in a transaction
Durham Goode <durham@fb.com>
parents: 20879
diff changeset
884 def write(self, tr):
13391
d00bbff8600e fncachestore: defer updating the fncache file to a single file open
Adrian Buehlmann <adrian@cadifra.com>
parents: 13169
diff changeset
885 pass
d00bbff8600e fncachestore: defer updating the fncache file to a single file open
Adrian Buehlmann <adrian@cadifra.com>
parents: 13169
diff changeset
886
20884
2efdd186925d caches: invalidate store caches when lock is taken
Durham Goode <durham@fb.com>
parents: 20883
diff changeset
887 def invalidatecaches(self):
2efdd186925d caches: invalidate store caches when lock is taken
Durham Goode <durham@fb.com>
parents: 20883
diff changeset
888 pass
2efdd186925d caches: invalidate store caches when lock is taken
Durham Goode <durham@fb.com>
parents: 20883
diff changeset
889
20885
f49d60fa40a5 fncache: clean up fncache during strips
Durham Goode <durham@fb.com>
parents: 20884
diff changeset
890 def markremoved(self, fn):
f49d60fa40a5 fncache: clean up fncache during strips
Durham Goode <durham@fb.com>
parents: 20884
diff changeset
891 pass
f49d60fa40a5 fncache: clean up fncache during strips
Durham Goode <durham@fb.com>
parents: 20884
diff changeset
892
17744
09d5b2055295 store: add a contains method to basicstore
smuralid
parents: 17738
diff changeset
893 def __contains__(self, path):
09d5b2055295 store: add a contains method to basicstore
smuralid
parents: 17738
diff changeset
894 '''Checks if the store contains path'''
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
895 path = b"/".join((b"data", path))
17744
09d5b2055295 store: add a contains method to basicstore
smuralid
parents: 17738
diff changeset
896 # file?
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
897 if self.vfs.exists(path + b".i"):
17744
09d5b2055295 store: add a contains method to basicstore
smuralid
parents: 17738
diff changeset
898 return True
09d5b2055295 store: add a contains method to basicstore
smuralid
parents: 17738
diff changeset
899 # dir?
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
900 if not path.endswith(b"/"):
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
901 path = path + b"/"
19903
ca875b271ac3 store: use "vfs.exists()" instead of "os.path.exists()"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 19177
diff changeset
902 return self.vfs.exists(path)
17744
09d5b2055295 store: add a contains method to basicstore
smuralid
parents: 17738
diff changeset
903
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
904
6898
69aeaaaf6e07 store: simplify class hierarchy
Matt Mackall <mpm@selenic.com>
parents: 6897
diff changeset
905 class encodedstore(basicstore):
17651
3b49c28658f6 store: rename "openertype" argument to "vfstype"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17649
diff changeset
906 def __init__(self, path, vfstype):
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
907 vfs = vfstype(path + b'/store')
17724
bf4b72d8dd4d store: initialize vfs field first to use it for initialization of others
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17722
diff changeset
908 self.path = vfs.base
17726
7cb7e17c23b2 store: invoke "os.stat()" for "createmode" initialization via vfs
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17725
diff changeset
909 self.createmode = _calcmode(vfs)
17652
2c6f7231becc store: rename "op" variables to "vfs"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17651
diff changeset
910 vfs.createmode = self.createmode
17728
004bd533880d store: invoke "os.path.isdir()" via vfs
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17727
diff changeset
911 self.rawvfs = vfs
31234
9b7a2ef4f27c vfs: use 'vfs' module directly in 'mercurial.store'
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31219
diff changeset
912 self.vfs = vfsmod.filtervfs(vfs, encodefilename)
17653
dacb50696b75 store: initialize "vfs" fields by "vfs" constructors
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17652
diff changeset
913 self.opener = self.vfs
6840
80e51429cb9a introduce store classes
Adrian Buehlmann <adrian@cadifra.com>
parents: 6839
diff changeset
914
50493
816e8bc6e066 store: split the wrapping of encodedstore between _wrap and datafiles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50492
diff changeset
915 def _walk(self, relpath, recurse, undecodable=None):
816e8bc6e066 store: split the wrapping of encodedstore between _wrap and datafiles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50492
diff changeset
916 old = super()._walk(relpath, recurse)
816e8bc6e066 store: split the wrapping of encodedstore between _wrap and datafiles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50492
diff changeset
917 new = []
816e8bc6e066 store: split the wrapping of encodedstore between _wrap and datafiles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50492
diff changeset
918 for f1, value in old:
6892
dab95717058d verify: check repo.store
Adrian Buehlmann <adrian@cadifra.com>
parents: 6890
diff changeset
919 try:
47877
2174f54aab18 store: return just one filename in walk functions
Valentin Gatien-Baron <valentin.gatienbaron@gmail.com>
parents: 47785
diff changeset
920 f2 = decodefilename(f1)
6892
dab95717058d verify: check repo.store
Adrian Buehlmann <adrian@cadifra.com>
parents: 6890
diff changeset
921 except KeyError:
47877
2174f54aab18 store: return just one filename in walk functions
Valentin Gatien-Baron <valentin.gatienbaron@gmail.com>
parents: 47785
diff changeset
922 if undecodable is None:
2174f54aab18 store: return just one filename in walk functions
Valentin Gatien-Baron <valentin.gatienbaron@gmail.com>
parents: 47785
diff changeset
923 msg = _(b'undecodable revlog name %s') % f1
2174f54aab18 store: return just one filename in walk functions
Valentin Gatien-Baron <valentin.gatienbaron@gmail.com>
parents: 47785
diff changeset
924 raise error.StorageError(msg)
2174f54aab18 store: return just one filename in walk functions
Valentin Gatien-Baron <valentin.gatienbaron@gmail.com>
parents: 47785
diff changeset
925 else:
2174f54aab18 store: return just one filename in walk functions
Valentin Gatien-Baron <valentin.gatienbaron@gmail.com>
parents: 47785
diff changeset
926 undecodable.append(f1)
2174f54aab18 store: return just one filename in walk functions
Valentin Gatien-Baron <valentin.gatienbaron@gmail.com>
parents: 47785
diff changeset
927 continue
50493
816e8bc6e066 store: split the wrapping of encodedstore between _wrap and datafiles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50492
diff changeset
928 new.append((f2, value))
816e8bc6e066 store: split the wrapping of encodedstore between _wrap and datafiles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50492
diff changeset
929 return new
816e8bc6e066 store: split the wrapping of encodedstore between _wrap and datafiles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50492
diff changeset
930
50504
862e3a13da44 store: rename `datafiles` to `data_entries`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50498
diff changeset
931 def data_entries(
50493
816e8bc6e066 store: split the wrapping of encodedstore between _wrap and datafiles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50492
diff changeset
932 self, matcher=None, undecodable=None
816e8bc6e066 store: split the wrapping of encodedstore between _wrap and datafiles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50492
diff changeset
933 ) -> Generator[BaseStoreEntry, None, None]:
50504
862e3a13da44 store: rename `datafiles` to `data_entries`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50498
diff changeset
934 entries = super(encodedstore, self).data_entries(
862e3a13da44 store: rename `datafiles` to `data_entries`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50498
diff changeset
935 undecodable=undecodable
862e3a13da44 store: rename `datafiles` to `data_entries`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50498
diff changeset
936 )
50493
816e8bc6e066 store: split the wrapping of encodedstore between _wrap and datafiles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50492
diff changeset
937 for entry in entries:
50494
b4953fad744e store: do the revlog matching on entry directly
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50493
diff changeset
938 if _match_tracked_entry(entry, matcher):
50493
816e8bc6e066 store: split the wrapping of encodedstore between _wrap and datafiles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50492
diff changeset
939 yield entry
6840
80e51429cb9a introduce store classes
Adrian Buehlmann <adrian@cadifra.com>
parents: 6839
diff changeset
940
80e51429cb9a introduce store classes
Adrian Buehlmann <adrian@cadifra.com>
parents: 6839
diff changeset
941 def join(self, f):
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
942 return self.path + b'/' + encodefilename(f)
6840
80e51429cb9a introduce store classes
Adrian Buehlmann <adrian@cadifra.com>
parents: 6839
diff changeset
943
6903
0642d9d7ec80 clone: get a list of files to clone from store
Matt Mackall <mpm@selenic.com>
parents: 6902
diff changeset
944 def copylist(self):
45351
909dafff6a78 store: refactor space delimited list to proper data structure
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
945 return [b'requires', b'00changelog.i'] + [b'store/' + f for f in _data]
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
946
6903
0642d9d7ec80 clone: get a list of files to clone from store
Matt Mackall <mpm@selenic.com>
parents: 6902
diff changeset
947
48946
642e31cb55f0 py3: use class X: instead of class X(object):
Gregory Szorc <gregory.szorc@gmail.com>
parents: 48941
diff changeset
948 class fncache:
8531
810387f59696 filelog encoding: move the encoding/decoding into store
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 8530
diff changeset
949 # the filename used to be partially encoded
810387f59696 filelog encoding: move the encoding/decoding into store
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 8530
diff changeset
950 # hence the encodedir/decodedir dance
17722
3b976051034d store: rename field name from "opener" to "vfs" in internal classes for fncache
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17721
diff changeset
951 def __init__(self, vfs):
3b976051034d store: rename field name from "opener" to "vfs" in internal classes for fncache
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17721
diff changeset
952 self.vfs = vfs
50315
cf6e1d535602 fncache: make it possible to ignore some file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 49306
diff changeset
953 self._ignores = set()
7229
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
954 self.entries = None
13391
d00bbff8600e fncachestore: defer updating the fncache file to a single file open
Adrian Buehlmann <adrian@cadifra.com>
parents: 13169
diff changeset
955 self._dirty = False
40736
0728d87a8631 store: append to fncache if there are only new files to write
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 40624
diff changeset
956 # set of new additions to fncache
0728d87a8631 store: append to fncache if there are only new files to write
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 40624
diff changeset
957 self.addls = set()
7229
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
958
42715
f59f8a5e9096 fncache: make debugrebuildfncache not fail on broken fncache
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents: 42341
diff changeset
959 def ensureloaded(self, warn=None):
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45483
diff changeset
960 """read the fncache file if not already read.
42715
f59f8a5e9096 fncache: make debugrebuildfncache not fail on broken fncache
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents: 42341
diff changeset
961
f59f8a5e9096 fncache: make debugrebuildfncache not fail on broken fncache
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents: 42341
diff changeset
962 If the file on disk is corrupted, raise. If warn is provided,
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45483
diff changeset
963 warn and keep going instead."""
42715
f59f8a5e9096 fncache: make debugrebuildfncache not fail on broken fncache
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents: 42341
diff changeset
964 if self.entries is None:
f59f8a5e9096 fncache: make debugrebuildfncache not fail on broken fncache
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents: 42341
diff changeset
965 self._load(warn)
f59f8a5e9096 fncache: make debugrebuildfncache not fail on broken fncache
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents: 42341
diff changeset
966
f59f8a5e9096 fncache: make debugrebuildfncache not fail on broken fncache
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents: 42341
diff changeset
967 def _load(self, warn=None):
8530
03196ac9a8b9 store: refactor the fncache handling
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 8480
diff changeset
968 '''fill the entries from the fncache file'''
13391
d00bbff8600e fncachestore: defer updating the fncache file to a single file open
Adrian Buehlmann <adrian@cadifra.com>
parents: 13169
diff changeset
969 self._dirty = False
8530
03196ac9a8b9 store: refactor the fncache handling
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 8480
diff changeset
970 try:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
971 fp = self.vfs(b'fncache', mode=b'rb')
8530
03196ac9a8b9 store: refactor the fncache handling
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 8480
diff changeset
972 except IOError:
03196ac9a8b9 store: refactor the fncache handling
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 8480
diff changeset
973 # skip nonexistent file
16404
9fca5b056c0a store: speed up read and write of large fncache files
Bryan O'Sullivan <bryano@fb.com>
parents: 15742
diff changeset
974 self.entries = set()
8530
03196ac9a8b9 store: refactor the fncache handling
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 8480
diff changeset
975 return
41978
a56487081109 store: don't read the whole fncache in memory
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 41973
diff changeset
976
a56487081109 store: don't read the whole fncache in memory
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 41973
diff changeset
977 self.entries = set()
a56487081109 store: don't read the whole fncache in memory
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 41973
diff changeset
978 chunk = b''
a56487081109 store: don't read the whole fncache in memory
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 41973
diff changeset
979 for c in iter(functools.partial(fp.read, fncache_chunksize), b''):
a56487081109 store: don't read the whole fncache in memory
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 41973
diff changeset
980 chunk += c
a56487081109 store: don't read the whole fncache in memory
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 41973
diff changeset
981 try:
a56487081109 store: don't read the whole fncache in memory
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 41973
diff changeset
982 p = chunk.rindex(b'\n')
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
983 self.entries.update(decodedir(chunk[: p + 1]).splitlines())
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
984 chunk = chunk[p + 1 :]
41978
a56487081109 store: don't read the whole fncache in memory
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 41973
diff changeset
985 except ValueError:
a56487081109 store: don't read the whole fncache in memory
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 41973
diff changeset
986 # substring '\n' not found, maybe the entry is bigger than the
a56487081109 store: don't read the whole fncache in memory
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 41973
diff changeset
987 # chunksize, so let's keep iterating
a56487081109 store: don't read the whole fncache in memory
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 41973
diff changeset
988 pass
a56487081109 store: don't read the whole fncache in memory
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 41973
diff changeset
989
41981
a920a9e1795a store: error out if fncache does not ends with a newline
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 41978
diff changeset
990 if chunk:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
991 msg = _(b"fncache does not ends with a newline")
42715
f59f8a5e9096 fncache: make debugrebuildfncache not fail on broken fncache
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents: 42341
diff changeset
992 if warn:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
993 warn(msg + b'\n')
42715
f59f8a5e9096 fncache: make debugrebuildfncache not fail on broken fncache
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents: 42341
diff changeset
994 else:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
995 raise error.Abort(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
996 msg,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
997 hint=_(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
998 b"use 'hg debugrebuildfncache' to "
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
999 b"rebuild the fncache"
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
1000 ),
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
1001 )
42715
f59f8a5e9096 fncache: make debugrebuildfncache not fail on broken fncache
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents: 42341
diff changeset
1002 self._checkentries(fp, warn)
41973
d7ef84e595f8 store: move logic to check for invalid entry in fncache to own function
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 41093
diff changeset
1003 fp.close()
d7ef84e595f8 store: move logic to check for invalid entry in fncache to own function
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 41093
diff changeset
1004
42715
f59f8a5e9096 fncache: make debugrebuildfncache not fail on broken fncache
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents: 42341
diff changeset
1005 def _checkentries(self, fp, warn):
47062
f38bf44e077f black: make codebase compatible with black v21.4b2 and v20.8b1
Kyle Lippincott <spectral@google.com>
parents: 46990
diff changeset
1006 """make sure there is no empty string in entries"""
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1007 if b'' in self.entries:
16404
9fca5b056c0a store: speed up read and write of large fncache files
Bryan O'Sullivan <bryano@fb.com>
parents: 15742
diff changeset
1008 fp.seek(0)
48941
fd5b8e696b75 py3: stop using util.iterfile()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 48913
diff changeset
1009 for n, line in enumerate(fp):
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1010 if not line.rstrip(b'\n'):
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1011 t = _(b'invalid entry in fncache, line %d') % (n + 1)
42715
f59f8a5e9096 fncache: make debugrebuildfncache not fail on broken fncache
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents: 42341
diff changeset
1012 if warn:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1013 warn(t + b'\n')
42715
f59f8a5e9096 fncache: make debugrebuildfncache not fail on broken fncache
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents: 42341
diff changeset
1014 else:
f59f8a5e9096 fncache: make debugrebuildfncache not fail on broken fncache
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents: 42341
diff changeset
1015 raise error.Abort(t)
7229
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
1016
20883
cd443c7589cc fncache: move fncache writing to be in a transaction
Durham Goode <durham@fb.com>
parents: 20879
diff changeset
1017 def write(self, tr):
16404
9fca5b056c0a store: speed up read and write of large fncache files
Bryan O'Sullivan <bryano@fb.com>
parents: 15742
diff changeset
1018 if self._dirty:
38696
89d93dd1a222 store: assert the fncache have been loaded if dirty
Boris Feld <boris.feld@octobus.net>
parents: 38661
diff changeset
1019 assert self.entries is not None
40748
df8ed31a8ad8 store: write fncache only once if there are both adds and removes
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 40736
diff changeset
1020 self.entries = self.entries | self.addls
df8ed31a8ad8 store: write fncache only once if there are both adds and removes
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 40736
diff changeset
1021 self.addls = set()
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1022 tr.addbackup(b'fncache')
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1023 fp = self.vfs(b'fncache', mode=b'wb', atomictemp=True)
20879
cd03854a2e06 fncache: remove the rewriting logic
Durham Goode <durham@fb.com>
parents: 19903
diff changeset
1024 if self.entries:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1025 fp.write(encodedir(b'\n'.join(self.entries) + b'\n'))
20879
cd03854a2e06 fncache: remove the rewriting logic
Durham Goode <durham@fb.com>
parents: 19903
diff changeset
1026 fp.close()
cd03854a2e06 fncache: remove the rewriting logic
Durham Goode <durham@fb.com>
parents: 19903
diff changeset
1027 self._dirty = False
40736
0728d87a8631 store: append to fncache if there are only new files to write
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 40624
diff changeset
1028 if self.addls:
0728d87a8631 store: append to fncache if there are only new files to write
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 40624
diff changeset
1029 # if we have just new entries, let's append them to the fncache
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1030 tr.addbackup(b'fncache')
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1031 fp = self.vfs(b'fncache', mode=b'ab', atomictemp=True)
40736
0728d87a8631 store: append to fncache if there are only new files to write
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 40624
diff changeset
1032 if self.addls:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1033 fp.write(encodedir(b'\n'.join(self.addls) + b'\n'))
40736
0728d87a8631 store: append to fncache if there are only new files to write
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 40624
diff changeset
1034 fp.close()
0728d87a8631 store: append to fncache if there are only new files to write
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 40624
diff changeset
1035 self.entries = None
0728d87a8631 store: append to fncache if there are only new files to write
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 40624
diff changeset
1036 self.addls = set()
8530
03196ac9a8b9 store: refactor the fncache handling
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 8480
diff changeset
1037
50315
cf6e1d535602 fncache: make it possible to ignore some file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 49306
diff changeset
1038 def addignore(self, fn):
cf6e1d535602 fncache: make it possible to ignore some file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 49306
diff changeset
1039 self._ignores.add(fn)
cf6e1d535602 fncache: make it possible to ignore some file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 49306
diff changeset
1040
8530
03196ac9a8b9 store: refactor the fncache handling
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 8480
diff changeset
1041 def add(self, fn):
50315
cf6e1d535602 fncache: make it possible to ignore some file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 49306
diff changeset
1042 if fn in self._ignores:
cf6e1d535602 fncache: make it possible to ignore some file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 49306
diff changeset
1043 return
8530
03196ac9a8b9 store: refactor the fncache handling
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 8480
diff changeset
1044 if self.entries is None:
03196ac9a8b9 store: refactor the fncache handling
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 8480
diff changeset
1045 self._load()
10577
d5bd1beff794 store: only add new entries to the fncache file
Adrian Buehlmann <adrian@cadifra.com>
parents: 10339
diff changeset
1046 if fn not in self.entries:
40736
0728d87a8631 store: append to fncache if there are only new files to write
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 40624
diff changeset
1047 self.addls.add(fn)
8530
03196ac9a8b9 store: refactor the fncache handling
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 8480
diff changeset
1048
20885
f49d60fa40a5 fncache: clean up fncache during strips
Durham Goode <durham@fb.com>
parents: 20884
diff changeset
1049 def remove(self, fn):
f49d60fa40a5 fncache: clean up fncache during strips
Durham Goode <durham@fb.com>
parents: 20884
diff changeset
1050 if self.entries is None:
f49d60fa40a5 fncache: clean up fncache during strips
Durham Goode <durham@fb.com>
parents: 20884
diff changeset
1051 self._load()
40736
0728d87a8631 store: append to fncache if there are only new files to write
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 40624
diff changeset
1052 if fn in self.addls:
0728d87a8631 store: append to fncache if there are only new files to write
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 40624
diff changeset
1053 self.addls.remove(fn)
0728d87a8631 store: append to fncache if there are only new files to write
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 40624
diff changeset
1054 return
20885
f49d60fa40a5 fncache: clean up fncache during strips
Durham Goode <durham@fb.com>
parents: 20884
diff changeset
1055 try:
f49d60fa40a5 fncache: clean up fncache during strips
Durham Goode <durham@fb.com>
parents: 20884
diff changeset
1056 self.entries.remove(fn)
f49d60fa40a5 fncache: clean up fncache during strips
Durham Goode <durham@fb.com>
parents: 20884
diff changeset
1057 self._dirty = True
f49d60fa40a5 fncache: clean up fncache during strips
Durham Goode <durham@fb.com>
parents: 20884
diff changeset
1058 except KeyError:
f49d60fa40a5 fncache: clean up fncache during strips
Durham Goode <durham@fb.com>
parents: 20884
diff changeset
1059 pass
f49d60fa40a5 fncache: clean up fncache during strips
Durham Goode <durham@fb.com>
parents: 20884
diff changeset
1060
17782
8095306c1fb2 store: move __contains__() implementation from class fncache into fncachestore
Adrian Buehlmann <adrian@cadifra.com>
parents: 17747
diff changeset
1061 def __contains__(self, fn):
40736
0728d87a8631 store: append to fncache if there are only new files to write
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 40624
diff changeset
1062 if fn in self.addls:
0728d87a8631 store: append to fncache if there are only new files to write
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 40624
diff changeset
1063 return True
8530
03196ac9a8b9 store: refactor the fncache handling
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 8480
diff changeset
1064 if self.entries is None:
03196ac9a8b9 store: refactor the fncache handling
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 8480
diff changeset
1065 self._load()
17782
8095306c1fb2 store: move __contains__() implementation from class fncache into fncachestore
Adrian Buehlmann <adrian@cadifra.com>
parents: 17747
diff changeset
1066 return fn in self.entries
8530
03196ac9a8b9 store: refactor the fncache handling
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 8480
diff changeset
1067
03196ac9a8b9 store: refactor the fncache handling
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 8480
diff changeset
1068 def __iter__(self):
03196ac9a8b9 store: refactor the fncache handling
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 8480
diff changeset
1069 if self.entries is None:
03196ac9a8b9 store: refactor the fncache handling
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 8480
diff changeset
1070 self._load()
40736
0728d87a8631 store: append to fncache if there are only new files to write
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 40624
diff changeset
1071 return iter(self.entries | self.addls)
7229
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
1072
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
1073
41093
6498f0e03526 vfs: fix proxyvfs inheritance
Boris Feld <boris.feld@octobus.net>
parents: 40748
diff changeset
1074 class _fncachevfs(vfsmod.proxyvfs):
17721
cf236e3501c3 store: rename argument name from "op"(ener) to "vfs"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17693
diff changeset
1075 def __init__(self, vfs, fnc, encode):
33412
a42369e04aee vfs: rename auditvfs to proxyvfs
Yuya Nishihara <yuya@tcha.org>
parents: 32372
diff changeset
1076 vfsmod.proxyvfs.__init__(self, vfs)
14194
3a90fb3addc1 store: break up reference cycle introduced in 9cbff8a39a2a
Adrian Buehlmann <adrian@cadifra.com>
parents: 14166
diff changeset
1077 self.fncache = fnc
3a90fb3addc1 store: break up reference cycle introduced in 9cbff8a39a2a
Adrian Buehlmann <adrian@cadifra.com>
parents: 14166
diff changeset
1078 self.encode = encode
3a90fb3addc1 store: break up reference cycle introduced in 9cbff8a39a2a
Adrian Buehlmann <adrian@cadifra.com>
parents: 14166
diff changeset
1079
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1080 def __call__(self, path, mode=b'r', *args, **kw):
38661
8ac0c9cd4c48 fncache: avoid loading the filename cache when not actually modifying it
Martijn Pieters <mj@zopatista.com>
parents: 37409
diff changeset
1081 encoded = self.encode(path)
50467
0dbab42adca5 store: do not record file that are not part of a revlog in fncache
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50451
diff changeset
1082 if (
0dbab42adca5 store: do not record file that are not part of a revlog in fncache
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50451
diff changeset
1083 mode not in (b'r', b'rb')
0dbab42adca5 store: do not record file that are not part of a revlog in fncache
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50451
diff changeset
1084 and (path.startswith(b'data/') or path.startswith(b'meta/'))
0dbab42adca5 store: do not record file that are not part of a revlog in fncache
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50451
diff changeset
1085 and revlog_type(path) is not None
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
1086 ):
38661
8ac0c9cd4c48 fncache: avoid loading the filename cache when not actually modifying it
Martijn Pieters <mj@zopatista.com>
parents: 37409
diff changeset
1087 # do not trigger a fncache load when adding a file that already is
8ac0c9cd4c48 fncache: avoid loading the filename cache when not actually modifying it
Martijn Pieters <mj@zopatista.com>
parents: 37409
diff changeset
1088 # known to exist.
8ac0c9cd4c48 fncache: avoid loading the filename cache when not actually modifying it
Martijn Pieters <mj@zopatista.com>
parents: 37409
diff changeset
1089 notload = self.fncache.entries is None and self.vfs.exists(encoded)
47217
8f6165c90163 revlog: open files in 'r+' instead of 'a+'
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47062
diff changeset
1090 if notload and b'r+' in mode and not self.vfs.stat(encoded).st_size:
38661
8ac0c9cd4c48 fncache: avoid loading the filename cache when not actually modifying it
Martijn Pieters <mj@zopatista.com>
parents: 37409
diff changeset
1091 # when appending to an existing file, if the file has size zero,
8ac0c9cd4c48 fncache: avoid loading the filename cache when not actually modifying it
Martijn Pieters <mj@zopatista.com>
parents: 37409
diff changeset
1092 # it should be considered as missing. Such zero-size files are
8ac0c9cd4c48 fncache: avoid loading the filename cache when not actually modifying it
Martijn Pieters <mj@zopatista.com>
parents: 37409
diff changeset
1093 # the result of truncation when a transaction is aborted.
8ac0c9cd4c48 fncache: avoid loading the filename cache when not actually modifying it
Martijn Pieters <mj@zopatista.com>
parents: 37409
diff changeset
1094 notload = False
8ac0c9cd4c48 fncache: avoid loading the filename cache when not actually modifying it
Martijn Pieters <mj@zopatista.com>
parents: 37409
diff changeset
1095 if not notload:
8ac0c9cd4c48 fncache: avoid loading the filename cache when not actually modifying it
Martijn Pieters <mj@zopatista.com>
parents: 37409
diff changeset
1096 self.fncache.add(path)
8ac0c9cd4c48 fncache: avoid loading the filename cache when not actually modifying it
Martijn Pieters <mj@zopatista.com>
parents: 37409
diff changeset
1097 return self.vfs(encoded, mode, *args, **kw)
14194
3a90fb3addc1 store: break up reference cycle introduced in 9cbff8a39a2a
Adrian Buehlmann <adrian@cadifra.com>
parents: 14166
diff changeset
1098
17725
ffd589d4b785 vfs: define "join()" in each classes derived from "abstractvfs"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17724
diff changeset
1099 def join(self, path):
ffd589d4b785 vfs: define "join()" in each classes derived from "abstractvfs"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17724
diff changeset
1100 if path:
ffd589d4b785 vfs: define "join()" in each classes derived from "abstractvfs"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17724
diff changeset
1101 return self.vfs.join(self.encode(path))
ffd589d4b785 vfs: define "join()" in each classes derived from "abstractvfs"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17724
diff changeset
1102 else:
ffd589d4b785 vfs: define "join()" in each classes derived from "abstractvfs"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17724
diff changeset
1103 return self.vfs.join(path)
ffd589d4b785 vfs: define "join()" in each classes derived from "abstractvfs"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17724
diff changeset
1104
47443
9ab54aa56982 vfs: add a `register_file` method on the vfs class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47389
diff changeset
1105 def register_file(self, path):
9ab54aa56982 vfs: add a `register_file` method on the vfs class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47389
diff changeset
1106 """generic hook point to lets fncache steer its stew"""
9ab54aa56982 vfs: add a `register_file` method on the vfs class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47389
diff changeset
1107 if path.startswith(b'data/') or path.startswith(b'meta/'):
9ab54aa56982 vfs: add a `register_file` method on the vfs class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47389
diff changeset
1108 self.fncache.add(path)
9ab54aa56982 vfs: add a `register_file` method on the vfs class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47389
diff changeset
1109
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
1110
7229
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
1111 class fncachestore(basicstore):
17651
3b49c28658f6 store: rename "openertype" argument to "vfstype"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17649
diff changeset
1112 def __init__(self, path, vfstype, dotencode):
17591
9a5c2ecd1158 store: move encode lambda logic into fncachestore
Adrian Buehlmann <adrian@cadifra.com>
parents: 17590
diff changeset
1113 if dotencode:
18435
8c019d2fd7c0 store: switch to C-based hashed path encoding
Bryan O'Sullivan <bryano@fb.com>
parents: 18430
diff changeset
1114 encode = _pathencode
17591
9a5c2ecd1158 store: move encode lambda logic into fncachestore
Adrian Buehlmann <adrian@cadifra.com>
parents: 17590
diff changeset
1115 else:
9a5c2ecd1158 store: move encode lambda logic into fncachestore
Adrian Buehlmann <adrian@cadifra.com>
parents: 17590
diff changeset
1116 encode = _plainhybridencode
12687
34d8247a4595 store: encode first period or space in filenames (issue1713)
Adrian Buehlmann <adrian@cadifra.com>
parents: 12171
diff changeset
1117 self.encode = encode
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1118 vfs = vfstype(path + b'/store')
17724
bf4b72d8dd4d store: initialize vfs field first to use it for initialization of others
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17722
diff changeset
1119 self.path = vfs.base
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1120 self.pathsep = self.path + b'/'
17726
7cb7e17c23b2 store: invoke "os.stat()" for "createmode" initialization via vfs
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17725
diff changeset
1121 self.createmode = _calcmode(vfs)
17652
2c6f7231becc store: rename "op" variables to "vfs"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17651
diff changeset
1122 vfs.createmode = self.createmode
17727
6492b39a44d5 store: replace invocation of "getsize()" by "vfs.stat()"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17726
diff changeset
1123 self.rawvfs = vfs
17652
2c6f7231becc store: rename "op" variables to "vfs"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17651
diff changeset
1124 fnc = fncache(vfs)
9133
996c1cd8f530 store: eliminate reference cycle in fncachestore
Simon Heimberg <simohe@besonet.ch>
parents: 8778
diff changeset
1125 self.fncache = fnc
17653
dacb50696b75 store: initialize "vfs" fields by "vfs" constructors
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17652
diff changeset
1126 self.vfs = _fncachevfs(vfs, fnc, encode)
dacb50696b75 store: initialize "vfs" fields by "vfs" constructors
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17652
diff changeset
1127 self.opener = self.vfs
7229
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
1128
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
1129 def join(self, f):
17562
b42b0729744d store: reduce string concatenation when joining
Bryan O'Sullivan <bryano@fb.com>
parents: 17555
diff changeset
1130 return self.pathsep + self.encode(f)
7229
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
1131
17731
c85dbae29684 store: restore getsize method
Matt Mackall <mpm@selenic.com>
parents: 17728
diff changeset
1132 def getsize(self, path):
c85dbae29684 store: restore getsize method
Matt Mackall <mpm@selenic.com>
parents: 17728
diff changeset
1133 return self.rawvfs.stat(path).st_size
c85dbae29684 store: restore getsize method
Matt Mackall <mpm@selenic.com>
parents: 17728
diff changeset
1134
50504
862e3a13da44 store: rename `datafiles` to `data_entries`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50498
diff changeset
1135 def data_entries(
50471
521fec115dad store: use a StoreEntry object instead of tuple for store files
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50469
diff changeset
1136 self, matcher=None, undecodable=None
50473
5a2fb64d38b2 store: use specialized class for store entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50472
diff changeset
1137 ) -> Generator[BaseStoreEntry, None, None]:
50480
d4f54aded22e store: also group files by revlog in fncache version of datafiles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50479
diff changeset
1138 files = ((f, revlog_type(f)) for f in self.fncache)
d4f54aded22e store: also group files by revlog in fncache version of datafiles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50479
diff changeset
1139 # Note: all files in fncache should be revlog related, However the
d4f54aded22e store: also group files by revlog in fncache version of datafiles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50479
diff changeset
1140 # fncache might contains such file added by previous version of
d4f54aded22e store: also group files by revlog in fncache version of datafiles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50479
diff changeset
1141 # Mercurial.
d4f54aded22e store: also group files by revlog in fncache version of datafiles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50479
diff changeset
1142 files = (f for f in files if f[1] is not None)
d4f54aded22e store: also group files by revlog in fncache version of datafiles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50479
diff changeset
1143 by_revlog = _gather_revlog(files)
d4f54aded22e store: also group files by revlog in fncache version of datafiles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50479
diff changeset
1144 for revlog, details in by_revlog:
50496
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50495
diff changeset
1145 file_details = {}
50482
1fc25227b068 store: actually tag tree manifest revlogs as manifest revlogs
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50481
diff changeset
1146 if revlog.startswith(b'data/'):
1fc25227b068 store: actually tag tree manifest revlogs as manifest revlogs
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50481
diff changeset
1147 rl_type = FILEFLAGS_FILELOG
50483
60e613f6a229 store: add a `target_id` attribute on RevlogStoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50482
diff changeset
1148 revlog_target_id = revlog.split(b'/', 1)[1]
50482
1fc25227b068 store: actually tag tree manifest revlogs as manifest revlogs
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50481
diff changeset
1149 elif revlog.startswith(b'meta/'):
1fc25227b068 store: actually tag tree manifest revlogs as manifest revlogs
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50481
diff changeset
1150 rl_type = FILEFLAGS_MANIFESTLOG
50483
60e613f6a229 store: add a `target_id` attribute on RevlogStoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50482
diff changeset
1151 # drop the initial directory and the `00manifest` file part
60e613f6a229 store: add a `target_id` attribute on RevlogStoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50482
diff changeset
1152 tmp = revlog.split(b'/', 1)[1]
60e613f6a229 store: add a `target_id` attribute on RevlogStoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50482
diff changeset
1153 revlog_target_id = tmp.rsplit(b'/', 1)[0] + b'/'
50482
1fc25227b068 store: actually tag tree manifest revlogs as manifest revlogs
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50481
diff changeset
1154 else:
1fc25227b068 store: actually tag tree manifest revlogs as manifest revlogs
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50481
diff changeset
1155 # unreachable
1fc25227b068 store: actually tag tree manifest revlogs as manifest revlogs
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50481
diff changeset
1156 assert False, revlog
50496
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50495
diff changeset
1157 for ext, t in details.items():
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50495
diff changeset
1158 file_details[ext] = {
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50495
diff changeset
1159 'is_volatile': bool(t & FILEFLAGS_VOLATILE),
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50495
diff changeset
1160 }
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50495
diff changeset
1161 entry = RevlogStoreEntry(
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50495
diff changeset
1162 path_prefix=revlog,
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50495
diff changeset
1163 revlog_type=rl_type,
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50495
diff changeset
1164 target_id=revlog_target_id,
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50495
diff changeset
1165 details=file_details,
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50495
diff changeset
1166 )
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50495
diff changeset
1167 if _match_tracked_entry(entry, matcher):
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50495
diff changeset
1168 yield entry
7229
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
1169
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
1170 def copylist(self):
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
1171 d = (
45351
909dafff6a78 store: refactor space delimited list to proper data structure
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
1172 b'bookmarks',
909dafff6a78 store: refactor space delimited list to proper data structure
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
1173 b'narrowspec',
909dafff6a78 store: refactor space delimited list to proper data structure
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
1174 b'data',
909dafff6a78 store: refactor space delimited list to proper data structure
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
1175 b'meta',
909dafff6a78 store: refactor space delimited list to proper data structure
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
1176 b'dh',
909dafff6a78 store: refactor space delimited list to proper data structure
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
1177 b'fncache',
909dafff6a78 store: refactor space delimited list to proper data structure
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
1178 b'phaseroots',
909dafff6a78 store: refactor space delimited list to proper data structure
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
1179 b'obsstore',
909dafff6a78 store: refactor space delimited list to proper data structure
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
1180 b'00manifest.d',
909dafff6a78 store: refactor space delimited list to proper data structure
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
1181 b'00manifest.i',
909dafff6a78 store: refactor space delimited list to proper data structure
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
1182 b'00changelog.d',
909dafff6a78 store: refactor space delimited list to proper data structure
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
1183 b'00changelog.i',
45483
d252f51ab032 share: introduce config option to store requires in .hg/store
Pulkit Goyal <7895pulkit@gmail.com>
parents: 45351
diff changeset
1184 b'requires',
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
1185 )
45351
909dafff6a78 store: refactor space delimited list to proper data structure
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
1186 return [b'requires', b'00changelog.i'] + [b'store/' + f for f in d]
7229
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
1187
20883
cd443c7589cc fncache: move fncache writing to be in a transaction
Durham Goode <durham@fb.com>
parents: 20879
diff changeset
1188 def write(self, tr):
cd443c7589cc fncache: move fncache writing to be in a transaction
Durham Goode <durham@fb.com>
parents: 20879
diff changeset
1189 self.fncache.write(tr)
13391
d00bbff8600e fncachestore: defer updating the fncache file to a single file open
Adrian Buehlmann <adrian@cadifra.com>
parents: 13169
diff changeset
1190
20884
2efdd186925d caches: invalidate store caches when lock is taken
Durham Goode <durham@fb.com>
parents: 20883
diff changeset
1191 def invalidatecaches(self):
2efdd186925d caches: invalidate store caches when lock is taken
Durham Goode <durham@fb.com>
parents: 20883
diff changeset
1192 self.fncache.entries = None
40736
0728d87a8631 store: append to fncache if there are only new files to write
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 40624
diff changeset
1193 self.fncache.addls = set()
20884
2efdd186925d caches: invalidate store caches when lock is taken
Durham Goode <durham@fb.com>
parents: 20883
diff changeset
1194
20885
f49d60fa40a5 fncache: clean up fncache during strips
Durham Goode <durham@fb.com>
parents: 20884
diff changeset
1195 def markremoved(self, fn):
f49d60fa40a5 fncache: clean up fncache during strips
Durham Goode <durham@fb.com>
parents: 20884
diff changeset
1196 self.fncache.remove(fn)
f49d60fa40a5 fncache: clean up fncache during strips
Durham Goode <durham@fb.com>
parents: 20884
diff changeset
1197
17783
df55ce6854c3 store: add new _exists helper function on fncachestore
Adrian Buehlmann <adrian@cadifra.com>
parents: 17782
diff changeset
1198 def _exists(self, f):
df55ce6854c3 store: add new _exists helper function on fncachestore
Adrian Buehlmann <adrian@cadifra.com>
parents: 17782
diff changeset
1199 ef = self.encode(f)
df55ce6854c3 store: add new _exists helper function on fncachestore
Adrian Buehlmann <adrian@cadifra.com>
parents: 17782
diff changeset
1200 try:
df55ce6854c3 store: add new _exists helper function on fncachestore
Adrian Buehlmann <adrian@cadifra.com>
parents: 17782
diff changeset
1201 self.getsize(ef)
df55ce6854c3 store: add new _exists helper function on fncachestore
Adrian Buehlmann <adrian@cadifra.com>
parents: 17782
diff changeset
1202 return True
49306
2e726c934fcd py3: catch FileNotFoundError instead of checking errno == ENOENT
Manuel Jacob <me@manueljacob.de>
parents: 49284
diff changeset
1203 except FileNotFoundError:
17783
df55ce6854c3 store: add new _exists helper function on fncachestore
Adrian Buehlmann <adrian@cadifra.com>
parents: 17782
diff changeset
1204 return False
df55ce6854c3 store: add new _exists helper function on fncachestore
Adrian Buehlmann <adrian@cadifra.com>
parents: 17782
diff changeset
1205
17745
b9a56b816ff2 store: add a contains method to fncachestore
smuralid
parents: 17744
diff changeset
1206 def __contains__(self, path):
b9a56b816ff2 store: add a contains method to fncachestore
smuralid
parents: 17744
diff changeset
1207 '''Checks if the store contains path'''
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1208 path = b"/".join((b"data", path))
17782
8095306c1fb2 store: move __contains__() implementation from class fncache into fncachestore
Adrian Buehlmann <adrian@cadifra.com>
parents: 17747
diff changeset
1209 # check for files (exact match)
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1210 e = path + b'.i'
17784
73e1ab39792c store: fncache may contain non-existent entries (fixes b9a56b816ff2)
Adrian Buehlmann <adrian@cadifra.com>
parents: 17783
diff changeset
1211 if e in self.fncache and self._exists(e):
17782
8095306c1fb2 store: move __contains__() implementation from class fncache into fncachestore
Adrian Buehlmann <adrian@cadifra.com>
parents: 17747
diff changeset
1212 return True
8095306c1fb2 store: move __contains__() implementation from class fncache into fncachestore
Adrian Buehlmann <adrian@cadifra.com>
parents: 17747
diff changeset
1213 # now check for directories (prefix match)
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1214 if not path.endswith(b'/'):
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1215 path += b'/'
17782
8095306c1fb2 store: move __contains__() implementation from class fncache into fncachestore
Adrian Buehlmann <adrian@cadifra.com>
parents: 17747
diff changeset
1216 for e in self.fncache:
17784
73e1ab39792c store: fncache may contain non-existent entries (fixes b9a56b816ff2)
Adrian Buehlmann <adrian@cadifra.com>
parents: 17783
diff changeset
1217 if e.startswith(path) and self._exists(e):
17782
8095306c1fb2 store: move __contains__() implementation from class fncache into fncachestore
Adrian Buehlmann <adrian@cadifra.com>
parents: 17747
diff changeset
1218 return True
8095306c1fb2 store: move __contains__() implementation from class fncache into fncachestore
Adrian Buehlmann <adrian@cadifra.com>
parents: 17747
diff changeset
1219 return False