annotate mercurial/utils/compression.py @ 43089:c59eb1560c44

py3: manually import getattr where it is needed The march continues. Differential Revision: https://phab.mercurial-scm.org/D7009
author Gregory Szorc <gregory.szorc@gmail.com>
date Sun, 06 Oct 2019 16:55:18 -0400
parents 687b865b95ad
children 9ce76155e0da
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
42041
3e47d1ec9da5 util: extract compression code in `mercurial.utils.compression`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41834
diff changeset
1 # compression.py - Mercurial utility functions for compression
8226
8b2cd04a6e97 put license and copyright info into comment blocks
Martin Geisler <mg@lazybytes.net>
parents: 8225
diff changeset
2 #
8b2cd04a6e97 put license and copyright info into comment blocks
Martin Geisler <mg@lazybytes.net>
parents: 8225
diff changeset
3 # This software may be used and distributed according to the terms of the
10263
25e572394f5c Update license to GPLv2+
Matt Mackall <mpm@selenic.com>
parents: 9996
diff changeset
4 # GNU General Public License version 2 or any later version.
1082
ce96e316278a Update util.py docstrings, fix walk test
mpm@selenic.com
parents: 1081
diff changeset
5
419
28511fc21073 [PATCH] file seperator handling for the other 'OS'
mpm@selenic.com
parents:
diff changeset
6
34137
a8994d08e4a2 doctest: use print_function and convert bytes to unicode where needed
Yuya Nishihara <yuya@tcha.org>
parents: 34134
diff changeset
7 from __future__ import absolute_import, print_function
27358
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
8
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
9 import bz2
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
10 import collections
26266
1e042e31bd0c changegroup: move all compressions utilities in util
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26201
diff changeset
11 import zlib
3769
96095d9ff1f8 Add encoding detection
Matt Mackall <mpm@selenic.com>
parents: 3767
diff changeset
12
43089
c59eb1560c44 py3: manually import getattr where it is needed
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43077
diff changeset
13 from ..pycompat import getattr
42041
3e47d1ec9da5 util: extract compression code in `mercurial.utils.compression`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41834
diff changeset
14 from .. import (
27358
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
15 error,
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
16 i18n,
28818
6041fb8f2da8 pycompat: add empty and queue to handle py3 divergence
timeless <timeless@mozdev.org>
parents: 28497
diff changeset
17 pycompat,
27358
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
18 )
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42044
diff changeset
19 from . import stringutil
3769
96095d9ff1f8 Add encoding detection
Matt Mackall <mpm@selenic.com>
parents: 3767
diff changeset
20
42041
3e47d1ec9da5 util: extract compression code in `mercurial.utils.compression`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41834
diff changeset
21 safehasattr = pycompat.safehasattr
32201
4462a981e8df base85: proxy through util module
Yuya Nishihara <yuya@tcha.org>
parents: 32154
diff changeset
22
14926
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
23
27358
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
24 _ = i18n._
14926
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
25
42041
3e47d1ec9da5 util: extract compression code in `mercurial.utils.compression`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41834
diff changeset
26 # compression code
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
27
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
28 SERVERROLE = b'server'
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
29 CLIENTROLE = b'client'
33793
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33737
diff changeset
30
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42044
diff changeset
31 compewireprotosupport = collections.namedtuple(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42044
diff changeset
32 r'compenginewireprotosupport',
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42044
diff changeset
33 (r'name', r'serverpriority', r'clientpriority'),
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42044
diff changeset
34 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42044
diff changeset
35
9097
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
36
8207
dd8d5be57d65 util: take propertycache from context.py
Matt Mackall <mpm@selenic.com>
parents: 8181
diff changeset
37 class propertycache(object):
dd8d5be57d65 util: take propertycache from context.py
Matt Mackall <mpm@selenic.com>
parents: 8181
diff changeset
38 def __init__(self, func):
dd8d5be57d65 util: take propertycache from context.py
Matt Mackall <mpm@selenic.com>
parents: 8181
diff changeset
39 self.func = func
dd8d5be57d65 util: take propertycache from context.py
Matt Mackall <mpm@selenic.com>
parents: 8181
diff changeset
40 self.name = func.__name__
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42044
diff changeset
41
8207
dd8d5be57d65 util: take propertycache from context.py
Matt Mackall <mpm@selenic.com>
parents: 8181
diff changeset
42 def __get__(self, obj, type=None):
dd8d5be57d65 util: take propertycache from context.py
Matt Mackall <mpm@selenic.com>
parents: 8181
diff changeset
43 result = self.func(obj)
18013
98c867ac1330 clfilter: add a propertycache that must be unfiltered
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 17962
diff changeset
44 self.cachevalue(obj, result)
8207
dd8d5be57d65 util: take propertycache from context.py
Matt Mackall <mpm@selenic.com>
parents: 8181
diff changeset
45 return result
dd8d5be57d65 util: take propertycache from context.py
Matt Mackall <mpm@selenic.com>
parents: 8181
diff changeset
46
18013
98c867ac1330 clfilter: add a propertycache that must be unfiltered
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 17962
diff changeset
47 def cachevalue(self, obj, value):
19951
d51c4d85ec23 spelling: random spell checker fixes
Mads Kiilerich <madski@unity3d.com>
parents: 19852
diff changeset
48 # __dict__ assignment required to bypass __setattr__ (eg: repoview)
19845
a1237a4b437d repoview: make propertycache.setcache compatible with repoview
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 19461
diff changeset
49 obj.__dict__[self.name] = value
18013
98c867ac1330 clfilter: add a propertycache that must be unfiltered
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 17962
diff changeset
50
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42044
diff changeset
51
30350
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
52 class compressormanager(object):
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
53 """Holds registrations of various compression engines.
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
54
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
55 This class essentially abstracts the differences between compression
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
56 engines to allow new compression formats to be added easily, possibly from
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
57 extensions.
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
58
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
59 Compressors are registered against the global instance by calling its
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
60 ``register()`` method.
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
61 """
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42044
diff changeset
62
30350
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
63 def __init__(self):
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
64 self._engines = {}
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
65 # Bundle spec human name to engine name.
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
66 self._bundlenames = {}
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
67 # Internal bundle identifier to engine name.
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
68 self._bundletypes = {}
30798
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
69 # Revlog header to engine name.
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
70 self._revlogheaders = {}
30761
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
71 # Wire proto identifier to engine name.
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
72 self._wiretypes = {}
30350
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
73
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
74 def __getitem__(self, key):
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
75 return self._engines[key]
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
76
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
77 def __contains__(self, key):
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
78 return key in self._engines
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
79
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
80 def __iter__(self):
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
81 return iter(self._engines.keys())
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
82
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
83 def register(self, engine):
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
84 """Register a compression engine with the manager.
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
85
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
86 The argument must be a ``compressionengine`` instance.
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
87 """
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
88 if not isinstance(engine, compressionengine):
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
89 raise ValueError(_(b'argument must be a compressionengine'))
30350
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
90
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
91 name = engine.name()
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
92
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
93 if name in self._engines:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42044
diff changeset
94 raise error.Abort(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
95 _(b'compression engine %s already registered') % name
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42044
diff changeset
96 )
30350
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
97
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
98 bundleinfo = engine.bundletype()
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
99 if bundleinfo:
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
100 bundlename, bundletype = bundleinfo
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
101
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
102 if bundlename in self._bundlenames:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42044
diff changeset
103 raise error.Abort(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
104 _(b'bundle name %s already registered') % bundlename
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42044
diff changeset
105 )
30350
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
106 if bundletype in self._bundletypes:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42044
diff changeset
107 raise error.Abort(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
108 _(b'bundle type %s already registered by %s')
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42044
diff changeset
109 % (bundletype, self._bundletypes[bundletype])
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42044
diff changeset
110 )
30350
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
111
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
112 # No external facing name declared.
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
113 if bundlename:
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
114 self._bundlenames[bundlename] = name
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
115
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
116 self._bundletypes[bundletype] = name
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
117
30761
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
118 wiresupport = engine.wireprotosupport()
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
119 if wiresupport:
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
120 wiretype = wiresupport.name
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
121 if wiretype in self._wiretypes:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42044
diff changeset
122 raise error.Abort(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42044
diff changeset
123 _(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
124 b'wire protocol compression %s already '
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
125 b'registered by %s'
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42044
diff changeset
126 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42044
diff changeset
127 % (wiretype, self._wiretypes[wiretype])
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42044
diff changeset
128 )
30761
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
129
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
130 self._wiretypes[wiretype] = name
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
131
30798
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
132 revlogheader = engine.revlogheader()
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
133 if revlogheader and revlogheader in self._revlogheaders:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42044
diff changeset
134 raise error.Abort(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
135 _(b'revlog header %s already registered by %s')
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42044
diff changeset
136 % (revlogheader, self._revlogheaders[revlogheader])
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42044
diff changeset
137 )
30798
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
138
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
139 if revlogheader:
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
140 self._revlogheaders[revlogheader] = name
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
141
30350
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
142 self._engines[name] = engine
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
143
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
144 @property
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
145 def supportedbundlenames(self):
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
146 return set(self._bundlenames.keys())
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
147
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
148 @property
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
149 def supportedbundletypes(self):
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
150 return set(self._bundletypes.keys())
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
151
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
152 def forbundlename(self, bundlename):
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
153 """Obtain a compression engine registered to a bundle name.
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
154
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
155 Will raise KeyError if the bundle type isn't registered.
30438
90933e4e44fd util: check for compression engine availability before returning
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30437
diff changeset
156
90933e4e44fd util: check for compression engine availability before returning
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30437
diff changeset
157 Will abort if the engine is known but not available.
30350
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
158 """
30438
90933e4e44fd util: check for compression engine availability before returning
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30437
diff changeset
159 engine = self._engines[self._bundlenames[bundlename]]
90933e4e44fd util: check for compression engine availability before returning
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30437
diff changeset
160 if not engine.available():
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42044
diff changeset
161 raise error.Abort(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
162 _(b'compression engine %s could not be loaded') % engine.name()
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42044
diff changeset
163 )
30438
90933e4e44fd util: check for compression engine availability before returning
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30437
diff changeset
164 return engine
30350
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
165
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
166 def forbundletype(self, bundletype):
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
167 """Obtain a compression engine registered to a bundle type.
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
168
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
169 Will raise KeyError if the bundle type isn't registered.
30438
90933e4e44fd util: check for compression engine availability before returning
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30437
diff changeset
170
90933e4e44fd util: check for compression engine availability before returning
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30437
diff changeset
171 Will abort if the engine is known but not available.
30350
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
172 """
30438
90933e4e44fd util: check for compression engine availability before returning
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30437
diff changeset
173 engine = self._engines[self._bundletypes[bundletype]]
90933e4e44fd util: check for compression engine availability before returning
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30437
diff changeset
174 if not engine.available():
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42044
diff changeset
175 raise error.Abort(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
176 _(b'compression engine %s could not be loaded') % engine.name()
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42044
diff changeset
177 )
30438
90933e4e44fd util: check for compression engine availability before returning
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30437
diff changeset
178 return engine
30350
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
179
30761
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
180 def supportedwireengines(self, role, onlyavailable=True):
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
181 """Obtain compression engines that support the wire protocol.
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
182
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
183 Returns a list of engines in prioritized order, most desired first.
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
184
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
185 If ``onlyavailable`` is set, filter out engines that can't be
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
186 loaded.
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
187 """
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
188 assert role in (SERVERROLE, CLIENTROLE)
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
189
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
190 attr = b'serverpriority' if role == SERVERROLE else b'clientpriority'
30761
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
191
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
192 engines = [self._engines[e] for e in self._wiretypes.values()]
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
193 if onlyavailable:
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
194 engines = [e for e in engines if e.available()]
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
195
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
196 def getkey(e):
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
197 # Sort first by priority, highest first. In case of tie, sort
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
198 # alphabetically. This is arbitrary, but ensures output is
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
199 # stable.
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
200 w = e.wireprotosupport()
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
201 return -1 * getattr(w, attr), w.name
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
202
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
203 return list(sorted(engines, key=getkey))
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
204
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
205 def forwiretype(self, wiretype):
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
206 engine = self._engines[self._wiretypes[wiretype]]
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
207 if not engine.available():
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42044
diff changeset
208 raise error.Abort(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
209 _(b'compression engine %s could not be loaded') % engine.name()
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42044
diff changeset
210 )
30761
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
211 return engine
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
212
30798
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
213 def forrevlogheader(self, header):
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
214 """Obtain a compression engine registered to a revlog header.
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
215
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
216 Will raise KeyError if the revlog header value isn't registered.
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
217 """
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
218 return self._engines[self._revlogheaders[header]]
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
219
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42044
diff changeset
220
30350
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
221 compengines = compressormanager()
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
222
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42044
diff changeset
223
30350
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
224 class compressionengine(object):
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
225 """Base class for compression engines.
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
226
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
227 Compression engines must implement the interface defined by this class.
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
228 """
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42044
diff changeset
229
30350
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
230 def name(self):
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
231 """Returns the name of the compression engine.
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
232
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
233 This is the key the engine is registered under.
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
234
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
235 This method must be implemented.
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
236 """
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
237 raise NotImplementedError()
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
238
30437
64d7275445d0 util: expose an "available" API on compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30418
diff changeset
239 def available(self):
64d7275445d0 util: expose an "available" API on compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30418
diff changeset
240 """Whether the compression engine is available.
64d7275445d0 util: expose an "available" API on compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30418
diff changeset
241
64d7275445d0 util: expose an "available" API on compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30418
diff changeset
242 The intent of this method is to allow optional compression engines
64d7275445d0 util: expose an "available" API on compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30418
diff changeset
243 that may not be available in all installations (such as engines relying
64d7275445d0 util: expose an "available" API on compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30418
diff changeset
244 on C extensions that may not be present).
64d7275445d0 util: expose an "available" API on compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30418
diff changeset
245 """
64d7275445d0 util: expose an "available" API on compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30418
diff changeset
246 return True
64d7275445d0 util: expose an "available" API on compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30418
diff changeset
247
30350
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
248 def bundletype(self):
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
249 """Describes bundle identifiers for this engine.
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
250
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
251 If this compression engine isn't supported for bundles, returns None.
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
252
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
253 If this engine can be used for bundles, returns a 2-tuple of strings of
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
254 the user-facing "bundle spec" compression name and an internal
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
255 identifier used to denote the compression format within bundles. To
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
256 exclude the name from external usage, set the first element to ``None``.
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
257
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
258 If bundle compression is supported, the class must also implement
30359
673f0fdc1046 util: remove compressorobj API from compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30356
diff changeset
259 ``compressstream`` and `decompressorreader``.
31792
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31777
diff changeset
260
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31777
diff changeset
261 The docstring of this method is used in the help system to tell users
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31777
diff changeset
262 about this engine.
30350
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
263 """
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
264 return None
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
265
30761
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
266 def wireprotosupport(self):
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
267 """Declare support for this compression format on the wire protocol.
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
268
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
269 If this compression engine isn't supported for compressing wire
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
270 protocol payloads, returns None.
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
271
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
272 Otherwise, returns ``compenginewireprotosupport`` with the following
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
273 fields:
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
274
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
275 * String format identifier
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
276 * Integer priority for the server
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
277 * Integer priority for the client
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
278
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
279 The integer priorities are used to order the advertisement of format
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
280 support by server and client. The highest integer is advertised
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
281 first. Integers with non-positive values aren't advertised.
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
282
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
283 The priority values are somewhat arbitrary and only used for default
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
284 ordering. The relative order can be changed via config options.
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
285
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
286 If wire protocol compression is supported, the class must also implement
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
287 ``compressstream`` and ``decompressorreader``.
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
288 """
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
289 return None
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
290
30798
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
291 def revlogheader(self):
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
292 """Header added to revlog chunks that identifies this engine.
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
293
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
294 If this engine can be used to compress revlogs, this method should
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
295 return the bytes used to identify chunks compressed with this engine.
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
296 Else, the method should return ``None`` to indicate it does not
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
297 participate in revlog compression.
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
298 """
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
299 return None
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
300
30356
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30355
diff changeset
301 def compressstream(self, it, opts=None):
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30355
diff changeset
302 """Compress an iterator of chunks.
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30355
diff changeset
303
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30355
diff changeset
304 The method receives an iterator (ideally a generator) of chunks of
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30355
diff changeset
305 bytes to be compressed. It returns an iterator (ideally a generator)
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30355
diff changeset
306 of bytes of chunks representing the compressed output.
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30355
diff changeset
307
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30355
diff changeset
308 Optionally accepts an argument defining how to perform compression.
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30355
diff changeset
309 Each engine treats this argument differently.
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30355
diff changeset
310 """
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30355
diff changeset
311 raise NotImplementedError()
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30355
diff changeset
312
30350
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
313 def decompressorreader(self, fh):
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
314 """Perform decompression on a file object.
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
315
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
316 Argument is an object with a ``read(size)`` method that returns
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
317 compressed data. Return value is an object with a ``read(size)`` that
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
318 returns uncompressed data.
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
319 """
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
320 raise NotImplementedError()
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
321
30794
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
322 def revlogcompressor(self, opts=None):
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
323 """Obtain an object that can be used to compress revlog entries.
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
324
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
325 The object has a ``compress(data)`` method that compresses binary
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
326 data. This method returns compressed binary data or ``None`` if
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
327 the data could not be compressed (too small, not compressible, etc).
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
328 The returned data should have a header uniquely identifying this
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
329 compression format so decompression can be routed to this engine.
30798
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
330 This header should be identified by the ``revlogheader()`` return
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
331 value.
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
332
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
333 The object has a ``decompress(data)`` method that decompresses
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
334 data. The method will only be called if ``data`` begins with
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
335 ``revlogheader()``. The method should return the raw, uncompressed
39777
b63dee7bd0d9 global: replace most uses of RevlogError with StorageError (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39571
diff changeset
336 data or raise a ``StorageError``.
30794
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
337
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
338 The object is reusable but is not thread safe.
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
339 """
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
340 raise NotImplementedError()
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
341
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42044
diff changeset
342
38713
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
343 class _CompressedStreamReader(object):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
344 def __init__(self, fh):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
345 if safehasattr(fh, 'unbufferedread'):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
346 self._reader = fh.unbufferedread
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
347 else:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
348 self._reader = fh.read
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
349 self._pending = []
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
350 self._pos = 0
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
351 self._eof = False
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
352
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
353 def _decompress(self, chunk):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
354 raise NotImplementedError()
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
355
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
356 def read(self, l):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
357 buf = []
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
358 while True:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
359 while self._pending:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
360 if len(self._pending[0]) > l + self._pos:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
361 newbuf = self._pending[0]
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42044
diff changeset
362 buf.append(newbuf[self._pos : self._pos + l])
38713
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
363 self._pos += l
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
364 return b''.join(buf)
38713
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
365
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
366 newbuf = self._pending.pop(0)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
367 if self._pos:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42044
diff changeset
368 buf.append(newbuf[self._pos :])
38713
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
369 l -= len(newbuf) - self._pos
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
370 else:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
371 buf.append(newbuf)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
372 l -= len(newbuf)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
373 self._pos = 0
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
374
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
375 if self._eof:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
376 return b''.join(buf)
38713
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
377 chunk = self._reader(65536)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
378 self._decompress(chunk)
39209
1af95139e5ec util: improve handling of truncated compressed streams
Joerg Sonnenberger <joerg@bec.de>
parents: 39060
diff changeset
379 if not chunk and not self._pending and not self._eof:
1af95139e5ec util: improve handling of truncated compressed streams
Joerg Sonnenberger <joerg@bec.de>
parents: 39060
diff changeset
380 # No progress and no new data, bail out
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
381 return b''.join(buf)
38713
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
382
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42044
diff changeset
383
38713
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
384 class _GzipCompressedStreamReader(_CompressedStreamReader):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
385 def __init__(self, fh):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
386 super(_GzipCompressedStreamReader, self).__init__(fh)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
387 self._decompobj = zlib.decompressobj()
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42044
diff changeset
388
38713
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
389 def _decompress(self, chunk):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
390 newbuf = self._decompobj.decompress(chunk)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
391 if newbuf:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
392 self._pending.append(newbuf)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
393 d = self._decompobj.copy()
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
394 try:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
395 d.decompress(b'x')
38713
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
396 d.flush()
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
397 if d.unused_data == b'x':
38713
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
398 self._eof = True
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
399 except zlib.error:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
400 pass
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
401
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42044
diff changeset
402
38713
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
403 class _BZ2CompressedStreamReader(_CompressedStreamReader):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
404 def __init__(self, fh):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
405 super(_BZ2CompressedStreamReader, self).__init__(fh)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
406 self._decompobj = bz2.BZ2Decompressor()
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42044
diff changeset
407
38713
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
408 def _decompress(self, chunk):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
409 newbuf = self._decompobj.decompress(chunk)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
410 if newbuf:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
411 self._pending.append(newbuf)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
412 try:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
413 while True:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
414 newbuf = self._decompobj.decompress(b'')
38713
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
415 if newbuf:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
416 self._pending.append(newbuf)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
417 else:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
418 break
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
419 except EOFError:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
420 self._eof = True
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
421
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42044
diff changeset
422
38713
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
423 class _TruncatedBZ2CompressedStreamReader(_BZ2CompressedStreamReader):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
424 def __init__(self, fh):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
425 super(_TruncatedBZ2CompressedStreamReader, self).__init__(fh)
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
426 newbuf = self._decompobj.decompress(b'BZ')
38713
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
427 if newbuf:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
428 self._pending.append(newbuf)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
429
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42044
diff changeset
430
38713
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
431 class _ZstdCompressedStreamReader(_CompressedStreamReader):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
432 def __init__(self, fh, zstd):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
433 super(_ZstdCompressedStreamReader, self).__init__(fh)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
434 self._zstd = zstd
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
435 self._decompobj = zstd.ZstdDecompressor().decompressobj()
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42044
diff changeset
436
38713
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
437 def _decompress(self, chunk):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
438 newbuf = self._decompobj.decompress(chunk)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
439 if newbuf:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
440 self._pending.append(newbuf)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
441 try:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
442 while True:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
443 newbuf = self._decompobj.decompress(b'')
38713
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
444 if newbuf:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
445 self._pending.append(newbuf)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
446 else:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
447 break
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
448 except self._zstd.ZstdError:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
449 self._eof = True
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
450
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42044
diff changeset
451
30350
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
452 class _zlibengine(compressionengine):
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
453 def name(self):
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
454 return b'zlib'
30350
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
455
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
456 def bundletype(self):
31792
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31777
diff changeset
457 """zlib compression using the DEFLATE algorithm.
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31777
diff changeset
458
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31777
diff changeset
459 All Mercurial clients should support this format. The compression
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31777
diff changeset
460 algorithm strikes a reasonable balance between compression ratio
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31777
diff changeset
461 and size.
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31777
diff changeset
462 """
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
463 return b'gzip', b'GZ'
30350
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
464
30761
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
465 def wireprotosupport(self):
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
466 return compewireprotosupport(b'zlib', 20, 20)
30761
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
467
30798
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
468 def revlogheader(self):
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
469 return b'x'
30798
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
470
30356
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30355
diff changeset
471 def compressstream(self, it, opts=None):
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30355
diff changeset
472 opts = opts or {}
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30355
diff changeset
473
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
474 z = zlib.compressobj(opts.get(b'level', -1))
30356
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30355
diff changeset
475 for chunk in it:
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30355
diff changeset
476 data = z.compress(chunk)
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30355
diff changeset
477 # Not all calls to compress emit data. It is cheaper to inspect
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30355
diff changeset
478 # here than to feed empty chunks through generator.
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30355
diff changeset
479 if data:
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30355
diff changeset
480 yield data
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30355
diff changeset
481
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30355
diff changeset
482 yield z.flush()
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30355
diff changeset
483
30350
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
484 def decompressorreader(self, fh):
38713
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
485 return _GzipCompressedStreamReader(fh)
30350
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
486
30794
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
487 class zlibrevlogcompressor(object):
42042
aaececb4b066 compression: accept level management for zlib compression
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 42041
diff changeset
488 def __init__(self, level=None):
aaececb4b066 compression: accept level management for zlib compression
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 42041
diff changeset
489 self._level = level
aaececb4b066 compression: accept level management for zlib compression
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 42041
diff changeset
490
30794
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
491 def compress(self, data):
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
492 insize = len(data)
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
493 # Caller handles empty input case.
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
494 assert insize > 0
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
495
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
496 if insize < 44:
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
497 return None
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
498
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
499 elif insize <= 1000000:
42042
aaececb4b066 compression: accept level management for zlib compression
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 42041
diff changeset
500 if self._level is None:
aaececb4b066 compression: accept level management for zlib compression
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 42041
diff changeset
501 compressed = zlib.compress(data)
aaececb4b066 compression: accept level management for zlib compression
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 42041
diff changeset
502 else:
aaececb4b066 compression: accept level management for zlib compression
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 42041
diff changeset
503 compressed = zlib.compress(data, self._level)
30794
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
504 if len(compressed) < insize:
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
505 return compressed
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
506 return None
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
507
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
508 # zlib makes an internal copy of the input buffer, doubling
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
509 # memory usage for large inputs. So do streaming compression
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
510 # on large inputs.
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
511 else:
42042
aaececb4b066 compression: accept level management for zlib compression
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 42041
diff changeset
512 if self._level is None:
aaececb4b066 compression: accept level management for zlib compression
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 42041
diff changeset
513 z = zlib.compressobj()
aaececb4b066 compression: accept level management for zlib compression
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 42041
diff changeset
514 else:
aaececb4b066 compression: accept level management for zlib compression
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 42041
diff changeset
515 z = zlib.compressobj(level=self._level)
30794
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
516 parts = []
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
517 pos = 0
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
518 while pos < insize:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42044
diff changeset
519 pos2 = pos + 2 ** 20
30794
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
520 parts.append(z.compress(data[pos:pos2]))
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
521 pos = pos2
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
522 parts.append(z.flush())
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
523
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
524 if sum(map(len, parts)) < insize:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
525 return b''.join(parts)
30794
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
526 return None
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
527
30798
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
528 def decompress(self, data):
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
529 try:
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
530 return zlib.decompress(data)
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
531 except zlib.error as e:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42044
diff changeset
532 raise error.StorageError(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
533 _(b'revlog decompress error: %s')
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42044
diff changeset
534 % stringutil.forcebytestr(e)
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42044
diff changeset
535 )
30798
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
536
30794
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
537 def revlogcompressor(self, opts=None):
42043
1fac9b931d46 compression: introduce a `storage.revlog.zlib.level` configuration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 42042
diff changeset
538 level = None
1fac9b931d46 compression: introduce a `storage.revlog.zlib.level` configuration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 42042
diff changeset
539 if opts is not None:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
540 level = opts.get(b'zlib.level')
42043
1fac9b931d46 compression: introduce a `storage.revlog.zlib.level` configuration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 42042
diff changeset
541 return self.zlibrevlogcompressor(level)
30794
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
542
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42044
diff changeset
543
30350
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
544 compengines.register(_zlibengine())
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
545
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42044
diff changeset
546
30350
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
547 class _bz2engine(compressionengine):
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
548 def name(self):
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
549 return b'bz2'
30350
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
550
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
551 def bundletype(self):
31792
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31777
diff changeset
552 """An algorithm that produces smaller bundles than ``gzip``.
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31777
diff changeset
553
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31777
diff changeset
554 All Mercurial clients should support this format.
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31777
diff changeset
555
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31777
diff changeset
556 This engine will likely produce smaller bundles than ``gzip`` but
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31777
diff changeset
557 will be significantly slower, both during compression and
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31777
diff changeset
558 decompression.
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31777
diff changeset
559
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31777
diff changeset
560 If available, the ``zstd`` engine can yield similar or better
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31777
diff changeset
561 compression at much higher speeds.
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31777
diff changeset
562 """
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
563 return b'bzip2', b'BZ'
30350
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
564
30761
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
565 # We declare a protocol name but don't advertise by default because
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
566 # it is slow.
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
567 def wireprotosupport(self):
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
568 return compewireprotosupport(b'bzip2', 0, 0)
30761
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
569
30356
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30355
diff changeset
570 def compressstream(self, it, opts=None):
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30355
diff changeset
571 opts = opts or {}
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
572 z = bz2.BZ2Compressor(opts.get(b'level', 9))
30356
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30355
diff changeset
573 for chunk in it:
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30355
diff changeset
574 data = z.compress(chunk)
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30355
diff changeset
575 if data:
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30355
diff changeset
576 yield data
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30355
diff changeset
577
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30355
diff changeset
578 yield z.flush()
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30355
diff changeset
579
30350
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
580 def decompressorreader(self, fh):
38713
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
581 return _BZ2CompressedStreamReader(fh)
30350
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
582
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42044
diff changeset
583
30350
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
584 compengines.register(_bz2engine())
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
585
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42044
diff changeset
586
30350
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
587 class _truncatedbz2engine(compressionengine):
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
588 def name(self):
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
589 return b'bz2truncated'
30350
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
590
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
591 def bundletype(self):
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
592 return None, b'_truncatedBZ'
30350
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
593
30359
673f0fdc1046 util: remove compressorobj API from compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30356
diff changeset
594 # We don't implement compressstream because it is hackily handled elsewhere.
30350
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
595
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
596 def decompressorreader(self, fh):
38713
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
597 return _TruncatedBZ2CompressedStreamReader(fh)
30350
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
598
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42044
diff changeset
599
30350
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
600 compengines.register(_truncatedbz2engine())
30265
6a8aff737a17 util: put compression code next to each other
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30181
diff changeset
601
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42044
diff changeset
602
30350
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
603 class _noopengine(compressionengine):
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
604 def name(self):
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
605 return b'none'
30350
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
606
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
607 def bundletype(self):
31792
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31777
diff changeset
608 """No compression is performed.
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31777
diff changeset
609
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31777
diff changeset
610 Use this compression engine to explicitly disable compression.
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31777
diff changeset
611 """
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
612 return b'none', b'UN'
30350
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
613
30761
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
614 # Clients always support uncompressed payloads. Servers don't because
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
615 # unless you are on a fast network, uncompressed payloads can easily
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
616 # saturate your network pipe.
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
617 def wireprotosupport(self):
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
618 return compewireprotosupport(b'none', 0, 10)
30761
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
619
30798
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
620 # We don't implement revlogheader because it is handled specially
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
621 # in the revlog class.
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
622
30356
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30355
diff changeset
623 def compressstream(self, it, opts=None):
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30355
diff changeset
624 return it
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30355
diff changeset
625
30350
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
626 def decompressorreader(self, fh):
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
627 return fh
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
628
30794
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
629 class nooprevlogcompressor(object):
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
630 def compress(self, data):
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
631 return None
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
632
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
633 def revlogcompressor(self, opts=None):
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
634 return self.nooprevlogcompressor()
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
635
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42044
diff changeset
636
30350
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30332
diff changeset
637 compengines.register(_noopengine())
26266
1e042e31bd0c changegroup: move all compressions utilities in util
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26201
diff changeset
638
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42044
diff changeset
639
30442
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30438
diff changeset
640 class _zstdengine(compressionengine):
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30438
diff changeset
641 def name(self):
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
642 return b'zstd'
30442
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30438
diff changeset
643
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30438
diff changeset
644 @propertycache
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30438
diff changeset
645 def _module(self):
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30438
diff changeset
646 # Not all installs have the zstd module available. So defer importing
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30438
diff changeset
647 # until first access.
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30438
diff changeset
648 try:
42041
3e47d1ec9da5 util: extract compression code in `mercurial.utils.compression`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41834
diff changeset
649 from .. import zstd
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42044
diff changeset
650
30442
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30438
diff changeset
651 # Force delayed import.
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30438
diff changeset
652 zstd.__version__
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30438
diff changeset
653 return zstd
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30438
diff changeset
654 except ImportError:
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30438
diff changeset
655 return None
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30438
diff changeset
656
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30438
diff changeset
657 def available(self):
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30438
diff changeset
658 return bool(self._module)
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30438
diff changeset
659
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30438
diff changeset
660 def bundletype(self):
31792
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31777
diff changeset
661 """A modern compression algorithm that is fast and highly flexible.
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31777
diff changeset
662
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31777
diff changeset
663 Only supported by Mercurial 4.1 and newer clients.
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31777
diff changeset
664
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31777
diff changeset
665 With the default settings, zstd compression is both faster and yields
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31777
diff changeset
666 better compression than ``gzip``. It also frequently yields better
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31777
diff changeset
667 compression than ``bzip2`` while operating at much higher speeds.
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31777
diff changeset
668
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31777
diff changeset
669 If this engine is available and backwards compatibility is not a
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31777
diff changeset
670 concern, it is likely the best available engine.
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31777
diff changeset
671 """
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
672 return b'zstd', b'ZS'
30442
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30438
diff changeset
673
30761
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
674 def wireprotosupport(self):
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
675 return compewireprotosupport(b'zstd', 50, 50)
30761
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
676
30798
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
677 def revlogheader(self):
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
678 return b'\x28'
30798
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
679
30442
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30438
diff changeset
680 def compressstream(self, it, opts=None):
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30438
diff changeset
681 opts = opts or {}
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30438
diff changeset
682 # zstd level 3 is almost always significantly faster than zlib
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30438
diff changeset
683 # while providing no worse compression. It strikes a good balance
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30438
diff changeset
684 # between speed and compression.
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
685 level = opts.get(b'level', 3)
30442
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30438
diff changeset
686
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30438
diff changeset
687 zstd = self._module
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30438
diff changeset
688 z = zstd.ZstdCompressor(level=level).compressobj()
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30438
diff changeset
689 for chunk in it:
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30438
diff changeset
690 data = z.compress(chunk)
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30438
diff changeset
691 if data:
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30438
diff changeset
692 yield data
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30438
diff changeset
693
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30438
diff changeset
694 yield z.flush()
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30438
diff changeset
695
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30438
diff changeset
696 def decompressorreader(self, fh):
38713
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
697 return _ZstdCompressedStreamReader(fh, self._module)
30442
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30438
diff changeset
698
30794
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
699 class zstdrevlogcompressor(object):
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
700 def __init__(self, zstd, level=3):
37496
1765ed63db40 util: drop write_content_size=True
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37455
diff changeset
701 # TODO consider omitting frame magic to save 4 bytes.
1765ed63db40 util: drop write_content_size=True
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37455
diff changeset
702 # This writes content sizes into the frame header. That is
1765ed63db40 util: drop write_content_size=True
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37455
diff changeset
703 # extra storage. But it allows a correct size memory allocation
1765ed63db40 util: drop write_content_size=True
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37455
diff changeset
704 # to hold the result.
1765ed63db40 util: drop write_content_size=True
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37455
diff changeset
705 self._cctx = zstd.ZstdCompressor(level=level)
30798
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
706 self._dctx = zstd.ZstdDecompressor()
30794
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
707 self._compinsize = zstd.COMPRESSION_RECOMMENDED_INPUT_SIZE
30798
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
708 self._decompinsize = zstd.DECOMPRESSION_RECOMMENDED_INPUT_SIZE
30794
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
709
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
710 def compress(self, data):
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
711 insize = len(data)
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
712 # Caller handles empty input case.
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
713 assert insize > 0
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
714
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
715 if insize < 50:
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
716 return None
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
717
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
718 elif insize <= 1000000:
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
719 compressed = self._cctx.compress(data)
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
720 if len(compressed) < insize:
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
721 return compressed
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
722 return None
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
723 else:
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
724 z = self._cctx.compressobj()
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
725 chunks = []
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
726 pos = 0
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
727 while pos < insize:
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
728 pos2 = pos + self._compinsize
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
729 chunk = z.compress(data[pos:pos2])
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
730 if chunk:
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
731 chunks.append(chunk)
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
732 pos = pos2
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
733 chunks.append(z.flush())
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
734
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
735 if sum(map(len, chunks)) < insize:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
736 return b''.join(chunks)
30794
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
737 return None
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
738
30798
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
739 def decompress(self, data):
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
740 insize = len(data)
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
741
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
742 try:
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
743 # This was measured to be faster than other streaming
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
744 # decompressors.
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
745 dobj = self._dctx.decompressobj()
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
746 chunks = []
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
747 pos = 0
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
748 while pos < insize:
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
749 pos2 = pos + self._decompinsize
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
750 chunk = dobj.decompress(data[pos:pos2])
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
751 if chunk:
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
752 chunks.append(chunk)
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
753 pos = pos2
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
754 # Frame should be exhausted, so no finish() API.
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
755
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
756 return b''.join(chunks)
30798
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
757 except Exception as e:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42044
diff changeset
758 raise error.StorageError(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
759 _(b'revlog decompress error: %s')
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42044
diff changeset
760 % stringutil.forcebytestr(e)
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42044
diff changeset
761 )
30798
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
762
30794
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
763 def revlogcompressor(self, opts=None):
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
764 opts = opts or {}
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
765 level = opts.get(b'zstd.level')
42044
bb271ec2fbfb compression: introduce a `storage.revlog.zstd.level` configuration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 42043
diff changeset
766 if level is None:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
767 level = opts.get(b'level')
42044
bb271ec2fbfb compression: introduce a `storage.revlog.zstd.level` configuration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 42043
diff changeset
768 if level is None:
bb271ec2fbfb compression: introduce a `storage.revlog.zstd.level` configuration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 42043
diff changeset
769 level = 3
bb271ec2fbfb compression: introduce a `storage.revlog.zstd.level` configuration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 42043
diff changeset
770 return self.zstdrevlogcompressor(self._module, level=level)
30794
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
771
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42044
diff changeset
772
30442
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30438
diff changeset
773 compengines.register(_zstdengine())
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30438
diff changeset
774
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42044
diff changeset
775
31792
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31777
diff changeset
776 def bundlecompressiontopics():
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31777
diff changeset
777 """Obtains a list of available bundle compressions for use in help."""
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31777
diff changeset
778 # help.makeitemsdocs() expects a dict of names to items with a .__doc__.
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31777
diff changeset
779 items = {}
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31777
diff changeset
780
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31777
diff changeset
781 # We need to format the docstring. So use a dummy object/type to hold it
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31777
diff changeset
782 # rather than mutating the original.
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31777
diff changeset
783 class docobject(object):
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31777
diff changeset
784 pass
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31777
diff changeset
785
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31777
diff changeset
786 for name in compengines:
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31777
diff changeset
787 engine = compengines[name]
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31777
diff changeset
788
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31777
diff changeset
789 if not engine.available():
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31777
diff changeset
790 continue
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31777
diff changeset
791
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31777
diff changeset
792 bt = engine.bundletype()
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31777
diff changeset
793 if not bt or not bt[0]:
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31777
diff changeset
794 continue
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31777
diff changeset
795
40251
3c89227788a2 py3: build help of compression engines in bytes
Yuya Nishihara <yuya@tcha.org>
parents: 40029
diff changeset
796 doc = b'``%s``\n %s' % (bt[0], pycompat.getdoc(engine.bundletype))
31792
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31777
diff changeset
797
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31777
diff changeset
798 value = docobject()
40251
3c89227788a2 py3: build help of compression engines in bytes
Yuya Nishihara <yuya@tcha.org>
parents: 40029
diff changeset
799 value.__doc__ = pycompat.sysstr(doc)
33818
ed04d7254a91 i18n: use saved object to get actual function information if available
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 33799
diff changeset
800 value._origdoc = engine.bundletype.__doc__
ed04d7254a91 i18n: use saved object to get actual function information if available
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 33799
diff changeset
801 value._origfunc = engine.bundletype
31792
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31777
diff changeset
802
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31777
diff changeset
803 items[bt[0]] = value
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31777
diff changeset
804
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31777
diff changeset
805 return items
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31777
diff changeset
806
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42044
diff changeset
807
33820
fa7e30efe05a i18n: get translation entries for description of each compression engines
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 33818
diff changeset
808 i18nfunctions = bundlecompressiontopics().values()