Mercurial > hg
comparison contrib/python-zstandard/NEWS.rst @ 37495:b1fb341d8a61
zstandard: vendor python-zstandard 0.9.0
This was just released. It features a number of goodies. More info at
https://gregoryszorc.com/blog/2018/04/09/release-of-python-zstandard-0.9/.
The clang-format ignore list was updated to reflect the new source
of files.
The project contains a vendored copy of zstandard 1.3.4. The old
version was 1.1.3. One of the changes between those versions is that
zstandard is now dual licensed BSD + GPLv2 and the patent rights grant
has been removed. Good riddance.
The API should be backwards compatible. So no changes in core
should be needed. However, there were a number of changes in the
library that we'll want to adapt to. Those will be addressed in
subsequent commits.
Differential Revision: https://phab.mercurial-scm.org/D3198
author | Gregory Szorc <gregory.szorc@gmail.com> |
---|---|
date | Mon, 09 Apr 2018 10:13:29 -0700 |
parents | 39d36c2db68e |
children | 73fef626dae3 |
comparison
equal
deleted
inserted
replaced
37494:1ce7a55b09d1 | 37495:b1fb341d8a61 |
---|---|
1 =============== | |
1 Version History | 2 Version History |
2 =============== | 3 =============== |
3 | 4 |
5 1.0.0 (not yet released) | |
6 ======================== | |
7 | |
8 Actions Blocking Release | |
9 ------------------------ | |
10 | |
11 * compression and decompression APIs that support ``io.rawIOBase`` interface | |
12 (#13). | |
13 * Refactor module names so C and CFFI extensions live under ``zstandard`` | |
14 package. | |
15 * Overall API design review. | |
16 * Use Python allocator where possible. | |
17 * Figure out what to do about experimental APIs not implemented by CFFI. | |
18 * APIs for auto adjusting compression parameters based on input size. e.g. | |
19 clamping the window log so it isn't too large for input. | |
20 * Consider allowing compressor and decompressor instances to be thread safe, | |
21 support concurrent operations. Or track when an operation is in progress and | |
22 refuse to let concurrent operations use the same instance. | |
23 * Support for magic-less frames for all decompression operations (``decompress()`` | |
24 doesn't work due to sniffing the content size and the lack of a ZSTD API to | |
25 sniff magic-less frames - this should be fixed in 1.3.5.). | |
26 * Audit for complete flushing when ending compression streams. | |
27 * Deprecate legacy APIs. | |
28 * Audit for ability to control read/write sizes on all APIs. | |
29 * Detect memory leaks via bench.py. | |
30 * Remove low-level compression parameters from ``ZstdCompressor.__init__`` and | |
31 require use of ``CompressionParameters``. | |
32 * Expose ``ZSTD_getFrameProgression()`` from more compressor types. | |
33 | |
34 Other Actions Not Blocking Release | |
35 --------------------------------------- | |
36 | |
37 * Support for block compression APIs. | |
38 * API for ensuring max memory ceiling isn't exceeded. | |
39 * Move off nose for testing. | |
40 | |
41 0.9.0 (released 2018-04-08) | |
42 =========================== | |
43 | |
44 Backwards Compatibility Notes | |
45 ----------------------------- | |
46 | |
47 * CFFI 1.11 or newer is now required (previous requirement was 1.8). | |
48 * The primary module is now ``zstandard``. Please change imports of ``zstd`` | |
49 and ``zstd_cffi`` to ``import zstandard``. See the README for more. Support | |
50 for importing the old names will be dropped in the next release. | |
51 * ``ZstdCompressor.read_from()`` and ``ZstdDecompressor.read_from()`` have | |
52 been renamed to ``read_to_iter()``. ``read_from()`` is aliased to the new | |
53 name and will be deleted in a future release. | |
54 * Support for Python 2.6 has been removed. | |
55 * Support for Python 3.3 has been removed. | |
56 * The ``selectivity`` argument to ``train_dictionary()`` has been removed, as | |
57 the feature disappeared from zstd 1.3. | |
58 * Support for legacy dictionaries has been removed. Cover dictionaries are now | |
59 the default. ``train_cover_dictionary()`` has effectively been renamed to | |
60 ``train_dictionary()``. | |
61 * The ``allow_empty`` argument from ``ZstdCompressor.compress()`` has been | |
62 deleted and the method now allows empty inputs to be compressed by default. | |
63 * ``estimate_compression_context_size()`` has been removed. Use | |
64 ``CompressionParameters.estimated_compression_context_size()`` instead. | |
65 * ``get_compression_parameters()`` has been removed. Use | |
66 ``CompressionParameters.from_level()`` instead. | |
67 * The arguments to ``CompressionParameters.__init__()`` have changed. If you | |
68 were using positional arguments before, the positions now map to different | |
69 arguments. It is recommended to use keyword arguments to construct | |
70 ``CompressionParameters`` instances. | |
71 * ``TARGETLENGTH_MAX`` constant has been removed (it disappeared from zstandard | |
72 1.3.4). | |
73 * ``ZstdCompressor.write_to()`` and ``ZstdDecompressor.write_to()`` have been | |
74 renamed to ``ZstdCompressor.stream_writer()`` and | |
75 ``ZstdDecompressor.stream_writer()``, respectively. The old names are still | |
76 aliased, but will be removed in the next major release. | |
77 * Content sizes are written into frame headers by default | |
78 (``ZstdCompressor(write_content_size=True)`` is now the default). | |
79 * ``CompressionParameters`` has been renamed to ``ZstdCompressionParameters`` | |
80 for consistency with other types. The old name is an alias and will be removed | |
81 in the next major release. | |
82 | |
83 Bug Fixes | |
84 --------- | |
85 | |
86 * Fixed memory leak in ``ZstdCompressor.copy_stream()`` (#40) (from 0.8.2). | |
87 * Fixed memory leak in ``ZstdDecompressor.copy_stream()`` (#35) (from 0.8.2). | |
88 * Fixed memory leak of ``ZSTD_DDict`` instances in CFFI's ``ZstdDecompressor``. | |
89 | |
90 New Features | |
91 ------------ | |
92 | |
93 * Bundlded zstandard library upgraded from 1.1.3 to 1.3.4. This delivers various | |
94 bug fixes and performance improvements. It also gives us access to newer | |
95 features. | |
96 * Support for negative compression levels. | |
97 * Support for *long distance matching* (facilitates compression ratios that approach | |
98 LZMA). | |
99 * Supporting for reading empty zstandard frames (with an embedded content size | |
100 of 0). | |
101 * Support for writing and partial support for reading zstandard frames without a | |
102 magic header. | |
103 * New ``stream_reader()`` API that exposes the ``io.RawIOBase`` interface (allows | |
104 you to ``.read()`` from a file-like object). | |
105 * Several minor features, bug fixes, and performance enhancements. | |
106 * Wheels for Linux and macOS are now provided with releases. | |
107 | |
108 Changes | |
109 ------- | |
110 | |
111 * Functions accepting bytes data now use the buffer protocol and can accept | |
112 more types (like ``memoryview`` and ``bytearray``) (#26). | |
113 * Add #includes so compilation on OS X and BSDs works (#20). | |
114 * New ``ZstdDecompressor.stream_reader()`` API to obtain a read-only i/o stream | |
115 of decompressed data for a source. | |
116 * New ``ZstdCompressor.stream_reader()`` API to obtain a read-only i/o stream of | |
117 compressed data for a source. | |
118 * Renamed ``ZstdDecompressor.read_from()`` to ``ZstdDecompressor.read_to_iter()``. | |
119 The old name is still available. | |
120 * Renamed ``ZstdCompressor.read_from()`` to ``ZstdCompressor.read_to_iter()``. | |
121 ``read_from()`` is still available at its old location. | |
122 * Introduce the ``zstandard`` module to import and re-export the C or CFFI | |
123 *backend* as appropriate. Behavior can be controlled via the | |
124 ``PYTHON_ZSTANDARD_IMPORT_POLICY`` environment variable. See README for | |
125 usage info. | |
126 * Vendored version of zstd upgraded to 1.3.4. | |
127 * Added module constants ``CONTENTSIZE_UNKNOWN`` and ``CONTENTSIZE_ERROR``. | |
128 * Add ``STRATEGY_BTULTRA`` compression strategy constant. | |
129 * Switch from deprecated ``ZSTD_getDecompressedSize()`` to | |
130 ``ZSTD_getFrameContentSize()`` replacement. | |
131 * ``ZstdCompressor.compress()`` can now compress empty inputs without requiring | |
132 special handling. | |
133 * ``ZstdCompressor`` and ``ZstdDecompressor`` now have a ``memory_size()`` | |
134 method for determining the current memory utilization of the underlying zstd | |
135 primitive. | |
136 * ``train_dictionary()`` has new arguments and functionality for trying multiple | |
137 variations of COVER parameters and selecting the best one. | |
138 * Added module constants ``LDM_MINMATCH_MIN``, ``LDM_MINMATCH_MAX``, and | |
139 ``LDM_BUCKETSIZELOG_MAX``. | |
140 * Converted all consumers to the zstandard *new advanced API*, which uses | |
141 ``ZSTD_compress_generic()`` | |
142 * ``CompressionParameters.__init__`` now accepts several more arguments, | |
143 including support for *long distance matching*. | |
144 * ``ZstdCompressionDict.__init__`` now accepts a ``dict_type`` argument that | |
145 controls how the dictionary should be interpreted. This can be used to | |
146 force the use of *content-only* dictionaries or to require the presence | |
147 of the dictionary magic header. | |
148 * ``ZstdCompressionDict.precompute_compress()`` can be used to precompute the | |
149 compression dictionary so it can efficiently be used with multiple | |
150 ``ZstdCompressor`` instances. | |
151 * Digested dictionaries are now stored in ``ZstdCompressionDict`` instances, | |
152 created automatically on first use, and automatically reused by all | |
153 ``ZstdDecompressor`` instances bound to that dictionary. | |
154 * All meaningful functions now accept keyword arguments. | |
155 * ``ZstdDecompressor.decompressobj()`` now accepts a ``write_size`` argument | |
156 to control how much work to perform on every decompressor invocation. | |
157 * ``ZstdCompressor.write_to()`` now exposes a ``tell()``, which exposes the | |
158 total number of bytes written so far. | |
159 * ``ZstdDecompressor.stream_reader()`` now supports ``seek()`` when moving | |
160 forward in the stream. | |
161 * Removed ``TARGETLENGTH_MAX`` constant. | |
162 * Added ``frame_header_size(data)`` function. | |
163 * Added ``frame_content_size(data)`` function. | |
164 * Consumers of ``ZSTD_decompress*`` have been switched to the new *advanced | |
165 decompression* API. | |
166 * ``ZstdCompressor`` and ``ZstdCompressionParams`` can now be constructed with | |
167 negative compression levels. | |
168 * ``ZstdDecompressor`` now accepts a ``max_window_size`` argument to limit the | |
169 amount of memory required for decompression operations. | |
170 * ``FORMAT_ZSTD1`` and ``FORMAT_ZSTD1_MAGICLESS`` constants to be used with | |
171 the ``format`` compression parameter to control whether the frame magic | |
172 header is written. | |
173 * ``ZstdDecompressor`` now accepts a ``format`` argument to control the | |
174 expected frame format. | |
175 * ``ZstdCompressor`` now has a ``frame_progression()`` method to return | |
176 information about the current compression operation. | |
177 * Error messages in CFFI no longer have ``b''`` literals. | |
178 * Compiler warnings and underlying overflow issues on 32-bit platforms have been | |
179 fixed. | |
180 * Builds in CI now build with compiler warnings as errors. This should hopefully | |
181 fix new compiler warnings from being introduced. | |
182 * Make ``ZstdCompressor(write_content_size=True)`` and | |
183 ``CompressionParameters(write_content_size=True)`` the default. | |
184 * ``CompressionParameters`` has been renamed to ``ZstdCompressionParameters``. | |
185 | |
186 0.8.2 (released 2018-02-22) | |
187 --------------------------- | |
188 | |
189 * Fixed memory leak in ``ZstdCompressor.copy_stream()`` (#40). | |
190 * Fixed memory leak in ``ZstdDecompressor.copy_stream()`` (#35). | |
191 | |
4 0.8.1 (released 2017-04-08) | 192 0.8.1 (released 2017-04-08) |
5 --------------------------- | 193 --------------------------- |
6 | 194 |
7 * Add #includes so compilation on OS X and BSDs works (#20). | 195 * Add #includes so compilation on OS X and BSDs works (#20). |
8 | 196 |
9 0.8.0 (released 2017-03-08) | 197 0.8.0 (released 2017-03-08) |
10 --------------------------- | 198 =========================== |
11 | 199 |
12 * CompressionParameters now has a estimated_compression_context_size() method. | 200 * CompressionParameters now has a estimated_compression_context_size() method. |
13 zstd.estimate_compression_context_size() is now deprecated and slated for | 201 zstd.estimate_compression_context_size() is now deprecated and slated for |
14 removal. | 202 removal. |
15 * Implemented a lot of fuzzing tests. | 203 * Implemented a lot of fuzzing tests. |
33 * Removed DictParameters type. | 221 * Removed DictParameters type. |
34 * train_dictionary() now accepts keyword arguments instead of a | 222 * train_dictionary() now accepts keyword arguments instead of a |
35 DictParameters instance to control dictionary generation. | 223 DictParameters instance to control dictionary generation. |
36 | 224 |
37 0.7.0 (released 2017-02-07) | 225 0.7.0 (released 2017-02-07) |
38 --------------------------- | 226 =========================== |
39 | 227 |
40 * Added zstd.get_frame_parameters() to obtain info about a zstd frame. | 228 * Added zstd.get_frame_parameters() to obtain info about a zstd frame. |
41 * Added ZstdDecompressor.decompress_content_dict_chain() for efficient | 229 * Added ZstdDecompressor.decompress_content_dict_chain() for efficient |
42 decompression of *content-only dictionary chains*. | 230 decompression of *content-only dictionary chains*. |
43 * CFFI module fully implemented; all tests run against both C extension and | 231 * CFFI module fully implemented; all tests run against both C extension and |
60 * CompressionParameters instances no longer are subscriptable nor behave | 248 * CompressionParameters instances no longer are subscriptable nor behave |
61 as tuples (backwards incompatible). Use attributes to obtain values. | 249 as tuples (backwards incompatible). Use attributes to obtain values. |
62 * DictParameters instances now expose their values as attributes. | 250 * DictParameters instances now expose their values as attributes. |
63 | 251 |
64 0.6.0 (released 2017-01-14) | 252 0.6.0 (released 2017-01-14) |
65 --------------------------- | 253 =========================== |
66 | 254 |
67 * Support for legacy zstd protocols (build time opt in feature). | 255 * Support for legacy zstd protocols (build time opt in feature). |
68 * Automation improvements to test against Python 3.6, latest versions | 256 * Automation improvements to test against Python 3.6, latest versions |
69 of Tox, more deterministic AppVeyor behavior. | 257 of Tox, more deterministic AppVeyor behavior. |
70 * CFFI "parser" improved to use a compiler preprocessor instead of rewriting | 258 * CFFI "parser" improved to use a compiler preprocessor instead of rewriting |
77 * ZstdCompressor.compressobj()'s flush() method now accepts an argument to | 265 * ZstdCompressor.compressobj()'s flush() method now accepts an argument to |
78 flush a block (as opposed to ending the stream). | 266 flush a block (as opposed to ending the stream). |
79 * Disallow compress(b'') when writing content sizes by default (issue #11). | 267 * Disallow compress(b'') when writing content sizes by default (issue #11). |
80 | 268 |
81 0.5.2 (released 2016-11-12) | 269 0.5.2 (released 2016-11-12) |
82 --------------------------- | 270 =========================== |
83 | 271 |
84 * more packaging fixes for source distribution | 272 * more packaging fixes for source distribution |
85 | 273 |
86 0.5.1 (released 2016-11-12) | 274 0.5.1 (released 2016-11-12) |
87 --------------------------- | 275 =========================== |
88 | 276 |
89 * setup_zstd.py is included in the source distribution | 277 * setup_zstd.py is included in the source distribution |
90 | 278 |
91 0.5.0 (released 2016-11-10) | 279 0.5.0 (released 2016-11-10) |
92 --------------------------- | 280 =========================== |
93 | 281 |
94 * Vendored version of zstd updated to 1.1.1. | 282 * Vendored version of zstd updated to 1.1.1. |
95 * Continuous integration for Python 3.6 and 3.7 | 283 * Continuous integration for Python 3.6 and 3.7 |
96 * Continuous integration for Conda | 284 * Continuous integration for Conda |
97 * Added compression and decompression APIs providing similar interfaces | 285 * Added compression and decompression APIs providing similar interfaces |
112 allows other projects with ``setup.py`` files to reuse the | 300 allows other projects with ``setup.py`` files to reuse the |
113 ``distutils`` code for this project without copying code. | 301 ``distutils`` code for this project without copying code. |
114 * The monolithic ``zstd.c`` file has been split into a header file defining | 302 * The monolithic ``zstd.c`` file has been split into a header file defining |
115 types and separate ``.c`` source files for the implementation. | 303 types and separate ``.c`` source files for the implementation. |
116 | 304 |
117 History of the Project | 305 Older History |
118 ====================== | 306 ============= |
119 | 307 |
120 2016-08-31 - Zstandard 1.0.0 is released and Gregory starts hacking on a | 308 2016-08-31 - Zstandard 1.0.0 is released and Gregory starts hacking on a |
121 Python extension for use by the Mercurial project. A very hacky prototype | 309 Python extension for use by the Mercurial project. A very hacky prototype |
122 is sent to the mercurial-devel list for RFC. | 310 is sent to the mercurial-devel list for RFC. |
123 | 311 |