Gregory Szorc <gregory.szorc@gmail.com> [Mon, 07 Nov 2016 18:57:54 -0800] rev 30359
util: remove compressorobj API from compression engines
All callers have been replaced with "compressstream." It is quite
low-level and redundant with "compressstream." So eliminate it.
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 07 Nov 2016 18:54:35 -0800] rev 30358
hgweb: use compression engine API for zlib compression
More low-level compression code elimination because we now have nice
APIs.
This patch also demonstrates why we needed and implemented the
"level" option on the "compressstream" API.
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 07 Nov 2016 18:46:37 -0800] rev 30357
bundle2: use compressstream compression engine API
Compression engines now have an API for compressing a stream of
chunks. Switch to it and make low-level compression code disappear.
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 07 Nov 2016 18:57:07 -0800] rev 30356
util: add a stream compression API to compression engines
It is a common pattern throughout the code to perform compression
on an iterator of chunks, yielding an iterator of compressed chunks.
Let's formalize that as part of the compression engine API.
The zlib and bzip2 implementations allow an optional "level" option
to control the compression level. The default values are the same as
what the Python modules use. This option will be used in subsequent
patches.
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 07 Nov 2016 18:39:08 -0800] rev 30355
util: remove decompressors dict (API)
All in-tree consumers are now using the compengines registrar.
Extensions should switch to it as well.
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 07 Nov 2016 18:38:13 -0800] rev 30354
changegroup: use compression engines API
The new API doesn't have the equivalence for None and 'UN' so we
introduce code to use 'UN' explicitly.
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 07 Nov 2016 18:36:48 -0800] rev 30353
bundle2: use compression engines API to obtain decompressor
Like the recent change for the compressor side, this too is
relatively straightforward. We now store a compression engine
on the instance instead of a low-level decompressor. Again, this
will allow us to easily transition to different compression engine
APIs when they are implemented.
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 07 Nov 2016 18:34:51 -0800] rev 30352
util: remove compressors dict (API)
We no longer have any in-tree consumers of this object. Use
util.compengines instead.
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 07 Nov 2016 18:35:43 -0800] rev 30351
bundle2: use new compression engine API for compression
Now that we have a new API to define compression engines, let's put it
to use!
The new code stores a reference to the compression engine instead of
a low-level compressor object. This will allow us to more easily
transition to different APIs on the compression engine interface
once we implement them.
As part of this, we change the registration in bundletypes to use 'UN'
instead of None. Previously, util.compressors had the no-op compressor
registered under both the 'UN' and None keys. Since we're switching to
a new API, I don't see the point in carrying this dual registration
forward.
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 07 Nov 2016 18:31:39 -0800] rev 30350
util: create new abstraction for compression engines
Currently, util.py has "compressors" and "decompressors" dicts
mapping compression algorithms to callables returning objects that
perform well-defined operations. In addition, revlog.py has code
for calling into a compressor or decompressor explicitly. And, there
is code in the wire protocol for performing zlib compression.
The 3rd party lz4revlog extension has demonstrated the utility of
supporting alternative compression formats for revlog storage. But
it stops short of supporting lz4 for bundles and the wire protocol.
There are also plans to support zstd as a general compression
replacement.
So, there appears to be a market for a unified API for registering
compression engines. This commit starts the process of establishing
one.
This commit establishes a base class/interface for defining
compression engines and how they will be used. A collection class
to hold references to registered compression engines has also been
introduced.
The built-in zlib, bz2, truncated bz2, and no-op compression engines
are registered with a singleton instance of the collection class.
The compression engine API will change once consumers are ported
to the new API and some common patterns can be simplified at the
engine API level. So don't get too attached to the API...