zstd: vendor python-zstandard 0.5.0
As the commit message for the previous changeset says, we wish
for zstd to be a 1st class citizen in Mercurial. To make that
happen, we need to enable Python to talk to the zstd C API. And
that requires bindings.
This commit vendors a copy of existing Python bindings. Why do we
need to vendor? As the commit message of the previous commit says,
relying on systems in the wild to have the bindings or zstd present
is a losing proposition. By distributing the zstd and bindings with
Mercurial, we significantly increase our chances that zstd will
work. Since zstd will deliver a better end-user experience by
achieving better performance, this benefits our users. Another
reason is that the Python bindings still aren't stable and the
API is somewhat fluid. While Mercurial could be coded to target
multiple versions of the Python bindings, it is safer to bundle
an explicit, known working version.
The added Python bindings are mostly a fully-featured interface
to the zstd C API. They allow one-shot operations, streaming,
reading and writing from objects implements the file object
protocol, dictionary compression, control over low-level compression
parameters, and more. The Python bindings work on Python 2.6,
2.7, and 3.3+ and have been tested on Linux and Windows. There are
CFFI bindings, but they are lacking compared to the C extension.
Upstream work will be needed before we can support zstd with PyPy.
But it will be possible.
The files added in this commit come from Git commit
e637c1b214d5f869cf8116c550dcae23ec13b677 from
https://github.com/indygreg/python-zstandard and are added without
modifications. Some files from the upstream repository have been
omitted, namely files related to continuous integration.
In the spirit of full disclosure, I'm the maintainer of the
"python-zstandard" project and have authored 100% of the code
added in this commit. Unfortunately, the Python bindings have
not been formally code reviewed by anyone. While I've tested
much of the code thoroughly (I even have tests that fuzz APIs),
there's a good chance there are bugs, memory leaks, not well
thought out APIs, etc. If someone wants to review the code and
send feedback to the GitHub project, it would be greatly
appreciated.
Despite my involvement with both projects, my opinions of code
style differ from Mercurial's. The code in this commit introduces
numerous code style violations in Mercurial's linters. So, the code
is excluded from most lints. However, some violations I agree with.
These have been added to the known violations ignore list for now.
#require test-repo
$ . "$TESTDIR/helpers-testrepo.sh"
$ import_checker="$TESTDIR"/../contrib/import-checker.py
Run the doctests from the import checker, and make sure
it's working correctly.
$ TERM=dumb
$ export TERM
$ python -m doctest $import_checker
Run additional tests for the import checker
$ mkdir testpackage
$ touch testpackage/__init__.py
$ cat > testpackage/multiple.py << EOF
> from __future__ import absolute_import
> import os, sys
> EOF
$ cat > testpackage/unsorted.py << EOF
> from __future__ import absolute_import
> import sys
> import os
> EOF
$ cat > testpackage/stdafterlocal.py << EOF
> from __future__ import absolute_import
> from . import unsorted
> import os
> EOF
$ cat > testpackage/requirerelative.py << EOF
> from __future__ import absolute_import
> import testpackage.unsorted
> EOF
$ cat > testpackage/importalias.py << EOF
> from __future__ import absolute_import
> import ui
> EOF
$ cat > testpackage/relativestdlib.py << EOF
> from __future__ import absolute_import
> from .. import os
> EOF
$ cat > testpackage/symbolimport.py << EOF
> from __future__ import absolute_import
> from .unsorted import foo
> EOF
$ cat > testpackage/latesymbolimport.py << EOF
> from __future__ import absolute_import
> from . import unsorted
> from mercurial.node import hex
> EOF
$ cat > testpackage/multiplegroups.py << EOF
> from __future__ import absolute_import
> from . import unsorted
> from . import more
> EOF
$ mkdir testpackage/subpackage
$ cat > testpackage/subpackage/levelpriority.py << EOF
> from __future__ import absolute_import
> from . import foo
> from .. import parent
> EOF
$ touch testpackage/subpackage/foo.py
$ cat > testpackage/subpackage/__init__.py << EOF
> from __future__ import absolute_import
> from . import levelpriority # should not cause cycle
> EOF
$ cat > testpackage/subpackage/localimport.py << EOF
> from __future__ import absolute_import
> from . import foo
> def bar():
> # should not cause "higher-level import should come first"
> from .. import unsorted
> # but other errors should be detected
> from .. import more
> import testpackage.subpackage.levelpriority
> EOF
$ cat > testpackage/importmodulefromsub.py << EOF
> from __future__ import absolute_import
> from .subpackage import foo # not a "direct symbol import"
> EOF
$ cat > testpackage/importsymbolfromsub.py << EOF
> from __future__ import absolute_import
> from .subpackage import foo, nonmodule
> EOF
$ cat > testpackage/sortedentries.py << EOF
> from __future__ import absolute_import
> from . import (
> foo,
> bar,
> )
> EOF
$ cat > testpackage/importfromalias.py << EOF
> from __future__ import absolute_import
> from . import ui
> EOF
$ cat > testpackage/importfromrelative.py << EOF
> from __future__ import absolute_import
> from testpackage.unsorted import foo
> EOF
$ mkdir testpackage2
$ touch testpackage2/__init__.py
$ cat > testpackage2/latesymbolimport.py << EOF
> from __future__ import absolute_import
> from testpackage import unsorted
> from mercurial.node import hex
> EOF
$ python "$import_checker" testpackage*/*.py testpackage/subpackage/*.py
testpackage/importalias.py:2: ui module must be "as" aliased to uimod
testpackage/importfromalias.py:2: ui from testpackage must be "as" aliased to uimod
testpackage/importfromrelative.py:2: import should be relative: testpackage.unsorted
testpackage/importfromrelative.py:2: direct symbol import foo from testpackage.unsorted
testpackage/importsymbolfromsub.py:2: direct symbol import nonmodule from testpackage.subpackage
testpackage/latesymbolimport.py:3: symbol import follows non-symbol import: mercurial.node
testpackage/multiple.py:2: multiple imported names: os, sys
testpackage/multiplegroups.py:3: multiple "from . import" statements
testpackage/relativestdlib.py:2: relative import of stdlib module
testpackage/requirerelative.py:2: import should be relative: testpackage.unsorted
testpackage/sortedentries.py:2: imports from testpackage not lexically sorted: bar < foo
testpackage/stdafterlocal.py:3: stdlib import "os" follows local import: testpackage
testpackage/subpackage/levelpriority.py:3: higher-level import should come first: testpackage
testpackage/subpackage/localimport.py:7: multiple "from .. import" statements
testpackage/subpackage/localimport.py:8: import should be relative: testpackage.subpackage.levelpriority
testpackage/symbolimport.py:2: direct symbol import foo from testpackage.unsorted
testpackage/unsorted.py:3: imports not lexically sorted: os < sys
testpackage2/latesymbolimport.py:3: symbol import follows non-symbol import: mercurial.node
[1]
$ cd "$TESTDIR"/..
There are a handful of cases here that require renaming a module so it
doesn't overlap with a stdlib module name. There are also some cycles
here that we should still endeavor to fix, and some cycles will be
hidden by deduplication algorithm in the cycle detector, so fixing
these may expose other cycles.
Known-bad files are excluded by -X as some of them would produce unstable
outputs, which should be fixed later.
$ hg locate 'set:**.py or grep(r"^#!.*?python")' \
> 'tests/**.t' \
> -X contrib/debugshell.py \
> -X contrib/python-zstandard/ \
> -X contrib/win32/hgwebdir_wsgi.py \
> -X doc/gendoc.py \
> -X doc/hgmanpage.py \
> -X i18n/posplit \
> -X tests/test-hgweb-auth.py \
> -X tests/hypothesishelpers.py \
> -X tests/test-ctxmanager.py \
> -X tests/test-lock.py \
> -X tests/test-verify-repo-operations.py \
> -X tests/test-hook.t \
> -X tests/test-import.t \
> -X tests/test-check-module-imports.t \
> -X tests/test-commit-interactive.t \
> -X tests/test-contrib-check-code.t \
> -X tests/test-extension.t \
> -X tests/test-hghave.t \
> -X tests/test-hgweb-no-path-info.t \
> -X tests/test-hgweb-no-request-uri.t \
> -X tests/test-hgweb-non-interactive.t \
> | sed 's-\\-/-g' | python "$import_checker" -