util: create new abstraction for compression engines Currently, util.py has "compressors" and "decompressors" dicts mapping compression algorithms to callables returning objects that perform well-defined operations. In addition, revlog.py has code for calling into a compressor or decompressor explicitly. And, there is code in the wire protocol for performing zlib compression. The 3rd party lz4revlog extension has demonstrated the utility of supporting alternative compression formats for revlog storage. But it stops short of supporting lz4 for bundles and the wire protocol. There are also plans to support zstd as a general compression replacement. So, there appears to be a market for a unified API for registering compression engines. This commit starts the process of establishing one. This commit establishes a base class/interface for defining compression engines and how they will be used. A collection class to hold references to registered compression engines has also been introduced. The built-in zlib, bz2, truncated bz2, and no-op compression engines are registered with a singleton instance of the collection class. The compression engine API will change once consumers are ported to the new API and some common patterns can be simplified at the engine API level. So don't get too attached to the API...

config: mark parser regexes as bytes explicitly r-strings are not transformed into bytes by our source transformer magic.

ui: explicitly open config files in binary mode This has been working mostly accidentally, but now it works explicitly.

help: fix double word usage "most" was used twice. (I fixed a grammar error before timeless spotted it!)

setup: move cffi stuff to mercurial/cffi This patch moves all setup*cffi stuff to mercurial/cffi to make the root directory cleaner. The idea was from mpm [1]: > It seems like we could have a fair amount of cffi definitions, and > cluttering the root directory (or mercurial/) with them is probably not > a great long-term solution. We could probably add a cffi/ directory > under mercurial/ to parallel pure/. [1]: https://www.mercurial-scm.org/pipermail/mercurial-devel/2016-July/086442.html

manifest: remove manifest.add and add memmfctx.write This removes one more dependency on the manifest class by moving the write functionality onto the memmanifestctx classes and changing the one consumer to use the new API. By moving the write path to a manifestctx, we now give the individual manifests control over how they're read and serialized. This will be useful in developing new manifest formats and storage systems.

context: add manifestctx property on changectx This allows us to access the manifestctx for a given commit. This will be used in a later patch to be able to copy the manifestctx when we want to make a new commit.

manifest: add copy to mfctx classes This adds copy functionality to the manifestctx classes. This will be used in an upcoming diff to copy a manifestctx during commit so we can modify the manifest before committing.

manifest: introduce memmanifestctx and memtreemanifestctx This introduces two new classes to represent in-memory manifest instances. Similar to memchangectx, this lets us prepare a manifest in memory, then in a future patch we will add the apis that can commit this in memory structure.

manifestctx: add _revlog() function The `self._repo.manifestlog._revlog` code is getting copy and pasted a lot in manifestctx. Let's make it a function so it can be reused. This will make future patches cleaner too.

manifest: remove manifest.find As part of removing dependencies on manifest, this drops the find function and fixes up the two existing callers to use the equivalent apis on manifestctx.

changegroup: remove remaining uses of repo.manifest The remaining uses of repo.manifest in the changegroup module are treating the manifest exclusively as a revlog, so let's replace them with instances of the revlog directly. This is part of dropping all dependencies on repo.manifest in favor of repo.manifestlog.

treemanifest: fix a "treeinmem" case f2c5b9d48b29 (manifest: make treemanifestctx store the repo, 2016-10-18) broke most tests when run with treeinmem=True. The treeinmem mode can not be enabled by the user, so this did not break anything in practice, but it's useful to have it working for testing the treemanifest code.

perf: support measuring bdiff for all changeset related data The --all argument changes the behavior of `perfbdiff` to pull in fulltext revision pairs for all changes related to a changeset. The p1 and p2 manifests will be bdiffed against current. Every file that changed between p1 and current will have its file revisions loaded and bdiffed. This mode of operation effectively measured the bdiff time required for `hg commit`.

perf: support bdiffing multiple revisions in a single revlog This is useful for testing bdiff performance on several revision pairs at a time.

perf: prepare to handle multiple pairs in perfbdiff Before, we only supported benchmarking a single pair of texts with bdiff. We want to enable feeding larger corpora into this benchmark. So rewrite the code to support that.

py3: document why os.fsencode() can be used to get back bytes argv And a possible Windows issue. I'm sad we have to do such ugly hack, but that's the unicode on Python 3.

py3: update test-check-py3-compat.t output 4b1af1c867fa (scmutil: move util.termwidth()) changed where the import fails.

spelling: fixes of non-dictionary words

manifest: add __nonzero__ method This adds a __nonzero__ method to manifestdict. This isn't strictly necessary in the vanilla Mercurial implementation, since Python will handle nonzero checks by using __len__, but having it implemented here makes it easier for alternative implementations to implement __nonzero__ and have them be plug-n-play with the normal implementation.

py3: have bytes version of sys.argv sys.argv returns unicodes on Python 3. We need a bytes version for us. There was also a python bug/feature request which wanted then to implement one. They rejected and it is quoted in one of the comments that we can use fsencode() to get a bytes version of sys.argv. Though not sure about its correctness. Link to the comment: http://bugs.python.org/issue8776#msg217416 After this patch we will have pycompat.sysargv which will return us bytes version of sys.argv. If this patch goes in, i will like to make transformer rewrite sys.argv with pycompat.argv because there are lot of occurences.

util: use '\\' rather than using r'\' We need bytes, and I find this just a little more immediately obvious than doing rb'\'.

util: use pycompat urlunquote function

pycompat: introduce an alias for urllib.unquote We have to use unquote_to_bytes on Python 3, so we need an abstraction for this.

keyword: handle filectx _customcmp Suggested by Yuya Nishihara: https://www.mercurial-scm.org/pipermail/mercurial-devel/2016-October/089461.html Related to issue5364.

mail: do not print(), use ui.debug() instead Since print() can't take a bytes output, it's pretty useless in Mercurial on Python 3. As this is a debug message, switching to ui.debug() seems fine.

progress: obtain stderr from ui This will help Python 3 porting.

simplemerge: obtain stdout from ui This will help Python 3 porting.

profiling: obtain stderr from ui This will help Python 3 porting.

scmutil: ignore EPERM at os.utime, which avoids ambiguity at closing According to POSIX specification, just having group write access to a file causes EPERM at invocation of os.utime() with an explicit time information (e.g. working on the repository shared by group access permission). To ignore EPERM at closing file object in such case, this patch makes checkambigatclosing._checkambig() use filestat.avoidambig() introduced by previous patch. Some functions below imply this code path at truncation of an existing (= might be owned by another user) file. - strip() in repair.py, introduced by e38d85be978f - _playback() in transaction.py, introduced by 599912a62ff6 This is a variant of issue5418.

vfs: ignore EPERM at os.utime, which avoids ambiguity at renaming (issue5418) According to POSIX specification, just having group write access to a file causes EPERM at invocation of os.utime() with an explicit time information (e.g. working on the repository shared by group access permission). To ignore EPERM at renaming in such case, this patch makes vfs.rename() use filestat.avoidambig() introduced by previous patch.

util: add utility function to skip avoiding file stat ambiguity if EPERM Now, advancing stat.st_mtime by os.utime() is used to avoid file stat ambiguity. But according to POSIX specification, utime(2) with an explicit time information is permitted only for a process with: - the effective user ID equal to the user ID of the file, or - appropriate privileges http://pubs.opengroup.org/onlinepubs/9699919799/functions/utime.html Therefore, just having group write access to a file causes EPERM at applying os.utime() on it (e.g. working on the repository shared by group access permission). This patch adds class filestat utility function avoidamgig() to avoid file stat ambiguity but skip it if EPERM. It is reasonable to always ignore EPERM, because utime(2) causes EPERM only in the case described above (EACCES is used only for utime(2) with NULL).