shelve: move possible shelve file extensions to a single place This and a couple of following patches are a preparation to implementing obsolescense-enabled shelve which was discussed on a Sprint. If this refactoring is not done, shelve is going to look even more hackish than now. This particular commit introduces a slight behavior change. Previously, if only .hg/shelve/name.patch file exists, but .hg/name.hg does not, 'hg shelve -d name' would fail saying "shelve not found". Now deletion will only fail if .patch file does not exist (since .patch is used as an indicator of an existing shelve). Other shelve files being absent are skipped silently to accommodate for future introduction of obs-based shelve, which will mean that for some shelves .hg and .patch files exist, while for others .hg and .oshelve.

manifest: delete manifest.manifest class Now that nothing uses the primary manifest class, we can delete it.

localrepo: delete localrepo.manifest Now that nothing uses normal manifests, we can delete localrepo.manifest.

manifest: remove last uses of repo.manifest Now that all the functionality has been moved to manifestlog/manifestrevlog/etc, we can finally change all the uses of repo.manifest to use the new versions. A future diff will then delete repo.manifest. One additional change in this commit is to change repo.manifestlog to be a @storecache property instead of @property. This is required by some uses of repo.manifest require that it be settable (contrib/perf.py and the static http server). We can't do this in a prior change because we can't use @storecache on this until repo.manifest is no longer used anywhere.

manifest: add unionmanifestlog support As part of deprecating manifest, we need to make the union repo support manifestlog.

manifest: add bundlemanifestlog support As part of deprecating manifest.manifest we need to make bundlerepo support manifestlog.

manifest: make manifestlog use it's own cache As we start to make manifestlog the primary manifest source, the dependency on manifest.manifest will cause circular dependency problems. Let's break this dependency by making manifestlog use it's own cache. In a near future patch we will remove the previous manifest cache so we're not duplicating it.

manifest: delete unused dirlog and _newmanifest functions As part of migrating all manifest functionality out of manifest.manifest, let's migrate a couple spots off of manifest.dirlog() to use the revlog specific accessor. Then we can delete manifest.dirlog() and other unused functions.

manifest: move clearcaches to manifestlog This is part of removing all functionality from manifest.manifest so we can delete the class entirely.

manifest: remove usages of manifest.read Now that the two manifestctx implementations have working read() functions, let's remove the existing uses of manifest.read and drop the function.

manifest: remove dependency on manifestrevlog being able to create trees A future patch will be removing the read() function from the manifest class. Since manifestrevlog currently depends on the read function that manifest implements (as a derived class), we need to break the dependency from manifestrevlog to read(). We do this by adding an argument to manifestrevlog.write() which provides it with the ability to read a manifest. This is good in general because it further separates revlog as the storage format from the actual inmemory data structure implementation.

color: show mode warning based on ui.formatted ui.interactive is only for input and ui.formatted is for output.

protocol: drop unused import of zlib Something weird is happening that breaks pyflakes installed via 'pip install --user'. I haven't had a chance to finish debugging this, but this at least fixes the build.

hook: lower inflated use of sys.__stdout__ and __stderr__ They were introduced at 9f76df0edb7d, where sys.stdout could be replaced by sys.stderr. After that, we've changed the way of stdout redirection by afccc64eea73, so we no longer need to reference the original __stdout__ and __stderr__ objects. Let's move away from using __std*__ objects so we can simply wrap sys.std* objects for Python 3 porting.

hook: flush stdout before restoring stderr redirection There was a similar issue to 8b011ededfb2. If an in-process hook writes to stdout, the data may be buffered. In which case, stdout must be flushed before restoring its file descriptor. Otherwise, remaining data would be sent over the ssh wire and corrupts the protocol. Note that this is a different redirection from the one I've just removed.

hook: do not redirect stdout/err/in to ui while running in-process hooks (BC) It was introduced by a59058fd074a to address command-server issues. After that, I've made a complete fix by 69f86b937035, so we don't need to replace sys.stdio objects to protect the IPC channels. This change means we no longer see data written to sys.stdout/err by an in-process hook on command server. I think that's okay because the canonical way is to use ui functions and in-process hooks should respect the Mercurial API. This will help Python 3 porting, where sys.stdout is TextIO but ui.fout is BytesIO.

merge: change modified indicator to be 20 bytes Previously we indicated that the .hgsubstate file was dirty by adding a '+' to the end of its hash in the wctx manifest. This made is complicated to have new manifest implementations that rely on the node length being fixed. In previous patches we added added and modified node placeholders, so let's use those to indicate dirty here as well. It doesn't look like anything ever depended on this '+' (aside from it being different to the parent), so nothing else needed to change here.

dirstate: change added/modified placeholder hash length to 20 bytes Previously the added/modified placeholder hash for manifests generated from the dirstate was a 21byte long string consisting of the p1 file hash plus a single character to indicate an add or a modify. Normal hashes are only 20 bytes long. This makes it complicated to implement more efficient manifest implementations which rely on the hashes being fixed length. Let's change this hash to just be 20 bytes long, and rely on the astronomical improbability of an actual hash being these 20 bytes (just like we rely on no hash every being the nullid). This changes the possible behavior slightly in that the hash for all added/modified entries in the dirstate manifest will now be the same (so simple node comparisons would say they are equal), but we should never be doing simple node comparisons on these nodes even with the old hashes, because they did not accurately represent the content (i.e. two files based off the same p1 file node, with different working copy contents would have the same hash (even with the appended character) in the old scheme too, so we couldn't depend on the hashes period).

dirstate: change placeholder hash length to 20 bytes Previously the new-node placeholder hash for manifests generated from the dirstate was a 21byte long string of "!" characters. Normal hashes are only 20 bytes long. This makes it complicated to implement more efficient manifest implementations which rely on the hashes being fixed length. Let's change this hash to just be 20 bytes long, and rely on the astronomical improbability of an actual hash being 20 "!" bytes in a row (just like we rely on no hash ever being the nullid). A future diff will do this for added and modified dirstate markers as well, so we're putting the new newnodeid in node.py so there's a common place for these placeholders.

util: remove compressorobj API from compression engines All callers have been replaced with "compressstream." It is quite low-level and redundant with "compressstream." So eliminate it.

hgweb: use compression engine API for zlib compression More low-level compression code elimination because we now have nice APIs. This patch also demonstrates why we needed and implemented the "level" option on the "compressstream" API.

bundle2: use compressstream compression engine API Compression engines now have an API for compressing a stream of chunks. Switch to it and make low-level compression code disappear.

util: add a stream compression API to compression engines It is a common pattern throughout the code to perform compression on an iterator of chunks, yielding an iterator of compressed chunks. Let's formalize that as part of the compression engine API. The zlib and bzip2 implementations allow an optional "level" option to control the compression level. The default values are the same as what the Python modules use. This option will be used in subsequent patches.

util: remove decompressors dict (API) All in-tree consumers are now using the compengines registrar. Extensions should switch to it as well.

changegroup: use compression engines API The new API doesn't have the equivalence for None and 'UN' so we introduce code to use 'UN' explicitly.

bundle2: use compression engines API to obtain decompressor Like the recent change for the compressor side, this too is relatively straightforward. We now store a compression engine on the instance instead of a low-level decompressor. Again, this will allow us to easily transition to different compression engine APIs when they are implemented.

util: remove compressors dict (API) We no longer have any in-tree consumers of this object. Use util.compengines instead.

bundle2: use new compression engine API for compression Now that we have a new API to define compression engines, let's put it to use! The new code stores a reference to the compression engine instead of a low-level compressor object. This will allow us to more easily transition to different APIs on the compression engine interface once we implement them. As part of this, we change the registration in bundletypes to use 'UN' instead of None. Previously, util.compressors had the no-op compressor registered under both the 'UN' and None keys. Since we're switching to a new API, I don't see the point in carrying this dual registration forward.

util: create new abstraction for compression engines Currently, util.py has "compressors" and "decompressors" dicts mapping compression algorithms to callables returning objects that perform well-defined operations. In addition, revlog.py has code for calling into a compressor or decompressor explicitly. And, there is code in the wire protocol for performing zlib compression. The 3rd party lz4revlog extension has demonstrated the utility of supporting alternative compression formats for revlog storage. But it stops short of supporting lz4 for bundles and the wire protocol. There are also plans to support zstd as a general compression replacement. So, there appears to be a market for a unified API for registering compression engines. This commit starts the process of establishing one. This commit establishes a base class/interface for defining compression engines and how they will be used. A collection class to hold references to registered compression engines has also been introduced. The built-in zlib, bz2, truncated bz2, and no-op compression engines are registered with a singleton instance of the collection class. The compression engine API will change once consumers are ported to the new API and some common patterns can be simplified at the engine API level. So don't get too attached to the API...

config: mark parser regexes as bytes explicitly r-strings are not transformed into bytes by our source transformer magic.