perf: unbust perfbdiff --alldata This broke in f84fc6a92817 due to a refactored manifest API. The fix is a bit hacky - perfbdiff doesn't yet support tree manifests for example. But it gets the job done. A test has been added for --alldata so this doesn't happen again.

worker: discard waited pid by anyone who noticed it first This makes sure all waited pids are removed before calling killworkers() even if waitpid()-pids.discard() sequence is interrupted by another SIGCHLD.

worker: kill workers after all zombie processes are reaped Since we now wait child processes in non-blocking way (changed by 7bc25549e084 and e8fb03cfbbde), we don't have to kill them in the middle of the waitpid() loop. This change will help solving a possible race of waitpid()-pids.discard() sequence and another SIGCHLD. waitforworkers() is called by cleanup(), in which case we do killworkers() beforehand so we can remove killworkers() from waitforworkers().

worker: make sure killworkers() never be interrupted by another SIGCHLD killworkers() iterates over pids, which can be updated by SIGCHLD handler. So we should either copy pids or prevent killworkers() from being interrupted by SIGCHLD. I chose the latter as it is simpler and can make pids handling more consistent. This fixes a possible "set changed size during iteration" error at killworkers() before cleanup().

worker: fix missed break on successful waitpid() Follow-up for 5069a8a40b1b.

filterpyflakes: dramatically simplify the entire thing by blacklisting We've only got one kind of pyflakes failure left in our codebase, so it's time to switch over to a blacklist-based checking scheme. I've left in the filtering of two undefined names for now out of paranoia, but those can probably go too.

run-tests: forward Python USER_BASE from site (issue5425) We do this so that any linters installed via pip install --user don't break. See https://docs.python.org/2/library/site.html#site.USER_BASE for a description of what this nonsense is all about. An alternative would be to not set HOME, but that'll cause other problems (see issue2707), or to forward every single path entry from sys.path in PYTHONPATH (which seems sketchy in its own way).

util: improve iterfile so it chooses code path wisely We have performance concerns on "iterfile" as it is 4X slower on normal files. While modern systems have the nice property that reading a "fast" (on-disk) file cannot be interrupted and should be made use of. This patch dumps the related knowledge in comments. And "iterfile" chooses code paths wisely: 1. If it's CPython 3, or PyPY, use the fast path. 2. If fp is a normal file, use the fast path. 3. If fp is not a normal file and CPython version >= 2.7.4, use the same workaround (4x slower) as before. 4. If fp is not a normal file and CPython version < 2.7.4, use another workaround (2x slower but may block longer then necessary) which basically re-invents the buffer + readline logic in Python. This will give us good confidence on both correctness and performance dealing with EINTR in iterfile(fp) for all known supported Python versions.

merge with stable

worker: stop using a separate thread waiting for children Now that we have a SIGCHLD hander, and it could get executed when waiting for I/O. It's no longer necessary to have a separated waitpid thread. So just remove it.

worker: add a SIGCHLD handler to collect worker immediately As planned by previous patches, add a SIGCHLD handler to get notifications about worker exits, and deals with worker failure immediately. Note that the SIGCHLD handler gets unregistered before killworkers(), so SIGCHLD won't interrupt "killworkers" - making it harder to send kill signals to waited processes.

worker: make waitforworkers reentrant We are going to use it in the SIGCHLD handler. The handler will be executed in the main thread with the non-blocking version of waitpid, while the waitforworkers thread runs the blocking version. It's possible that one of them collects a worker and makes the other error out (no child to wait). This patch handles these errors: ECHILD is ignored. EINTR needs a retry. The "pids" set is designed to be only modifiable by "waitforworkers". And we only remove items after a successful waitpid. Since a child process can only be "waitpid"-ed once. It's guaranteed that "pids.remove(p)" won't be called with duplicated "p"s. And once a "p" is removed from "pids", that "p" does not need to be killed or waited any more.

worker: change "pids" to a set There is no need to keep any order of the "pids" array. A set is more efficient for the "remove" operation. And the following patch will use that.

worker: allow waitforworkers to be non-blocking This patch adds a boolean flag to waitforworkers and makes it non-blocking if set to True. This is to make it possible that we can reap our workers while keep other unrelated children untouched, after receiving SIGCHLD.

worker: wait worker pid explicitly Before this patch, waitforworkers uses os.wait() to collect child workers, and only wait len(pids) processes. This can have serious issues if other code spawns new processes and does not reap them: 1. worker.py may get wrong exit code and kill innocent workers. 2. worker.py may continue without waiting for all workers to complete. This patch fixes the issue by using waitpid to wait worker pid explicitly. However, this patch introduces a new issue: worker failure may not be handled immediately. The issue will be addressed in next patches.

worker: move killworkers and waitforworkers up We need to use them in the SIGCHLD handler and SIGCHLD handler should be installed before fork.

osutil: implement setprocname to set process title for some platforms This patch adds a simple setprocname method to osutil. The operation is not defined by any standard and is platform-specific, the current implementation tries to cover some major platforms (ex. Linux, OS X, FreeBSD) that is relatively easy to support. Other platforms (Windows [4], other BSDs, ...) can be added in the future. The current implementation supports two methods to change process title: a. setproctitle if available (works in FreeBSD). b. rewrite argv in place (works in Linux [1] and Mac OS X). [2] [3] [1]: Linux has "prctl(PR_SET_NAME, ...)" but 1) it has 16-byte limit, which is too small; 2) it is not quite equivalent to what we want - it changes "/proc/self/comm", not "/proc/self/cmdline" - "comm" change won't show up in "ps" output unless "-o comm" is used. [2]: The implementation does not rewrite the **environ buffer like some other implementations do, just to make the code simpler and safer. However, this also means the buffer size we can rewrite is significantly shorter. If we are really greedy and want the "environ" space, we can change the implementation later. [3]: It requires a CPython private API: Py_GetArgcArgv to get the original argv. Unfortunately Python 3 makes a copy of argv and returns the wchar_t version, so it is not supported for now. (if we really want to, we could count backwards from "char **environ", given known argc and argv, not sure if that's a good idea - probably not) [4]: The feature is aimed to make it easier for forked command server processes to show what they are doing. Since Windows does not support fork(), despite it's a major platform, its support is not added in this patch.

setup: test setproctitle before building osutil We are going to use setproctitle (provided by FreeBSD) if it's available in the next patch. Therefore provide a macro to give some clues to the C pre-processor so it could choose code path wisely.

patch: remove unused git parameter from patch.diffstat() Since 628a4a9e411d the parameter is not used anymore.

perf: add asv benchmarks Airspeed velocity (ASV) is a python framework for benchmarking Python packages over their lifetime. The results are displayed in an interactive web frontend. Add ASV benchmarks for mercurial that use contrib/perf.py extension that could be run against multiple reference repositories. The benchmark suite now includes revsets from contrib/base-revsets.txt with variants, perftags, perfstatus, perfmanifest and perfheads. Installation requires asv>=0.2, python-hglib and virtualenv This is part of PerformanceTrackingSuitePlan https://www.mercurial-scm.org/wiki/PerformanceTrackingSuitePlan

perf: omit copying ui and redirect to ferr if buffer API is in use This allow to get the output of contrib/perf.py commands using the ui.pushbuffer() API.

manifest: change treemanifestctx to construct subtrees from the manifestlog Previously, treemanifestctx would directly construct its subtrees. By making it get the subtrees through manifestlog.get() we consolidate all treemanifestctx creation into manifestlog.get() and therefore extensions that need to wrap manifestctx creation (like narrow-hg) can intercept manifestctxs at that single place. This also means fetching subtrees will take advantage of the manifestlog ctx cache now, which it did not before.

manifest: make revlog verification optional This patches adds an parameter to manifestlog.get() to disable hash checking. This will be used in an upcoming patch to support treemanifestctx reading sub-trees without loading them from the revlog. (This is already supported but does not go through the manifestlog.get() code path)

debugcommands: move debugbuilddag And we drop some now unused imports from commands.py.

debugcommands: introduce standalone module for debug commands commands.py is our largest .py file by nearly 2x. Debug commands live in a world of their own. So let's extract them to their own module. We start with "debugancestor." We currently reuse the commands table with commands.py and have a hack in dispatch.py for loading debugcommands.py. In the future, we could potentially use a separate commands table and avoid the import of debugcommands.py.

convert: migrate to util.iterfile

match: migrate to util.iterfile

store: migrate to util.iterfile

patch: migrate to util.iterfile

worker: migrate to util.iterfile

util: add iterfile to workaround a fileobj.__iter__ issue with EINTR The fileobj.__iter__ implementation in Python 2.7.12 (hg changeset 45d4cea97b04) is buggy: it cannot handle EINTR correctly. In Objects/fileobject.c: size_t Py_UniversalNewlineFread(....) { .... if (!f->f_univ_newline) return fread(buf, 1, n, stream); .... } According to the "fread" man page: If an error occurs, or the end of the file is reached, the return value is a short item count (or zero). Therefore it's possible for "fread" (and "Py_UniversalNewlineFread") to return a positive value while errno is set to EINTR and ferror(stream) changes from zero to non-zero. There are multiple "Py_UniversalNewlineFread": "file_read", "file_readinto", "file_readlines", "readahead". While the first 3 have code to handle the EINTR case, the last one "readahead" doesn't: static int readahead(PyFileObject *f, Py_ssize_t bufsize) { .... chunksize = Py_UniversalNewlineFread( f->f_buf, bufsize, f->f_fp, (PyObject *)f); .... if (chunksize == 0) { if (ferror(f->f_fp)) { PyErr_SetFromErrno(PyExc_IOError); .... } } .... } It means "readahead" could ignore EINTR, if "Py_UniversalNewlineFread" returns a non-zero value. And at the next time "readahead" got executed, if "Py_UniversalNewlineFread" returns 0, "readahead" would raise a Python error without a incorrect errno - could be 0 - thus "IOError: [Errno 0] Error". The only user of "readahead" is "readahead_get_line_skip". The only user of "readahead_get_line_skip" is "file_iternext", aka. "fileobj.__iter__", which should be avoided. There are multiple places where the pattern "for x in fp" is used. This patch adds a "iterfile" method in "util.py" so we can migrate our code from "for x in fp" to "fox x in util.iterfile(fp)".

filterpyflakes: whitelist listcomp aliasing checking The test change is because of how filterpyflakes is organized - a line number changed.

verify: avoid shadowing two variables with a list comprehension The variable names are clearly worse now, but since we're really just transposing key and value I'm not too worried about the clarity loss.

revset: avoid shadowing a variable with a list comprehension

revlog: avoid shadowing several variables using list comprehensions

minirst: avoid shadowing a variable in a list comprehension

hbisect: avoid shadowing a variable in a list comprehension

filemerge: avoid shadowing a variable in a list comprehension

color: avoid shadowing a variable inside a list comprehension

memory: avoid shadowing variables inside a list comprehension

shelve: move shelve-finishing logic to a separate function With future obs-based shelve, finishing shelve will be different from just aborting a transaction and I would like to keep both variants of this functionality in a separate function.

shelve: move unknown files handling to a separate function This change has nothing to do with future obsshelve introduction, it is done just for readability purposes.

shelve: move actual created commit shelving to a separate function Currently, this code does not have any branching, it just bundles a commit and saves a patch file. Later, obsolescence-based shelve will be added, so this code will also create some obsmarkers and will be one of the few places where obsshelve will be different from traditional shelve.

shelve: move 'nothing changed' messaging to a separate function This has nothing to do with the future obsshelve implementation, I just thought that moving this messaging to a separate function will improve shelve code readability.

shelve: move commitfunc creation to a separate function Special commitfuncs are created as closures at least twice in shelve's code and one time special commitfunc is used within another closure. They all serve very specific purposes like temporarily tweak some configuration or enable editor, etc. This is not immediately important to someone reading shelve code, so I think moving this logic to a separate function is a good idea.

shelve: move mutableancestors to not be a closure There's no value in it being a closure and everyone who tries to read the outer function code will be distracted by it. IMO moving it out significantly improves readability, especially given how clear it is what mutableancestors function does from its name.

shelve: move shelve name generation to a separate function This has nothing to do with future obsshelve introduction, done just for readability purposes.

shelve: move possible shelve file extensions to a single place This and a couple of following patches are a preparation to implementing obsolescense-enabled shelve which was discussed on a Sprint. If this refactoring is not done, shelve is going to look even more hackish than now. This particular commit introduces a slight behavior change. Previously, if only .hg/shelve/name.patch file exists, but .hg/name.hg does not, 'hg shelve -d name' would fail saying "shelve not found". Now deletion will only fail if .patch file does not exist (since .patch is used as an indicator of an existing shelve). Other shelve files being absent are skipped silently to accommodate for future introduction of obs-based shelve, which will mean that for some shelves .hg and .patch files exist, while for others .hg and .oshelve.

manifest: delete manifest.manifest class Now that nothing uses the primary manifest class, we can delete it.

localrepo: delete localrepo.manifest Now that nothing uses normal manifests, we can delete localrepo.manifest.

manifest: remove last uses of repo.manifest Now that all the functionality has been moved to manifestlog/manifestrevlog/etc, we can finally change all the uses of repo.manifest to use the new versions. A future diff will then delete repo.manifest. One additional change in this commit is to change repo.manifestlog to be a @storecache property instead of @property. This is required by some uses of repo.manifest require that it be settable (contrib/perf.py and the static http server). We can't do this in a prior change because we can't use @storecache on this until repo.manifest is no longer used anywhere.

manifest: add unionmanifestlog support As part of deprecating manifest, we need to make the union repo support manifestlog.

manifest: add bundlemanifestlog support As part of deprecating manifest.manifest we need to make bundlerepo support manifestlog.

manifest: make manifestlog use it's own cache As we start to make manifestlog the primary manifest source, the dependency on manifest.manifest will cause circular dependency problems. Let's break this dependency by making manifestlog use it's own cache. In a near future patch we will remove the previous manifest cache so we're not duplicating it.

manifest: delete unused dirlog and _newmanifest functions As part of migrating all manifest functionality out of manifest.manifest, let's migrate a couple spots off of manifest.dirlog() to use the revlog specific accessor. Then we can delete manifest.dirlog() and other unused functions.

manifest: move clearcaches to manifestlog This is part of removing all functionality from manifest.manifest so we can delete the class entirely.

manifest: remove usages of manifest.read Now that the two manifestctx implementations have working read() functions, let's remove the existing uses of manifest.read and drop the function.

manifest: remove dependency on manifestrevlog being able to create trees A future patch will be removing the read() function from the manifest class. Since manifestrevlog currently depends on the read function that manifest implements (as a derived class), we need to break the dependency from manifestrevlog to read(). We do this by adding an argument to manifestrevlog.write() which provides it with the ability to read a manifest. This is good in general because it further separates revlog as the storage format from the actual inmemory data structure implementation.

color: show mode warning based on ui.formatted ui.interactive is only for input and ui.formatted is for output.

protocol: drop unused import of zlib Something weird is happening that breaks pyflakes installed via 'pip install --user'. I haven't had a chance to finish debugging this, but this at least fixes the build.

hook: lower inflated use of sys.__stdout__ and __stderr__ They were introduced at 9f76df0edb7d, where sys.stdout could be replaced by sys.stderr. After that, we've changed the way of stdout redirection by afccc64eea73, so we no longer need to reference the original __stdout__ and __stderr__ objects. Let's move away from using __std*__ objects so we can simply wrap sys.std* objects for Python 3 porting.

hook: flush stdout before restoring stderr redirection There was a similar issue to 8b011ededfb2. If an in-process hook writes to stdout, the data may be buffered. In which case, stdout must be flushed before restoring its file descriptor. Otherwise, remaining data would be sent over the ssh wire and corrupts the protocol. Note that this is a different redirection from the one I've just removed.

hook: do not redirect stdout/err/in to ui while running in-process hooks (BC) It was introduced by a59058fd074a to address command-server issues. After that, I've made a complete fix by 69f86b937035, so we don't need to replace sys.stdio objects to protect the IPC channels. This change means we no longer see data written to sys.stdout/err by an in-process hook on command server. I think that's okay because the canonical way is to use ui functions and in-process hooks should respect the Mercurial API. This will help Python 3 porting, where sys.stdout is TextIO but ui.fout is BytesIO.

merge: change modified indicator to be 20 bytes Previously we indicated that the .hgsubstate file was dirty by adding a '+' to the end of its hash in the wctx manifest. This made is complicated to have new manifest implementations that rely on the node length being fixed. In previous patches we added added and modified node placeholders, so let's use those to indicate dirty here as well. It doesn't look like anything ever depended on this '+' (aside from it being different to the parent), so nothing else needed to change here.

dirstate: change added/modified placeholder hash length to 20 bytes Previously the added/modified placeholder hash for manifests generated from the dirstate was a 21byte long string consisting of the p1 file hash plus a single character to indicate an add or a modify. Normal hashes are only 20 bytes long. This makes it complicated to implement more efficient manifest implementations which rely on the hashes being fixed length. Let's change this hash to just be 20 bytes long, and rely on the astronomical improbability of an actual hash being these 20 bytes (just like we rely on no hash every being the nullid). This changes the possible behavior slightly in that the hash for all added/modified entries in the dirstate manifest will now be the same (so simple node comparisons would say they are equal), but we should never be doing simple node comparisons on these nodes even with the old hashes, because they did not accurately represent the content (i.e. two files based off the same p1 file node, with different working copy contents would have the same hash (even with the appended character) in the old scheme too, so we couldn't depend on the hashes period).

dirstate: change placeholder hash length to 20 bytes Previously the new-node placeholder hash for manifests generated from the dirstate was a 21byte long string of "!" characters. Normal hashes are only 20 bytes long. This makes it complicated to implement more efficient manifest implementations which rely on the hashes being fixed length. Let's change this hash to just be 20 bytes long, and rely on the astronomical improbability of an actual hash being 20 "!" bytes in a row (just like we rely on no hash ever being the nullid). A future diff will do this for added and modified dirstate markers as well, so we're putting the new newnodeid in node.py so there's a common place for these placeholders.

util: remove compressorobj API from compression engines All callers have been replaced with "compressstream." It is quite low-level and redundant with "compressstream." So eliminate it.

hgweb: use compression engine API for zlib compression More low-level compression code elimination because we now have nice APIs. This patch also demonstrates why we needed and implemented the "level" option on the "compressstream" API.

bundle2: use compressstream compression engine API Compression engines now have an API for compressing a stream of chunks. Switch to it and make low-level compression code disappear.

util: add a stream compression API to compression engines It is a common pattern throughout the code to perform compression on an iterator of chunks, yielding an iterator of compressed chunks. Let's formalize that as part of the compression engine API. The zlib and bzip2 implementations allow an optional "level" option to control the compression level. The default values are the same as what the Python modules use. This option will be used in subsequent patches.

util: remove decompressors dict (API) All in-tree consumers are now using the compengines registrar. Extensions should switch to it as well.

changegroup: use compression engines API The new API doesn't have the equivalence for None and 'UN' so we introduce code to use 'UN' explicitly.

bundle2: use compression engines API to obtain decompressor Like the recent change for the compressor side, this too is relatively straightforward. We now store a compression engine on the instance instead of a low-level decompressor. Again, this will allow us to easily transition to different compression engine APIs when they are implemented.

util: remove compressors dict (API) We no longer have any in-tree consumers of this object. Use util.compengines instead.

bundle2: use new compression engine API for compression Now that we have a new API to define compression engines, let's put it to use! The new code stores a reference to the compression engine instead of a low-level compressor object. This will allow us to more easily transition to different APIs on the compression engine interface once we implement them. As part of this, we change the registration in bundletypes to use 'UN' instead of None. Previously, util.compressors had the no-op compressor registered under both the 'UN' and None keys. Since we're switching to a new API, I don't see the point in carrying this dual registration forward.

util: create new abstraction for compression engines Currently, util.py has "compressors" and "decompressors" dicts mapping compression algorithms to callables returning objects that perform well-defined operations. In addition, revlog.py has code for calling into a compressor or decompressor explicitly. And, there is code in the wire protocol for performing zlib compression. The 3rd party lz4revlog extension has demonstrated the utility of supporting alternative compression formats for revlog storage. But it stops short of supporting lz4 for bundles and the wire protocol. There are also plans to support zstd as a general compression replacement. So, there appears to be a market for a unified API for registering compression engines. This commit starts the process of establishing one. This commit establishes a base class/interface for defining compression engines and how they will be used. A collection class to hold references to registered compression engines has also been introduced. The built-in zlib, bz2, truncated bz2, and no-op compression engines are registered with a singleton instance of the collection class. The compression engine API will change once consumers are ported to the new API and some common patterns can be simplified at the engine API level. So don't get too attached to the API...

config: mark parser regexes as bytes explicitly r-strings are not transformed into bytes by our source transformer magic.

ui: explicitly open config files in binary mode This has been working mostly accidentally, but now it works explicitly.

help: fix double word usage "most" was used twice. (I fixed a grammar error before timeless spotted it!)

setup: move cffi stuff to mercurial/cffi This patch moves all setup*cffi stuff to mercurial/cffi to make the root directory cleaner. The idea was from mpm [1]: > It seems like we could have a fair amount of cffi definitions, and > cluttering the root directory (or mercurial/) with them is probably not > a great long-term solution. We could probably add a cffi/ directory > under mercurial/ to parallel pure/. [1]: https://www.mercurial-scm.org/pipermail/mercurial-devel/2016-July/086442.html

manifest: remove manifest.add and add memmfctx.write This removes one more dependency on the manifest class by moving the write functionality onto the memmanifestctx classes and changing the one consumer to use the new API. By moving the write path to a manifestctx, we now give the individual manifests control over how they're read and serialized. This will be useful in developing new manifest formats and storage systems.

context: add manifestctx property on changectx This allows us to access the manifestctx for a given commit. This will be used in a later patch to be able to copy the manifestctx when we want to make a new commit.

manifest: add copy to mfctx classes This adds copy functionality to the manifestctx classes. This will be used in an upcoming diff to copy a manifestctx during commit so we can modify the manifest before committing.

manifest: introduce memmanifestctx and memtreemanifestctx This introduces two new classes to represent in-memory manifest instances. Similar to memchangectx, this lets us prepare a manifest in memory, then in a future patch we will add the apis that can commit this in memory structure.

manifestctx: add _revlog() function The `self._repo.manifestlog._revlog` code is getting copy and pasted a lot in manifestctx. Let's make it a function so it can be reused. This will make future patches cleaner too.

manifest: remove manifest.find As part of removing dependencies on manifest, this drops the find function and fixes up the two existing callers to use the equivalent apis on manifestctx.

changegroup: remove remaining uses of repo.manifest The remaining uses of repo.manifest in the changegroup module are treating the manifest exclusively as a revlog, so let's replace them with instances of the revlog directly. This is part of dropping all dependencies on repo.manifest in favor of repo.manifestlog.

treemanifest: fix a "treeinmem" case f2c5b9d48b29 (manifest: make treemanifestctx store the repo, 2016-10-18) broke most tests when run with treeinmem=True. The treeinmem mode can not be enabled by the user, so this did not break anything in practice, but it's useful to have it working for testing the treemanifest code.

perf: support measuring bdiff for all changeset related data The --all argument changes the behavior of `perfbdiff` to pull in fulltext revision pairs for all changes related to a changeset. The p1 and p2 manifests will be bdiffed against current. Every file that changed between p1 and current will have its file revisions loaded and bdiffed. This mode of operation effectively measured the bdiff time required for `hg commit`.

perf: support bdiffing multiple revisions in a single revlog This is useful for testing bdiff performance on several revision pairs at a time.

perf: prepare to handle multiple pairs in perfbdiff Before, we only supported benchmarking a single pair of texts with bdiff. We want to enable feeding larger corpora into this benchmark. So rewrite the code to support that.

py3: document why os.fsencode() can be used to get back bytes argv And a possible Windows issue. I'm sad we have to do such ugly hack, but that's the unicode on Python 3.

py3: update test-check-py3-compat.t output 4b1af1c867fa (scmutil: move util.termwidth()) changed where the import fails.

spelling: fixes of non-dictionary words

manifest: add __nonzero__ method This adds a __nonzero__ method to manifestdict. This isn't strictly necessary in the vanilla Mercurial implementation, since Python will handle nonzero checks by using __len__, but having it implemented here makes it easier for alternative implementations to implement __nonzero__ and have them be plug-n-play with the normal implementation.

py3: have bytes version of sys.argv sys.argv returns unicodes on Python 3. We need a bytes version for us. There was also a python bug/feature request which wanted then to implement one. They rejected and it is quoted in one of the comments that we can use fsencode() to get a bytes version of sys.argv. Though not sure about its correctness. Link to the comment: http://bugs.python.org/issue8776#msg217416 After this patch we will have pycompat.sysargv which will return us bytes version of sys.argv. If this patch goes in, i will like to make transformer rewrite sys.argv with pycompat.argv because there are lot of occurences.