worker: fix missed break on successful waitpid() Follow-up for 5069a8a40b1b.

filterpyflakes: dramatically simplify the entire thing by blacklisting We've only got one kind of pyflakes failure left in our codebase, so it's time to switch over to a blacklist-based checking scheme. I've left in the filtering of two undefined names for now out of paranoia, but those can probably go too.

run-tests: forward Python USER_BASE from site (issue5425) We do this so that any linters installed via pip install --user don't break. See https://docs.python.org/2/library/site.html#site.USER_BASE for a description of what this nonsense is all about. An alternative would be to not set HOME, but that'll cause other problems (see issue2707), or to forward every single path entry from sys.path in PYTHONPATH (which seems sketchy in its own way).

util: improve iterfile so it chooses code path wisely We have performance concerns on "iterfile" as it is 4X slower on normal files. While modern systems have the nice property that reading a "fast" (on-disk) file cannot be interrupted and should be made use of. This patch dumps the related knowledge in comments. And "iterfile" chooses code paths wisely: 1. If it's CPython 3, or PyPY, use the fast path. 2. If fp is a normal file, use the fast path. 3. If fp is not a normal file and CPython version >= 2.7.4, use the same workaround (4x slower) as before. 4. If fp is not a normal file and CPython version < 2.7.4, use another workaround (2x slower but may block longer then necessary) which basically re-invents the buffer + readline logic in Python. This will give us good confidence on both correctness and performance dealing with EINTR in iterfile(fp) for all known supported Python versions.

merge with stable

worker: stop using a separate thread waiting for children Now that we have a SIGCHLD hander, and it could get executed when waiting for I/O. It's no longer necessary to have a separated waitpid thread. So just remove it.

worker: add a SIGCHLD handler to collect worker immediately As planned by previous patches, add a SIGCHLD handler to get notifications about worker exits, and deals with worker failure immediately. Note that the SIGCHLD handler gets unregistered before killworkers(), so SIGCHLD won't interrupt "killworkers" - making it harder to send kill signals to waited processes.

worker: make waitforworkers reentrant We are going to use it in the SIGCHLD handler. The handler will be executed in the main thread with the non-blocking version of waitpid, while the waitforworkers thread runs the blocking version. It's possible that one of them collects a worker and makes the other error out (no child to wait). This patch handles these errors: ECHILD is ignored. EINTR needs a retry. The "pids" set is designed to be only modifiable by "waitforworkers". And we only remove items after a successful waitpid. Since a child process can only be "waitpid"-ed once. It's guaranteed that "pids.remove(p)" won't be called with duplicated "p"s. And once a "p" is removed from "pids", that "p" does not need to be killed or waited any more.

worker: change "pids" to a set There is no need to keep any order of the "pids" array. A set is more efficient for the "remove" operation. And the following patch will use that.

worker: allow waitforworkers to be non-blocking This patch adds a boolean flag to waitforworkers and makes it non-blocking if set to True. This is to make it possible that we can reap our workers while keep other unrelated children untouched, after receiving SIGCHLD.

worker: wait worker pid explicitly Before this patch, waitforworkers uses os.wait() to collect child workers, and only wait len(pids) processes. This can have serious issues if other code spawns new processes and does not reap them: 1. worker.py may get wrong exit code and kill innocent workers. 2. worker.py may continue without waiting for all workers to complete. This patch fixes the issue by using waitpid to wait worker pid explicitly. However, this patch introduces a new issue: worker failure may not be handled immediately. The issue will be addressed in next patches.

worker: move killworkers and waitforworkers up We need to use them in the SIGCHLD handler and SIGCHLD handler should be installed before fork.

osutil: implement setprocname to set process title for some platforms This patch adds a simple setprocname method to osutil. The operation is not defined by any standard and is platform-specific, the current implementation tries to cover some major platforms (ex. Linux, OS X, FreeBSD) that is relatively easy to support. Other platforms (Windows [4], other BSDs, ...) can be added in the future. The current implementation supports two methods to change process title: a. setproctitle if available (works in FreeBSD). b. rewrite argv in place (works in Linux [1] and Mac OS X). [2] [3] [1]: Linux has "prctl(PR_SET_NAME, ...)" but 1) it has 16-byte limit, which is too small; 2) it is not quite equivalent to what we want - it changes "/proc/self/comm", not "/proc/self/cmdline" - "comm" change won't show up in "ps" output unless "-o comm" is used. [2]: The implementation does not rewrite the **environ buffer like some other implementations do, just to make the code simpler and safer. However, this also means the buffer size we can rewrite is significantly shorter. If we are really greedy and want the "environ" space, we can change the implementation later. [3]: It requires a CPython private API: Py_GetArgcArgv to get the original argv. Unfortunately Python 3 makes a copy of argv and returns the wchar_t version, so it is not supported for now. (if we really want to, we could count backwards from "char **environ", given known argc and argv, not sure if that's a good idea - probably not) [4]: The feature is aimed to make it easier for forked command server processes to show what they are doing. Since Windows does not support fork(), despite it's a major platform, its support is not added in this patch.

setup: test setproctitle before building osutil We are going to use setproctitle (provided by FreeBSD) if it's available in the next patch. Therefore provide a macro to give some clues to the C pre-processor so it could choose code path wisely.

patch: remove unused git parameter from patch.diffstat() Since 628a4a9e411d the parameter is not used anymore.

perf: add asv benchmarks Airspeed velocity (ASV) is a python framework for benchmarking Python packages over their lifetime. The results are displayed in an interactive web frontend. Add ASV benchmarks for mercurial that use contrib/perf.py extension that could be run against multiple reference repositories. The benchmark suite now includes revsets from contrib/base-revsets.txt with variants, perftags, perfstatus, perfmanifest and perfheads. Installation requires asv>=0.2, python-hglib and virtualenv This is part of PerformanceTrackingSuitePlan https://www.mercurial-scm.org/wiki/PerformanceTrackingSuitePlan

perf: omit copying ui and redirect to ferr if buffer API is in use This allow to get the output of contrib/perf.py commands using the ui.pushbuffer() API.

manifest: change treemanifestctx to construct subtrees from the manifestlog Previously, treemanifestctx would directly construct its subtrees. By making it get the subtrees through manifestlog.get() we consolidate all treemanifestctx creation into manifestlog.get() and therefore extensions that need to wrap manifestctx creation (like narrow-hg) can intercept manifestctxs at that single place. This also means fetching subtrees will take advantage of the manifestlog ctx cache now, which it did not before.

manifest: make revlog verification optional This patches adds an parameter to manifestlog.get() to disable hash checking. This will be used in an upcoming patch to support treemanifestctx reading sub-trees without loading them from the revlog. (This is already supported but does not go through the manifestlog.get() code path)

debugcommands: move debugbuilddag And we drop some now unused imports from commands.py.

debugcommands: introduce standalone module for debug commands commands.py is our largest .py file by nearly 2x. Debug commands live in a world of their own. So let's extract them to their own module. We start with "debugancestor." We currently reuse the commands table with commands.py and have a hack in dispatch.py for loading debugcommands.py. In the future, we could potentially use a separate commands table and avoid the import of debugcommands.py.

convert: migrate to util.iterfile

match: migrate to util.iterfile

store: migrate to util.iterfile

patch: migrate to util.iterfile

worker: migrate to util.iterfile

util: add iterfile to workaround a fileobj.__iter__ issue with EINTR The fileobj.__iter__ implementation in Python 2.7.12 (hg changeset 45d4cea97b04) is buggy: it cannot handle EINTR correctly. In Objects/fileobject.c: size_t Py_UniversalNewlineFread(....) { .... if (!f->f_univ_newline) return fread(buf, 1, n, stream); .... } According to the "fread" man page: If an error occurs, or the end of the file is reached, the return value is a short item count (or zero). Therefore it's possible for "fread" (and "Py_UniversalNewlineFread") to return a positive value while errno is set to EINTR and ferror(stream) changes from zero to non-zero. There are multiple "Py_UniversalNewlineFread": "file_read", "file_readinto", "file_readlines", "readahead". While the first 3 have code to handle the EINTR case, the last one "readahead" doesn't: static int readahead(PyFileObject *f, Py_ssize_t bufsize) { .... chunksize = Py_UniversalNewlineFread( f->f_buf, bufsize, f->f_fp, (PyObject *)f); .... if (chunksize == 0) { if (ferror(f->f_fp)) { PyErr_SetFromErrno(PyExc_IOError); .... } } .... } It means "readahead" could ignore EINTR, if "Py_UniversalNewlineFread" returns a non-zero value. And at the next time "readahead" got executed, if "Py_UniversalNewlineFread" returns 0, "readahead" would raise a Python error without a incorrect errno - could be 0 - thus "IOError: [Errno 0] Error". The only user of "readahead" is "readahead_get_line_skip". The only user of "readahead_get_line_skip" is "file_iternext", aka. "fileobj.__iter__", which should be avoided. There are multiple places where the pattern "for x in fp" is used. This patch adds a "iterfile" method in "util.py" so we can migrate our code from "for x in fp" to "fox x in util.iterfile(fp)".

filterpyflakes: whitelist listcomp aliasing checking The test change is because of how filterpyflakes is organized - a line number changed.

verify: avoid shadowing two variables with a list comprehension The variable names are clearly worse now, but since we're really just transposing key and value I'm not too worried about the clarity loss.

revset: avoid shadowing a variable with a list comprehension

revlog: avoid shadowing several variables using list comprehensions

minirst: avoid shadowing a variable in a list comprehension