tests: make test-bdiff.py easier to maintain Add more stdout logging to help navigate the .out file.

perf: unbust perfbdiff --alldata This broke in f84fc6a92817 due to a refactored manifest API. The fix is a bit hacky - perfbdiff doesn't yet support tree manifests for example. But it gets the job done. A test has been added for --alldata so this doesn't happen again.

worker: discard waited pid by anyone who noticed it first This makes sure all waited pids are removed before calling killworkers() even if waitpid()-pids.discard() sequence is interrupted by another SIGCHLD.

worker: kill workers after all zombie processes are reaped Since we now wait child processes in non-blocking way (changed by 7bc25549e084 and e8fb03cfbbde), we don't have to kill them in the middle of the waitpid() loop. This change will help solving a possible race of waitpid()-pids.discard() sequence and another SIGCHLD. waitforworkers() is called by cleanup(), in which case we do killworkers() beforehand so we can remove killworkers() from waitforworkers().

worker: make sure killworkers() never be interrupted by another SIGCHLD killworkers() iterates over pids, which can be updated by SIGCHLD handler. So we should either copy pids or prevent killworkers() from being interrupted by SIGCHLD. I chose the latter as it is simpler and can make pids handling more consistent. This fixes a possible "set changed size during iteration" error at killworkers() before cleanup().

worker: fix missed break on successful waitpid() Follow-up for 5069a8a40b1b.

filterpyflakes: dramatically simplify the entire thing by blacklisting We've only got one kind of pyflakes failure left in our codebase, so it's time to switch over to a blacklist-based checking scheme. I've left in the filtering of two undefined names for now out of paranoia, but those can probably go too.

run-tests: forward Python USER_BASE from site (issue5425) We do this so that any linters installed via pip install --user don't break. See https://docs.python.org/2/library/site.html#site.USER_BASE for a description of what this nonsense is all about. An alternative would be to not set HOME, but that'll cause other problems (see issue2707), or to forward every single path entry from sys.path in PYTHONPATH (which seems sketchy in its own way).

shelve: add missing space in help text The change is trivial and unlikely to have been translated so we update translation files too.

util: improve iterfile so it chooses code path wisely We have performance concerns on "iterfile" as it is 4X slower on normal files. While modern systems have the nice property that reading a "fast" (on-disk) file cannot be interrupted and should be made use of. This patch dumps the related knowledge in comments. And "iterfile" chooses code paths wisely: 1. If it's CPython 3, or PyPY, use the fast path. 2. If fp is a normal file, use the fast path. 3. If fp is not a normal file and CPython version >= 2.7.4, use the same workaround (4x slower) as before. 4. If fp is not a normal file and CPython version < 2.7.4, use another workaround (2x slower but may block longer then necessary) which basically re-invents the buffer + readline logic in Python. This will give us good confidence on both correctness and performance dealing with EINTR in iterfile(fp) for all known supported Python versions.