run-tests: fix permission to clean up unreadable directories I found many hgtests.* directories left in $TMPDIR, which couldn't be deleted because test-ssh-repoerror.t created some directories with a-rx mode.

context: reimplement memfilectx.cmp() If I added a sanity check to basefilectx, test-context.py exploded. This patch copies the naive implementation from overlayworkingfilectx.

rust-cpython: build and support for Python3 Defined Cargo features for Python3, making them overall simpler to use, hooked them in build and made mercurial.rustext importable. This is tested with Python 3.6.7. Differential Revision: https://phab.mercurial-scm.org/D5446

py3: convert popen() command arguments in hgclient to str on Windows This fixes test-commandserver.t and test-keyword.t, which was previously complaining TypeError("a bytes-like object is required, not 'str'")

py3: byteify tests/pullext.py # skip-blame for b'' prepending

py3: byteify the fakeversion extension in test-http-bad-server.t # skip-blame for b'' prefixing

py3: spawn all python instances with legacy stdio enabled on Windows This fixes 6 more tests. See ef7119cd4965.

setup: avoid attempting to invoke the system-wide hg.exe on Windows On Windows, the executable in the current directory gets priority over anything in $PATH (both for cmd.exe and MSYS). That means, the former code was launching the local hg.exe instead of the system-wide one, if it was previously built. If that failed, it then fell back to the local hg code, but run through python.exe. I'm not sure what it is about ef7119cd4965, but that started throwing up a messagebox that python37.dll couldn't be loaded. (And indeed, python37 is not in $PATH by default.) Invoking the local hg via the current python avoids that.

delta: ignore base whose chains already don't match expectations If we know the existing chain does not match our criteria, there is no point to build a delta to append. This is especially useful when dealing with a full text much smaller than its parent. In that case, the parent chain is probably already too large. example affected manifest write before: 1.421005s after: 0.815520s (-42%)

delta: exclude base candidate much smaller than the target If a revision's full text is that much bigger than a base candidate full text, we no longer consider that candidate. This solves a pathological case we encountered on a very specify repository. It contains a long series of changesets with a very small manifest (one file) co-existing with others changesets using a very large manifest. Without this filtering, we ended up considering a large number of tiny full snapshots as a potential base. It resulted in very large delta (the size of the full text) and mercurial spending 99% of its time compressing these deltas. The timing of a commit moved from about 400s to about 10s (still slow, but not ridiculously slow).

perfrevflogwrite: clear revlog cache between each write We want to measure write time from a cold cache (similar to commit). So we need to clear the cache to prevent computation from rev N-1 to interfere with rev N.

py3: enable legacy stdio mode in exewrapper This drops the test failure count from 166 to 117. The failures were typically in the form of `hg serve -d` spawning but crashing immediately, leaving clients with "bad http status" lines, connection refusals, and so forth. The underlying message on the server side was either "OSError: [WinError 6] The handle is invalid" or "OSError: [WinError 1] Incorrect function". Additionally, no output was rendered if the pager was activated. Thanks to Yuya for diagnosing the problem. The failure count drops to 107 when PYTHONLEGACYWINDOWSSTDIO=1 is defined in the environment. These failures seem to revolve around the dummyssh server process, and dumbhttp.py. So I'll probably add that to the test runner. One small regression here (only in py3) is that if hg.exe is already built, a messagebox appears when building it again saying that python37.dll can't be loaded. Python3 isn't in PATH by default, and setup.py tries running bare `hg` first. But MSYS prepends '.' to PATH, so it runs the local hg, but can't find the library. It falls back to the python used to invoke setup.py, so ultimately it works. I'm not sure if it's better to strip '.' from PATH or just skip right to `sys.executable hg` on Windows. Also, something seems to be wrong with run-tests._usecorrectpython(). I accidentially left off the 'PYTHON="py -3"' when building (thus making py2 stuff), and yet managed to invoke run-tests.py with "py -3". (And that only had 67 failures.)

run-tests: alias hg to hg.exe on Windows To enable legacy stdio mode on Windows, hg.exe needs to be updated. But before that, we actually have to use it when running the tests. I *think* what was happening before was when MSYS invoked `hg`, it looked at the shbang line and ran python.exe found there. The test harness must be updating $PATH to include the python used to launch it, and therefore it ran py3. As a side note, this also fixed `py -3 run-tests.py` (without --local), which complained about the space in the shbang line before this. This should also help in WSL, because the explicit '.exe' is needed to invoke a Windows app instead of invoking the Linux app. I have no idea why this change capitalizes Lib in the tests, as it was previously lowercase for both py2 and py3.

py3: conditionalize the python version in test-install.t

tests: followup on network related errors on Debian 9 First, update test-clonebundles.t which was failing since the introduction of the `$EADDRNOTAVAIL$` common pattern. Also update two tests from 8695fbe17f7c with a more compact condition line. Differential Revision: https://phab.mercurial-scm.org/D5445

py3: unimplement RevlogError.__str__() On Python 2, str(exc) would crash if __str__() returned a unicode containing non-ASCII characters.

py3: use forcebytestr() to stringify hook exception This fixes test-narrow-exchange.t.

color: fix a documentation typo

py3: quote $PYTHON in test-patchbomb.t for Windows I couldn't get the quoting right in the environment variable, so now it's a function.

rust-cpython: testing the bindings from Python This is easier and more convincing than doing the same tests from a Rust tests module. Differential Revision: https://phab.mercurial-scm.org/D5437

rust-cpython: build via HGWITHRUSTEXT=cpython The existing behaviour, building the direct ffi bindings if HGIWTHRUSTEXT is just set is unchanged, but if HGWITHRUSTEXT is cpython, then the cpython bindings (aka mercurial/rustext.so) are built. Differential Revision: https://phab.mercurial-scm.org/D5436

rust: better treatment of cargo/rustc errors Differential Revision: https://phab.mercurial-scm.org/D5435

rust-cpython: start cpython crate bindings This changeset introduces the hg-cpython crate, that compiles as a shared library holding a whole Python package (mercurial.rustext), with only the empty 'ancestor' submodule for now. Such bindings will be easier and safer to develop and maintain that those of `hg-direct-ffi`. They don't involve C code, only unsafe Rust that's mostly isolated within the cpython crate. The long-term goal would be to import the provided modules, such as rustext.ancestor with mercurial.policy.importmod, same as we already do with cext modules. Differential Revision: https://phab.mercurial-scm.org/D5434

rust-cpython: exclude hgcli from workspace hgcli uses a specific rust-cpython commit by indygreg, of which a PR has been derived which is not merged nor released yet. But we can't use several versions of the sys-python2.7 crate in a single workspace: it makes for a build error. Since hgcli does not at the time being need anything from hg-core, whereas the upcoming hg-cpython will. So for now we're moving hgcli aside, hoping we could base all of them on the same version of rust-cpython again in the future. Differential Revision: https://phab.mercurial-scm.org/D5433

sparse-revlog: protect C code against delta chain including nullrev For unclear reasons, some repositories include nullrev (-1). Re-computing delta for such repo remove nullrev from all chain, so some older versions have been creating them. This currently raise an IndexError with the new C code doing chain slicing as it expect all item to be positive. Both python and C code for reading delta chain preserve nullrev, and the Python code for chain slicing handle the case fine. So we take the safe route and make the new C code works fine in that case.

sparse-revlog: handle nullrev in index_get_length The more generic index_get method handle nullrev fine, we apply the same logic here.

sparse-revlog: handle nullrev in index_get_start The more generic index_get method handle nullrev fine, we apply the same logic here.

revlog: introduce a constant for nullrev in `revlog.c` The value is important enough to be explicitly tracked.

rust: translation of missingancestors This is as direct as possible a translation of the ancestor.missingancestors Python class in pure Rust. The goal for this changeset is to make it easy to compare with the Python version. We also add to Python tests the cases that helped us develop and debug this implementation. Some possible optimizations are marked along the way as TODO comments Differential Revision: https://phab.mercurial-scm.org/D5416

contrib: provide a small script that draw performance plot We have been using this script to look into the result of various runs of the `hg perfrevlogwrite` command. It seems useful enough to be shared more widely.

delta: filter nullrev out first When picking a potential candidate, we filter them on various criteria. The "different from nullrev" criteria is very fast to compute and we should process it first.

perf: report more of the higher range in perfrevlogwrite Since the delta chain length is limited to 1000 revisions, we get a new snapshot about every 1000 revisions. If we assume that the snapshot will be most of the slowest revision, the current display (99% and max) are not very precise in their area. We now include more information about this space in the default report.

help: present boolean arguments as "--[no-]foo" This should make it much more discoverable (we document it in `hg help flags`, but most users don't think to look there). Note that flags that default to None (and not False) will not get this new presentation. We can change the defaults to False later for flags where it makes sense (probably almost all boolean flags). Differential Revision: https://phab.mercurial-scm.org/D5432

help: show "default: off" for boolean flags that default to off Differential Revision: https://phab.mercurial-scm.org/D5431

help: use "default: on" instead of "default: True" "True" feels like a Python thing and not something that users should see. Differential Revision: https://phab.mercurial-scm.org/D5430

check-commit: disallow capitalization only right after topic It should be okay to write "topic: change 'default: True' to 'default: on'". Differential Revision: https://phab.mercurial-scm.org/D5429

absorb: don't prompt to apply changes when there are none to apply Differential Revision: https://phab.mercurial-scm.org/D5428

py3: add a b'' prefix in tests/test-remotefilelog-pull-noshallow.t This makes the test pass on Python 3. # skip-blame because just b'' prefix. Differential Revision: https://phab.mercurial-scm.org/D5427

error: implement __str__ on RevlogError to fix some output defects on Py3 We open-code encoding.unimethod here to avoid cycles, and do a local import of encoding when someone str()s a RevlogError. It's not my favorite solution, but it gets the job done. Differential Revision: https://phab.mercurial-scm.org/D5426

perf: add two more missing b prefixes for Python 3 # skip-blame just two more b prefixes Differential Revision: https://phab.mercurial-scm.org/D5425

match: fix doctest to use bytes instead of str Fixes doctests on Python 3. Differential Revision: https://phab.mercurial-scm.org/D5423

py3: use pycompat.bytechr() instead of chr() in test-fastannotate-revmap.py This makes the test pass on Python 3. Differential Revision: https://phab.mercurial-scm.org/D5422

py3: add a missing b'' prefix in contrib/perf.py # skip-blame because just b'' prefixes This fixes test-contrib-perf.t on Python 3 which started failing. Differential Revision: https://phab.mercurial-scm.org/D5421

py3: use '%d' for integers instead of '%s' This should fix test-rebase-inmemory.t which started failing on Python 3 after recent changes. Differential Revision: https://phab.mercurial-scm.org/D5420

py3: whitelist another passing test caught by buildbot Differential Revision: https://phab.mercurial-scm.org/D5419

py3: fix bad escapes of sub() replacement pattern in test-template-basic.t Python 3.7 starts complaining about it. We have to double the backslash or '\x5c' to get around.

perfdiscovery: benching findcommonheads() This works between the local repo and any peer given by its path, and should be useful for further work on discovery Differential Revision: https://phab.mercurial-scm.org/D5418

windows: ensure pure posixfile fd doesn't escape by entering context manager There are tests in test-revlog-mmapindex.t and test-rebase-mq-skip.t that are fixed by this, but we usually don't use --pure on Windows. For whatever reason, the remaining --pure failures are various errors like $ENOTDIR$ and "Access is denied" have a trailing '.'.