largefiles: reset "lfstatus" attribute to previous value in "finally" We were resetting it to False, which means nesting of these overrides would be incorrect. Differential Revision: https://phab.mercurial-scm.org/D7137

largefiles: add context manager for setting/clearing "lfstatus" attribute Differential Revision: https://phab.mercurial-scm.org/D7136

py3: adjust expected traceback in test-hook.t In Python 3, traceback.format_exception() displays the chain of exceptions so we get extra results from our grep. Also, ModuleNotFoundError is raised instead of ImportError from Python 3.6.

tests: use non-reverse grep in traceback in test-hook.t This will prepare for updating test output for Python 3.

py3: add Python 3 exception output to test-hook.t This is similar to, e.g., 3e9c6cef949b.

py3: ajust abort message in test-hook.t

rust-cpython: prepare for writing tests that require libpython What I wanted is to disable the "cpython/extension-module<ver>" feature while building tests executable, but that seems not doable. Instead, this patch adds new features dedicated for tests. The make rule is extracted so that we can easily run cargo tests. Added a minimal test of ref-sharing as an example. More tests will follow.

rust-cpython: make inner functions and structs of ref_sharing private Most of these methods were public because they had to be accessible from macro-generated functions. Some "unsafe" can be removed since we can guarantee the data consistency across non-public operations.

rust-cpython: keep Python<'a> token in PyRefMut This just clarifies that the GIL is obtained while PyRefMut is dereferenced, so there's no need of extra acquire_gil() to drop the reference.

rust-cpython: require GIL to borrow immutable reference from PySharedRefCell Since the inner value may be leaked, we probably need GIL to guarantee that there's no data race. inner(py).borrow() is replaced with inner_shared(py).borrow(), which basically means any PySharedRefCell data should be accessed through PySharedRef wrapper.

rust-cpython: make PyLeakedRef operations relatively safe This patch encapsulates the access to the leaked reference to make most leaked-ref operations safe. The only exception is leaked_ref.map(). I couldn't figure out how to allow arbitrary map operation safely over an unsafe static reference. See the docstring and inline comment for details. Now leak_immutable() can be safely implemented as the PyLeakedRef owns its inner data.

rust-cpython: put leaked reference in PyLeakedRef The next patch will make PyLeakedRef manage the lifetime of the underlying object. leak_handle.data.take() will be removed soon.

run-tests: make code coverage work on Python 3 This code path was obviously not tested on Python 3 because it blew up in several places due to str/bytes mismatch. For internal code, we normalize paths to bytes. For code calling into `coverage`, we normalize paths to str, which is what `coverage` seems to expect. After this, `run-tests.py -H` works on Python 3! Differential Revision: https://phab.mercurial-scm.org/D7133

py3: don't index into bytes in phabricator's _tokenize() `phabread`ing a stack using `hg phabread :D1234` under py3 will currently die with a KeyError because it will index into `b':D1234'` and return `58` instead of `b':'` as a token. Differential Revision: https://phab.mercurial-scm.org/D7129

rust-dirstate-status: use fast-path even with fsmonitor and sparse extensions When I initially ran the tests on my series, there were test failures related to those extensions. Now that the initial series has landed, I felt like going back to those issues because people with performance issues will often want to use fsmonitor and sparse. Either because of flaky tests or because the series has changed so much, I can't seem to reproduce these issues... let's widen the scope of the fast-path. Differential Revision: https://phab.mercurial-scm.org/D7128

rust-dirstatemap: remove additional lookups in dirstatemap We're copying this shortcut from the Python implementation, pretty standard for this codebase. Differential Revision: https://phab.mercurial-scm.org/D7117

tests: avoid chmod on windows in hgrc tests Follow up on changeset d201a637c971 introducing this test, which fails on Windows.

py3: fix sorting of obsolete markers during push This fixes similar errors as in 01e8eefd9434: TypeError: '<' not supported between instances of 'NoneType' and 'tuple'

exchange: extract a function to sort obsolete markers Follows up on changeset 01e8eefd9434, several other occurrences of sorted() need to be fixed. The _sortedmarkers() handles sorting obsmarkers with a None value as last element on Python 3.

py3: encode underlying error message during parse error of %include

convert: convert os.devnull to bytes before trying to join it with other bytes Together with the previous commit relating to emailparser this gets test-convert-tla.t passing under py3. Differential Revision: https://phab.mercurial-scm.org/D7062

rust-dirstate-status: add call to rust-fast path for `dirstate.status` The reasoning behind this patch is explained in the first patch of the series. Differential Revision: https://phab.mercurial-scm.org/D7060

rust-dirstate-status: rust-cpython bindings for `dirstate.status` The ref-sharing mechanism has improved, but its ergonomics still left a bit to be desired, as expected. Differential Revision: https://phab.mercurial-scm.org/D7059

rust-refsharing: add missing lifetime parameter in ref_sharing Differential Revision: https://phab.mercurial-scm.org/D7110

rust-dirstate-status: add first Rust implementation of `dirstate.status` Note: This patch also added the rayon crate as a Cargo dependency. It will help us immensely in making Rust code parallel and easy to maintain. It is a stable, well-known, and supported crate maintained by people on the Rust team. The current `dirstate.status` method has grown over the years through bug reports and new features to the point where it got too big and too complex. This series does not yet improve the logic, but adds a Rust fast-path to speed up certain cases. Tested on mozilla-try-2019-02-18 with zstd compression: - `hg diff` on an empty working copy: - c: 1.64(+-)0.04s - rust+c before this change: 2.84(+-)0.1s - rust+c: 849(+-)40ms - `hg commit` when creating a file: - c: 5.960s - rust+c before this change: 5.828s - rust+c: 4.668s - `hg commit` when updating a file: - c: 4.866s - rust+c before this change: 4.371s - rust+c: 3.855s - `hg status -mard` - c: 1.82(+-)0.04s - rust+c before this change: 2.64(+-)0.1s - rust+c: 896(+-)30ms The numbers are clear: the current Rust `dirstatemap` implementation is super slow, its performance needs to be addressed. This will be done in a future series, immediately after this one, with the goal of getting Rust to be at least to the speed of the Python + C implementation in all cases before the 5.2 freeze. At worse, we gate dirstatemap to only be used in those cases. Cases where the fast-path is not executed: - for commands that need ignore support (`status`, for example) - if subrepos are found (should not be hard to add, but winter is coming) - any other matcher than an `alwaysmatcher`, like patterns, etc. - with extensions like `sparse` and `fsmonitor` The next step after this is to rethink the logic to be closer to Jane Street's Valentin Gatien-Baron's Rust fast-path which does a lot less work when possible. Differential Revision: https://phab.mercurial-scm.org/D7058

share: unmark --relative as EXPERIMENTAL Differential Revision: https://phab.mercurial-scm.org/D7001

rust-cpython: removed now useless py_set() conversion In rust-cpython 0.3.0, HashSets implement the appropriate ToPythonObject, we can therefore get rid of this hacky conversion. There still remains an inefficiency in `MissingAncestors.bases()`: we have to clone, because `to_py_object()` requires full ownership. However: - the only use case outside of unit tests used to be from `setdiscovery.partialdiscovery` which is now fully implemented in Rust. - it's not worse than what `py_set()` used to do Differential Revision: https://phab.mercurial-scm.org/D7120

tests: stabilize test-eol-update on Windows These appeared in the recent eol fix series. I'm not sure about the identity mismatch warning- it seems like a general timing issue, because it happened on the first run but not when run with `--loop`. The buildbot also flagged this. Differential Revision: https://phab.mercurial-scm.org/D7112

tests: opt into python3 when testing virtualenv installation Otherwise it complains that Mercurial requires '~=2.7'. Since the existing linux py3 test doesn't flag this, I'm assuming that virtualenv simply isn't installed on that system. Differential Revision: https://phab.mercurial-scm.org/D7111

grep: put --diff in the first line of usage Differential Revision: https://phab.mercurial-scm.org/D7003