copies: simplify chain loop This is a mostly gratuitous change to make the code more direct. Differential Revision: https://phab.mercurial-scm.org/D7131

sidedatacopies: only fetch information once for merge Before this change, merge would result in reading the data from revlog twice. With this change, we keep the information in memory until we encounter the other parent. When looking at pypy, I see about 1/3 of the changesets with copy information being merge. Not doing duplicated fetch for them provide a significant speedup. revision: large amount; added files: large amount; rename small amount; c3b14617fbd7 9ba6ab77fd29 before: ! wall 0.767042 comb 0.760000 user 0.750000 sys 0.010000 (median of 11) after: ! wall 0.671162 comb 0.670000 user 0.650000 sys 0.020000 (median of 13) revision: large amount; added files: small amount; rename small amount; c3b14617fbd7 f650a9b140d2 before: ! wall 1.170169 comb 1.170000 user 1.130000 sys 0.040000 (median of 10) after: ! wall 1.030596 comb 1.040000 user 1.010000 sys 0.030000 (median of 10) revision: large amount; added files: large amount; rename large amount; 08ea3258278e d9fa043f30c0 before: ! wall 0.209846 comb 0.200000 user 0.200000 sys 0.000000 (median of 46) after: ! wall 0.170981 comb 0.170000 user 0.170000 sys 0.000000 (median of 56) revision: small amount; added files: large amount; rename large amount; df6f7a526b60 a83dc6a2d56f before: ! wall 0.013248 comb 0.010000 user 0.010000 sys 0.000000 (median of 223) after: ! wall 0.013295 comb 0.020000 user 0.020000 sys 0.000000 (median of 222) revision: small amount; added files: large amount; rename small amount; 4aa4e1f8e19a 169138063d63 before: ! wall 0.001672 comb 0.000000 user 0.000000 sys 0.000000 (median of 1000) after: ! wall 0.001666 comb 0.000000 user 0.000000 sys 0.000000 (median of 1000) revision: small amount; added files: small amount; rename small amount; 4bc173b045a6 964879152e2e before: ! wall 0.000119 comb 0.000000 user 0.000000 sys 0.000000 (median of 8010) after: ! wall 0.000119 comb 0.000000 user 0.000000 sys 0.000000 (median of 8007) revision: medium amount; added files: large amount; rename medium amount; c95f1ced15f2 2c68e87c3efe before: ! wall 0.168599 comb 0.160000 user 0.160000 sys 0.000000 (median of 58) after: ! wall 0.133316 comb 0.140000 user 0.140000 sys 0.000000 (median of 73) revision: medium amount; added files: medium amount; rename small amount; d343da0c55a8 d7746d32bf9d before: ! wall 0.036052 comb 0.030000 user 0.030000 sys 0.000000 (median of 100) after: ! wall 0.032558 comb 0.030000 user 0.030000 sys 0.000000 (median of 100) Differential Revision: https://phab.mercurial-scm.org/D7127

copies: move from a copy on branchpoint to a copy on write approach Before this changes, any branch points results in a copy of the dictionary containing the copy information. This can be very costly for branchy history with few rename information. Instead, we take a "copy on write" approach. Copying the input data only when we are about to update them. In practice we where already doing the copying in half of these case (because `_chain` makes a copy), so we don't add a significant cost here even in the linear case. However the speed up in branchy case is very significant. Here are some timing on the pypy repository. revision: large amount; added files: large amount; rename small amount; c3b14617fbd7 9ba6ab77fd29 before: ! wall 1.399863 comb 1.400000 user 1.370000 sys 0.030000 (median of 10) after: ! wall 0.766453 comb 0.770000 user 0.750000 sys 0.020000 (median of 11) revision: large amount; added files: small amount; rename small amount; c3b14617fbd7 f650a9b140d2 before: ! wall 1.876748 comb 1.890000 user 1.870000 sys 0.020000 (median of 10) after: ! wall 1.167223 comb 1.170000 user 1.150000 sys 0.020000 (median of 10) revision: large amount; added files: large amount; rename large amount; 08ea3258278e d9fa043f30c0 before: ! wall 0.242457 comb 0.240000 user 0.240000 sys 0.000000 (median of 39) after: ! wall 0.211476 comb 0.210000 user 0.210000 sys 0.000000 (median of 45) revision: small amount; added files: large amount; rename large amount; df6f7a526b60 a83dc6a2d56f before: ! wall 0.013193 comb 0.020000 user 0.020000 sys 0.000000 (median of 224) after: ! wall 0.013290 comb 0.010000 user 0.010000 sys 0.000000 (median of 222) revision: small amount; added files: large amount; rename small amount; 4aa4e1f8e19a 169138063d63 before: ! wall 0.001673 comb 0.000000 user 0.000000 sys 0.000000 (median of 1000) after: ! wall 0.001677 comb 0.000000 user 0.000000 sys 0.000000 (median of 1000) revision: small amount; added files: small amount; rename small amount; 4bc173b045a6 964879152e2e before: ! wall 0.000119 comb 0.000000 user 0.000000 sys 0.000000 (median of 8023) after: ! wall 0.000119 comb 0.000000 user 0.000000 sys 0.000000 (median of 7997) revision: medium amount; added files: large amount; rename medium amount; c95f1ced15f2 2c68e87c3efe before: ! wall 0.201898 comb 0.210000 user 0.200000 sys 0.010000 (median of 48) after: ! wall 0.167415 comb 0.170000 user 0.160000 sys 0.010000 (median of 58) revision: medium amount; added files: medium amount; rename small amount; d343da0c55a8 d7746d32bf9d before: ! wall 0.036820 comb 0.040000 user 0.040000 sys 0.000000 (median of 100) after: ! wall 0.035797 comb 0.040000 user 0.040000 sys 0.000000 (median of 100) The extra cost in the linear case can be reclaimed later with some extra logic. Differential Revision: https://phab.mercurial-scm.org/D7124

copies: compute the exact set of revision to walk This change make the code clearer by removing the revision queue. It comes without very noticeable performance impact. However the simpler code will be easier to update in later changesets. revision: large amount; added files: large amount; rename small amount; c3b14617fbd7 9ba6ab77fd29 before: ! wall 1.430082 comb 1.430000 user 1.390000 sys 0.040000 (median of 10) after: ! wall 1.405192 comb 1.410000 user 1.390000 sys 0.020000 (median of 10) revision: large amount; added files: small amount; rename small amount; c3b14617fbd7 f650a9b140d2 before: ! wall 1.971366 comb 1.970000 user 1.950000 sys 0.020000 (median of 10) after: ! wall 1.892541 comb 1.890000 user 1.870000 sys 0.020000 (median of 10) revision: large amount; added files: large amount; rename large amount; 08ea3258278e d9fa043f30c0 before: ! wall 0.252594 comb 0.250000 user 0.240000 sys 0.010000 (median of 38) after: ! wall 0.240075 comb 0.240000 user 0.240000 sys 0.000000 (median of 40) revision: small amount; added files: large amount; rename large amount; df6f7a526b60 a83dc6a2d56f before: ! wall 0.013100 comb 0.010000 user 0.010000 sys 0.000000 (median of 226) after: ! wall 0.013247 comb 0.010000 user 0.010000 sys 0.000000 (median of 223) revision: small amount; added files: large amount; rename small amount; 4aa4e1f8e19a 169138063d63 before: ! wall 0.001633 comb 0.000000 user 0.000000 sys 0.000000 (median of 1000) after: ! wall 0.001670 comb 0.000000 user 0.000000 sys 0.000000 (median of 1000) revision: small amount; added files: small amount; rename small amount; 4bc173b045a6 964879152e2e before: ! wall 0.000078 comb 0.000000 user 0.000000 sys 0.000000 (median of 11984) after: ! wall 0.000119 comb 0.000000 user 0.000000 sys 0.000000 (median of 7982) revision: medium amount; added files: large amount; rename medium amount; c95f1ced15f2 2c68e87c3efe before: ! wall 0.207093 comb 0.210000 user 0.210000 sys 0.000000 (median of 47) after: ! wall 0.201551 comb 0.200000 user 0.200000 sys 0.000000 (median of 48) revision: medium amount; added files: medium amount; rename small amount; d343da0c55a8 d7746d32bf9d before: ! wall 0.038462 comb 0.040000 user 0.040000 sys 0.000000 (median of 100) after: ! wall 0.036578 comb 0.030000 user 0.030000 sys 0.000000 (median of 100) Differential Revision: https://phab.mercurial-scm.org/D7076

copies: add an explicit test using multiple roots We explicitly add a test checking for copies of connected revision but involving a different roots. This make sure the copies coming purely from the new roots does not interfere with the merge. Differential Revision: https://phab.mercurial-scm.org/D7075

sidedatacopies: fast path data fetching if revision has no sidedata When using the side data mode, we know their won't be any copy information sidedata. Skipping revision restoration give an important speed boost. In the future, there will be other user of sidedata, reducing the efficiency of this. We should consider adding a dedicated flag in revlog V2 to preserve this optimisation. The current situation is good enough for now. revision: large amount; added files: large amount; rename small amount; c3b14617fbd7 9ba6ab77fd29 before: ! wall 2.401569 comb 2.400000 user 2.390000 sys 0.010000 (median of 10) after: ! wall 1.429294 comb 1.430000 user 1.410000 sys 0.020000 (median of 10) revision: large amount; added files: small amount; rename small amount; c3b14617fbd7 f650a9b140d2 before: ! wall 3.519140 comb 3.520000 user 3.470000 sys 0.050000 (median of 10) after: ! wall 1.963332 comb 1.960000 user 1.960000 sys 0.000000 (median of 10) revision: large amount; added files: large amount; rename large amount; 08ea3258278e d9fa043f30c0 before: ! wall 0.593880 comb 0.600000 user 0.590000 sys 0.010000 (median of 15) after: ! wall 0.251679 comb 0.250000 user 0.250000 sys 0.000000 (median of 38) revision: small amount; added files: large amount; rename large amount; df6f7a526b60 a83dc6a2d56f before: ! wall 0.013414 comb 0.020000 user 0.020000 sys 0.000000 (median of 220) after: ! wall 0.013222 comb 0.020000 user 0.020000 sys 0.000000 (median of 223) revision: small amount; added files: large amount; rename small amount; 4aa4e1f8e19a 169138063d63 before: ! wall 0.002711 comb 0.000000 user 0.000000 sys 0.000000 (median of 1000) after: ! wall 0.001631 comb 0.000000 user 0.000000 sys 0.000000 (median of 1000) revision: small amount; added files: small amount; rename small amount; 4bc173b045a6 964879152e2e before: ! wall 0.000077 comb 0.000000 user 0.000000 sys 0.000000 (median of 12208) after: ! wall 0.000078 comb 0.000000 user 0.000000 sys 0.000000 (median of 12012) revision: medium amount; added files: large amount; rename medium amount; c95f1ced15f2 2c68e87c3efe before: ! wall 0.410067 comb 0.410000 user 0.410000 sys 0.000000 (median of 23) after: ! wall 0.207786 comb 0.200000 user 0.200000 sys 0.000000 (median of 46) revision: medium amount; added files: medium amount; rename small amount; d343da0c55a8 d7746d32bf9d before: ! wall 0.097004 comb 0.090000 user 0.090000 sys 0.000000 (median of 100) after: ! wall 0.038495 comb 0.030000 user 0.030000 sys 0.000000 (median of 100) Differential Revision: https://phab.mercurial-scm.org/D7074

largefiles: use context manager for setting "lfstatus" on subrepos too It's safe since we now reset to the previous value. Differential Revision: https://phab.mercurial-scm.org/D7138

largefiles: reset "lfstatus" attribute to previous value in "finally" We were resetting it to False, which means nesting of these overrides would be incorrect. Differential Revision: https://phab.mercurial-scm.org/D7137

largefiles: add context manager for setting/clearing "lfstatus" attribute Differential Revision: https://phab.mercurial-scm.org/D7136

py3: adjust expected traceback in test-hook.t In Python 3, traceback.format_exception() displays the chain of exceptions so we get extra results from our grep. Also, ModuleNotFoundError is raised instead of ImportError from Python 3.6.

tests: use non-reverse grep in traceback in test-hook.t This will prepare for updating test output for Python 3.

py3: add Python 3 exception output to test-hook.t This is similar to, e.g., 3e9c6cef949b.

py3: ajust abort message in test-hook.t

rust-cpython: prepare for writing tests that require libpython What I wanted is to disable the "cpython/extension-module<ver>" feature while building tests executable, but that seems not doable. Instead, this patch adds new features dedicated for tests. The make rule is extracted so that we can easily run cargo tests. Added a minimal test of ref-sharing as an example. More tests will follow.

rust-cpython: make inner functions and structs of ref_sharing private Most of these methods were public because they had to be accessible from macro-generated functions. Some "unsafe" can be removed since we can guarantee the data consistency across non-public operations.

rust-cpython: keep Python<'a> token in PyRefMut This just clarifies that the GIL is obtained while PyRefMut is dereferenced, so there's no need of extra acquire_gil() to drop the reference.

rust-cpython: require GIL to borrow immutable reference from PySharedRefCell Since the inner value may be leaked, we probably need GIL to guarantee that there's no data race. inner(py).borrow() is replaced with inner_shared(py).borrow(), which basically means any PySharedRefCell data should be accessed through PySharedRef wrapper.

rust-cpython: make PyLeakedRef operations relatively safe This patch encapsulates the access to the leaked reference to make most leaked-ref operations safe. The only exception is leaked_ref.map(). I couldn't figure out how to allow arbitrary map operation safely over an unsafe static reference. See the docstring and inline comment for details. Now leak_immutable() can be safely implemented as the PyLeakedRef owns its inner data.

rust-cpython: put leaked reference in PyLeakedRef The next patch will make PyLeakedRef manage the lifetime of the underlying object. leak_handle.data.take() will be removed soon.

run-tests: make code coverage work on Python 3 This code path was obviously not tested on Python 3 because it blew up in several places due to str/bytes mismatch. For internal code, we normalize paths to bytes. For code calling into `coverage`, we normalize paths to str, which is what `coverage` seems to expect. After this, `run-tests.py -H` works on Python 3! Differential Revision: https://phab.mercurial-scm.org/D7133

py3: don't index into bytes in phabricator's _tokenize() `phabread`ing a stack using `hg phabread :D1234` under py3 will currently die with a KeyError because it will index into `b':D1234'` and return `58` instead of `b':'` as a token. Differential Revision: https://phab.mercurial-scm.org/D7129

rust-dirstate-status: use fast-path even with fsmonitor and sparse extensions When I initially ran the tests on my series, there were test failures related to those extensions. Now that the initial series has landed, I felt like going back to those issues because people with performance issues will often want to use fsmonitor and sparse. Either because of flaky tests or because the series has changed so much, I can't seem to reproduce these issues... let's widen the scope of the fast-path. Differential Revision: https://phab.mercurial-scm.org/D7128

rust-dirstatemap: remove additional lookups in dirstatemap We're copying this shortcut from the Python implementation, pretty standard for this codebase. Differential Revision: https://phab.mercurial-scm.org/D7117

tests: avoid chmod on windows in hgrc tests Follow up on changeset d201a637c971 introducing this test, which fails on Windows.

py3: fix sorting of obsolete markers during push This fixes similar errors as in 01e8eefd9434: TypeError: '<' not supported between instances of 'NoneType' and 'tuple'

exchange: extract a function to sort obsolete markers Follows up on changeset 01e8eefd9434, several other occurrences of sorted() need to be fixed. The _sortedmarkers() handles sorting obsmarkers with a None value as last element on Python 3.

py3: encode underlying error message during parse error of %include

convert: convert os.devnull to bytes before trying to join it with other bytes Together with the previous commit relating to emailparser this gets test-convert-tla.t passing under py3. Differential Revision: https://phab.mercurial-scm.org/D7062

rust-dirstate-status: add call to rust-fast path for `dirstate.status` The reasoning behind this patch is explained in the first patch of the series. Differential Revision: https://phab.mercurial-scm.org/D7060

rust-dirstate-status: rust-cpython bindings for `dirstate.status` The ref-sharing mechanism has improved, but its ergonomics still left a bit to be desired, as expected. Differential Revision: https://phab.mercurial-scm.org/D7059