py3: add test-dirstate-race2.t to list of passing tests This test was added new recently. The py3 buildbot found that it passes, so let's add it to the list of passing tests. Differential Revision: https://phab.mercurial-scm.org/D6530

strip: during merge allow strip only when -f is used This ensures to abort strip to `hg strip` when we have a merge in progress and allow it only when a `--force` flag is used. Differential Revision: https://phab.mercurial-scm.org/D6529

deltas: set estimated compression upper bound to "3x" instead of "10x" In pratice, we very rarely observer compression better than "3x" on manifest deltas. Having a more aggressive estimate significantly helps our pathological use case on a private repository. Here are a comparison of timings using different upper bound. Estimated compression | ø | ×10 | ×5 | ×3 | timing | 14.11 | 2.61 | 1.96 | 1.53 | We also tested the impact of this series on an array of public repositories. This shown no impact in either size nor timing. Full data set below for those interested. Size ---- Regarding size, not significant impact have been noticed on neither public nor private repositories. Here are the number we gathered on public repositories: zlib/upperbound | no | 10x | 5x | 3x mercurial | 5 875 730 | 5 875 730 | 5 875 730 | 5 875 730 pypy | 27 782 913 | 27 782 913 | 27 782 913 | 27 782 913 netbeans | 159 161 207 | 159 161 207 | 159 161 207 | 159 959 879 (+0.5%) mozilla-central | 323 841 642 | 323 841 642 | 323 841 642 | 319 867 519 (-2.5%) mozilla-try | 746 649 123 | 746 649 123 | 746 649 123 | 741 155 568 (-0.7%) private-repo | 1 485 287 294 | 1 485 287 294 | 1 485 287 294 | 1 409 248 382 (-5.1%) zstd/upperbound | no | 10x | 5x | 3x mercurial | 5 895 206 | 5 895 206 | 5 895 206 | 5 895 206 pypy | 28 689 230 | 28 689 230 | 28 689 230 | 28 689 230 netbeans | 157 636 387 | 157 636 387 | 157 636 387 | 159 692 678 (+1.3%) mozilla-central | 317 650 281 | 317 650 281 | 317 650 281 | 319 613 603 (+0.6%) mozilla-try | 737 555 275 | 737 555 275 | 737 555 275 | 738 079 473 (+0.1%) private-repo | 1 352 362 982 | 1 352 362 982 | 1 346 961 880 | 1 361 327 384 (+0.7%) Speed ------ Timing gathered using `hg perfrevlogwrite -m`. Value are in seconds. mercurial zlib | no | 10x | 5x | 3x | total | 65.551783 | 65.388887 | 65.260658 | 65.321199 | max | 0.034544 | 0.034571 | 0.034659 | 0.034521 | 99.99% | 0.034544 | 0.034571 | 0.034659 | 0.034521 | zstd | no | 10x | 5x | 3x | total | 49.118449 | 49.054062 | 48.753588 | 48.740230 | max | 0.009338 | 0.009239 | 0.009202 | 0.009178 | 99.99% | 0.007618 | 0.007639 | 0.007626 | 0.007621 | pypy zlib | no | 10x | 5x | 3x | total | 560.865984 | 558.983817 | 559.083815 | 559.349152 | max | 0.219614 | 0.215922 | 0.218112 | 0.218107 | 99.99% | 0.219614 | 0.215922 | 0.218112 | 0.218107 | zstd | no | 10x | 5x | 3x | total | 349.393280 | 347.395819 | 347.185407 | 345.643985 | max | 0.084143 | 0.083536 | 0.081834 | 0.082178 | 99.99% | 0.039445 | 0.039639 | 0.039612 | 0.039175 | netbeans zlib | no | 10x | 5x | 3x | total | 33103.327727 | 33314.932260 | 33211.745233 | 33345.891778 | max | 2.666852 | 2.672059 | 2.662453 | 2.662936 | 99.99% | 2.058772 | 2.070429 | 2.069569 | 2.064653 | zstd | no | 10x | 5x | 3x | total | 20112.102708 | 20095.879719 | 20083.390300 | 20123.221859 | max | 2.063482 | 2.062851 | 2.065229 | 2.060147 | 99.99% | 1.146647 | 1.143794 | 1.142933 | 1.146529 | mozilla zlib | no | 10x | 5x | 3x | total | 41374.102138 | 41418.816773 | 41381.956370 | 41334.280732 | max | 3.383474 | 3.387400 | 3.405711 | 3.387316 | 99.99% | 1.006755 | 1.005954 | 1.007700 | 1.007373 | zstd | no | 10x | 5x | 3x | total | 24689.691520 | 24643.939662 | 24664.630027 | 24664.512714 | max | 1.460822 | 1.449640 | 1.439747 | 1.465304 | 99.99% | 0.527111 | 0.527377 | 0.527807 | 0.527226 |

deltas: skip if projected compressed size is bigger than previous snapshot If we have a delta, we check constraints against a lower bound estimate of the resulting compressed delta. We then checks this projected size against the `size(snapshotⁿ) > size(snapshotⁿ⁺¹)` constraint. This allows to exclude potential base candidates before doing any expensive computation. This only apply to the intermediate-snapshot case since this constraint only apply to them. For some pathological cases of a private repository this step provide a further performance boost (timing from `hg perfrevlogwrite`): before: 3.010646 seconds after: 2.609307 seconds

deltas: skip if projected compressed size does not match text size constraint If we have a delta, we check constraints against a lower bound estimate of the resulting compressed delta. We then checks this projected size against the ½ⁿ size constraints. This allows to exclude potential base candidates before doing any expensive computation. This only apply to the intermediate-snapshot case since this constraint only apply to them. For some pathological cases of a private repository this step provide a further performance boost (timing from `hg perfrevlogwrite`): before: 3.145906 seconds after: 3.010646 seconds

deltas: accept and skip None return for delta info They are some extra computation that will shortcut the delta compression if the delta seems hopeless, returning None.

delta: move some delta chain related computation earlier in deltainfo They are some more optimization change that will make use of this in the function. So we retrieve the data earlier.

deltas: skip if projected delta size is bigger than previous snapshot Before computing any delta, we get a basic estimation of the delta size we can expect and the resulted compressed value. We then checks this projected size against the `size(snapshotⁿ) > size(snapshotⁿ⁺¹)` constraint. This allows to exclude potential base candidates before doing any expensive computation. This only apply to the intermediate-snapshot case since this constraint only apply to them. For some pathological cases of a private repository this step provide a significant performance boost (timing from `hg perfrevlogwrite`): before: 14.115908 seconds after: 3.145906 seconds

deltas: skip if projected delta size does not match text size constraint Before computing any delta, we get a basic estimation of the delta size we can expect and the resulted compressed value. We then checks this projected size against the ½ⁿ size constraints. This allows to exclude potential base candidates before doing any expensive computation. This only apply to the intermediate-snapshot case since this constraint only apply to them. In practice we only perform this new checks for the manifestlog. Manifest log combine two property: it is likely to have delta chain issue and its diffing/compression is fairly predictable. The initial author of this changeset is Valentin Gatien-Baron providing the initial idea and initial testing, Pierre-Yves David later consolidated the code in the right location and run more extensive testing.

revlog: add the option to track the expected compression upper bound There are various optimization we can do if we can estimate the size of delta before actually spending CPU compressing them. So we add a attributed dedicated to tracking that. We only use it on Manifest because (1) it structure is quite stable across all Mercurial repository so its compression ratio is fairly universal. This is the revlog with most extreme delta (cf the sparse-revlog optimization). This will be put to use in later changesets. Right now the compression upper bound is set to 10. This is a fairly conservative value (observed value is more around 3), but I prefer to be safe while introducing the optimization principles. We can tune the optimization threshold later.

perf: clarify some of the custom behavior of `perfrevlogwrite` This reduce the chance of developers being surprised by special cases.

perf: fix perfrevlogwrite --count documentation The help text was copy pasted from the previous option.

rust: switched to 'cargo rustc' in setup.py This is more flexible in the passing of additional flags, also what setuptools_rust does, giving less uncertainty about non-Linux platforms.

rust-cpython: fix build for MacOSX MacOSX needs special link flags. Quoting the README of rust-cpython: create a `.cargo/config` with the following content: ``` [target.x86_64-apple-darwin] rustflags = [ "-C", "link-arg=-undefined", "-C", "link-arg=dynamic_lookup", ] ``` This is tested with Python 2.7 (Anaconda install) and Python 3 (Homebrew install)

rust-cpython: management of shared libray suffix Before this changeset, the shared library objects suffixes were both (rustc output and Python input) hardcoded to '.so', which is wrong for Python3 and non Linux targets.

merge: fix race that could cause wrong size in dirstate The problem is that hg merge/update/etc work the following way: 1. figure out what files to update 2. apply the update to disk 3. apply the update to in-memory dirstate 4. write dirstate where step3 looks at the filesystem and assumes it sees the result of step2. If a file is changed between step2 and step3, step3 will record incorrect information in the dirstate. I avoid this by passing the size step3 needs directly from step2, for the common path (not implemented for change/delete conflicts for instance). I didn't fix the same race for the exec bit for now, because it's less likely to be problematic and I had trouble due to the fact that the dirstate stores the permissions differently from the manifest (st_mode vs '' 'l' 'x'), in combination with tests that pretend that symlinks are not supported. However, I moved the lstat from step3 to step2, which should tighten the race window markedly, both for the exec bit and for the mtime. Differential Revision: https://phab.mercurial-scm.org/D6475

worker: support parallelization of functions with return values Currently worker supports running functions that return a progress iterator. Generalize it to handle function that return a progress iterator then a return value. It's unused in this commit, but will be used in the next one. Differential Revision: https://phab.mercurial-scm.org/D6515

tests: show how the dirstate can end up containing wrong information which can result in bad status output. Concretely, this seems to be easily triggered by having a build system watching the filesystem for changes, and rebuilding files that are both tracked and generated while an update is happening. Differential Revision: https://phab.mercurial-scm.org/D6474

rust: new rust options in setup.py The --rust global option turns on usage (and by default compilation) of the rust-cpython based mercurial.rustext. Similarly to what's previously done for zstd, there is a --no-rust option for the build_ext subcommand in order not to build mercurial.rustext, allowing for an OS distribution to prebuild it. The HGWITHRUSTEXT environment variable is still honored, and has the same effect as before, but now it works mostly by making the --rust global option defaulting to True, with some special cases for the direct-ffi case (see more about that below) Coincidentally, the --rust flag can also be passed from the make commands, like actually all global options, in the PURE variable make local PURE=--rust This feels inappropriate, though, and we should follow up with a proper make variable for that case. Although the direct-ffi bindings aren't directly useful any more, we keep them at this stage because - they provide a short prototyping path for experiments in which a C extension module has to call into a Rust extension. The proper way of doing that would be to use capsules, and it's best to wait for our pull request onto rust-cpython for that: https://github.com/dgrunwald/rust-cpython/pull/169 - Build support for capsules defined in Rust will probably need to reuse some of what's currently in use for direct-ffi.

rust: using policy.importrust from Python callers This commit converts all current Python callers of mercurial.rustext to the new policy.importrust system. After this point, going through policy.importrust or policy.importmod (in some more distant future) is mandatory for callers of Rust code outside of Python tests. We felt it to be appropriate to keep Rust-specific tests run inconditionally if the Rust extensions are present.

rust: module policy with importrust We introduce two rust+c module policies and a new `policy.importrust()` that makes use of them. This simple approach provides runtime switching of implementations, which is crucial for the performance measurements such as those Octobus does with ASV. It can also be useful for bug analysis. It also has the advantage of making conditionals in Rust callers more uniform, in particular abstracting over specifics like `demandimport` At this point, the build stays unchanged, with the rust-cpython based `rustext` module being built if HGWITHRUSTEXT=cpython. More transparency for the callers, i.e., just using `policy.importmod` would be a much longer term and riskier effort for the following reasons: 1. It would require to define common module boundaries for the three or four cases (pure, c, rust+ext, cffi) and that is premature with the Rust extension currently under heavy development in areas that are outside the scope of the C extensions. 2. It would imply internal API changes that are not currently wished, as the case of ancestors demonstrates. 3. The lack of data or property-like attributes (tp_member and tp_getset) in current `rust-cpython` makes it impossible to achieve direct transparent replacement of pure Python classes by Rust extension code, meaning that the caller sometimes has to be able to make adjustments or provide additional wrapping.

help: add help entry for internals.mergestate This patch adds an entry for `internals.mergestate` as suggested by @marmoute. Most of the help text is taken from `merge.mergestate`. Differential Revision: https://phab.mercurial-scm.org/D6448 Differential Revision: https://phab.mercurial-scm.org/D6528

phabricator: use parents.set to always set dependencies Now that Mercurial's Phabricator instance has been updated to a version that supports the parents.set transaction on revision.edit we can use that to set dependency relationships in patch stacks instead of abusing the summary. This has the advantage that we can use it on every `phabsend` so commit reordering is picked up without spamming changes like abusing the summary would, and using parents.set will clear previous parents unlike parents.add. Differential Revision: https://phab.mercurial-scm.org/D6514

help: remove repeated word in 'hg help rebase' Specifically, the second 'with' in 'with which to merge with'. Differential Revision: https://phab.mercurial-scm.org/D6483