scmutil: speed up relativization of paths when it's a no-op Running commands from the root is commmon, in particular for automation. Running `hg files > /tmp/a` from the root of mozilla-central on linux: before: real 0m1,510s user 0m1,387s sys 0m0,090s after: real 0m1,266s user 0m1,165s sys 0m0,073s (there are 280k paths, so this was costing ~1us per path somehow) Differential Revision: https://phab.mercurial-scm.org/D8585

context: implement mergestate() method This will let us have the mergestate storage be controlled by the context. In particular, for working contexts we should use the existing mergestate, but for overlay contexts it's inappropriate to drop files in .hg/merge. Differential Revision: https://phab.mercurial-scm.org/D8551

mergestate: split out merge state handling code from main merge module There's already some pretty reasonable encapsulation here, but I want to make the mergestate storage a property of the context so memctx instances can do a reasonable thing. This is the first step in a reshuffle to make that easier. Differential Revision: https://phab.mercurial-scm.org/D8550

tests: add coverage for repo.changelog.children() in the git extension Differential Revision: https://phab.mercurial-scm.org/D8548

tests: add coverage for repo.changelog.findmissing() in test-git-interop.t This at least does a basic test of the method. It's not super-complete, but it's better than the nothing we'd otherwise have. Differential Revision: https://phab.mercurial-scm.org/D8547

relnotes: add API change note per request in D8502 Differential Revision: https://phab.mercurial-scm.org/D8549

merge with stable

grep: grep the working copy faster `hg grep qqqq` in the mercurial repo: before: 0,859s after: 0,233s `hg grep somethingwithnomatch` in mozilla-central: before: 51s after: 19s This is probably also a tiny bug fix, because the code was looking up a node for filename `pfn` on a filelog for filename `fn`, which are most of the time the same filename, but don't have to be. Ignoring performance and the bug fix, the code should have the same behavior. Differential Revision: https://phab.mercurial-scm.org/D8545

grep: stop computing information for --diff when unnecessary This is one reason why `hg grep pattern` essentially does `hg cat -r . 'set:**'` inside. There is no speed improvement in this commit, because the rest of the code still greps data from filelog instead of working copy when possible. Differential Revision: https://phab.mercurial-scm.org/D8544

grep: don't go in an infinite loop when given empty regex Differential Revision: https://phab.mercurial-scm.org/D8543

grep: improve test coverage Differential Revision: https://phab.mercurial-scm.org/D8542

phabricator: avoid passing None to pycompat.fsdecode Differential Revision: https://phab.mercurial-scm.org/D8525

setup: stop asking cargo to spam It prints this kind of stuff by default: Compiling regex v1.3.6 Compiling rand v0.7.3 Compiling crossbeam-deque v0.7.3 which is way better than we ask for before this change, which is screen after screen of compilation commands and irrelevant warnings in crates we use. Differential Revision: https://phab.mercurial-scm.org/D8537

git: implement some changelog methods Differential Revision: https://phab.mercurial-scm.org/D8540

git: avoid looking-up parents for the null commit Differential Revision: https://phab.mercurial-scm.org/D8541

git: fix probable missing return Differential Revision: https://phab.mercurial-scm.org/D8539

rust: fix warning about unnecessary mut If there's a reason to use mut (like compability with older compilers), then we should stick `#[allow(unused_mut)]` on the declaration. Differential Revision: https://phab.mercurial-scm.org/D8538

upgrade: support upgrade and downgrade from persistent nodemap The requirements is now recognised and dealt with and the associated files properly handled. The persistent nodemap should be ready for usage in the field now. Differential Revision: https://phab.mercurial-scm.org/D8431

status: also support for `traversedir` callback in the Rust fast-path Repeating the performance numbers from the `hg-core` change: Running `hg clean/purge` on Netbeans' repo (100k files): ``` | No-op | 30% unknown -------------------------- Rust | 1.0s | 1.67s C | 2.0s | 2.87s ``` Differential Revision: https://phab.mercurial-scm.org/D8520

rust-hg-cpython: update status bridge with the new `traversedir` support Differential Revision: https://phab.mercurial-scm.org/D8519

rust-status: collect traversed directories if required Some commands (`hg purge` notably) register the `traversedir` callback on their matcher to run said callback every time a directory is traversed. This is the first of three patches, further broadening Rust support for status. Unfortunately, there is no way around collecting a full `Vec` (or any other owned datastructure, like a radix tree) and pushing it back up the Python layer since keeping the Python callback in a closure would mean giving up multithreading because of the GIL, which is obviously unacceptable. Performance is still a lot better than the Python+C path. Running `hg clean/purge` on Netbeans' repo (100k files): ``` | No-op | 30% unknown -------------------------- Rust | 1.0s | 1.67s C | 2.0s | 2.87s ``` Differential Revision: https://phab.mercurial-scm.org/D8518

rust-status: don't dispatch unknown file when traversing if not listing unknowns This usually isn't a (functional) problem since we ignore the unknown files anyway, but when specifically using `hg purge`, unknown files were iterated over regardless of the option being true. This is both more correct and more efficient. Differential Revision: https://phab.mercurial-scm.org/D8517

status: update comment to reflect the more recent situation This is a gratuitous cleanup. Differential Revision: https://phab.mercurial-scm.org/D8516

hooks: provide access to transaction changes for internal hooks External hooks are skipped here as the environment often has a size limit in the low MBs and that can easily be reached by larger transactions. Differential Revision: https://phab.mercurial-scm.org/D8490

rust-regex: add test for verbatim regex syntax This makes sure it's not modified. Differential Revision: https://phab.mercurial-scm.org/D8508

rust-regex: prevent nonsensical `.*.*` pattern from happening Differential Revision: https://phab.mercurial-scm.org/D8507

rust-regex: fix issues with regex anchoring and performance It turns out that the way I tried to work around `regex`'s behavior difference with `re2` and Python's `re` was 1) buggy and 2) much more complicated than needed. In a few words: `regex` adds `.*` on either side of patterns when no start or end anchor is present. My previous workaround put `^` or `$` for every pattern, which is wrong even without the other 2 bugs on top of it. Using `^(?:<patterns>)` right at the end of the `regex` path fixes the issue. I've opened an issue to get a build option instead: https://github.com/rust-lang/regex/issues/675 Differential Revision: https://phab.mercurial-scm.org/D8506

diff: avoid going from contexts to nodes and back This will allow us to pass in-memory contexts that may not have a valid node to the diffing logic. Differential Revision: https://phab.mercurial-scm.org/D8503

setup: raise minimum Python version to 2.7.4 On older Python versions, Mercurial is not really usable because of https://bugs.python.org/issue10211. Recently someone reported a crash on the mailing list when running Mercurial on Python 2.7.3. There was consensus that fixing compatibility for a Python version more than 7 years old is not worth it. So, instead of making Mercurial crash with an obscure exception, this patch raises the minimum Python version to 2.7.4.

fsmonitor: coerce `clock` variable to byte-string (issue6321) Callers of `fsmonitor.state.setlastclock` pass their arguments wrapped in `pycompat.sysbytes` to ensure the value is a `bytes` on Python 3. However in `fsmonitor.poststatus.__call__`, if the return value of `getlastclock()` is `None`, we use the value of `fsmonitor.poststatus._startclock` instead, which is not converted to a byte string in the same manner. This commit converts the value of `startclock` to a byte string using `pycompat.sysbytes` in the constructor for `poststatus`, to avoid the "`str` + `bytes`" error from issue 6321. Differential Revision: https://phab.mercurial-scm.org/D8573

py3: change default priority and length used for sorting hooks to be compatible with python 3 The call to `sorted(hooks.values())` can on line 213 of hooks.py can raise when using python 3. For instance, when hooks.values is `[(0, 2, b'post-commit.check-status', b''), (None, None, b'changegroup.app-hooks', <object object at 0x7f5279885590>)]`, the error is `TypeError: '<' not supported between instances of 'NoneType' and 'int'` This fix keeps the same order that was used in python 2 without relying on comparison with None. Differential Revision: https://phab.mercurial-scm.org/D8527

relnotes: copy "next" to "5.4" and clear "next" This is the same thing as we've done for the previous few releases. Differential Revision: https://phab.mercurial-scm.org/D8546

dirstate: make sure the dirstate is loaded before the changelog (issue6303) Before this change, it was possible for the changelog to be loaded before the dirstate. If a transaction happens betwen the changelog and dirstate reading, the dirstate can up end poitning toward a revision not existing in the (olded) changelog. This lead to a warning. With this revision, we preload the dirstate parent before reading the changelog. This has a negligible performance impact on performance for all case we are tracking. Differential Revision: https://phab.mercurial-scm.org/D8528

rust-dirstatemap: don't read the dirstate when requesting parents A future patch for issue 6303 reveals a big performance regression in the Rust `DirstateMap` that reads the entire dirstate when requesting parents instead of the first 40 bytes. `perfdiscovery` gets a *significant* speedup (from 0.101 to 0.016) when applied against said patch. I'm assuming it has other performance benefits, but this is already a good enough win. Differential Revision: https://phab.mercurial-scm.org/D8513

py3: fix exception in pull when several things happen to a bookmark Specifically, when `changes` is: [(b'@upstream-committed', None, <function ui.status>, b'updating bookmark @upstream-committed\n'), (b'@upstream-committed', binary-node, <function ui.warn>, b'divergent bookmark @ stored as @upstream-committed\n')] sorting the list raises: TypeError: '<' not supported between instances of 'bytes' and 'NoneType' Differential Revision: https://phab.mercurial-scm.org/D8523

cleanup: avoid extra node/ctx conversions in logcmdutil.diffordiffstat I'm about to write some code that wants to pass a memctx to diffordiffstat, but this feels like a meritorious cleanup anyway, since the first thing this method does is turn nodes into contexts, and most callers have a context handy. Differential Revision: https://phab.mercurial-scm.org/D8502

pyoxidizer: formatting bazel definitions This meets the Bazel style guide: https://docs.bazel.build/versions/master/skylark/bzl-style.html and was mostly done automatically with buildifier. Differential Revision: https://phab.mercurial-scm.org/D8521

fastexport: adjust output to be more canonical For time zones, git doesn't consider +0 and -0 the same timezone, so use the former canonically. Add a test case to ensure that non-UTC offsets are handled correctly. The real name part of the committer name is normally not quoted, so don't enforce that. Differential Revision: https://phab.mercurial-scm.org/D8522

bash_completion: do not use aliased hg if it sources a script (issue6308) I have an alias that sources a script around hg. Mercurial's bash_completion script tries to use this as its main hg binary. But sourcing a wrapper breaks Bash's completion. So this patch disables using the alias as the hg binary if it starts with "source ". Alias resolution was introduced in rev 191ab08e7099 for users with "alias hg='hg --some_opts'". See https://www.mercurial-scm.org/repo/hg/rev/191ab08e7099

demandimport: fix compatibility with meta path finders w/o find_spec() method Meta path finders got a find_spec() method in Python version 3.4. The sys.meta_path documentation says that the deprecated find_module() method is used as a fallback. Setuptool’s VendorImporter still doesn’t have the find_spec() method, which resulted in a crash e.g. within a virtual environment. For reference, I opened an issue for that: https://github.com/pypa/setuptools/issues/2104. An alternative implementation would have been to implement a wrapper for find_module() itself and raise an AttributeError when accessing find_spec() if the wrapped finder doesn’t have it.

tests: fix timer scaling in wait-on-file When using the default test timeouts, wait-on-file would not wait for $n seconds, but $n/100 seconds. This resulted in easy timeouts on even moderately busy fast machines. Fix the scaling to apply in all cases. Adjust the stepping slightly to be nicer to systems with the historic 100Hz time base to ensure that the scheduler actually switches to a different process and gives them time to work. Differential Revision: https://phab.mercurial-scm.org/D8505

manifest-cache: ignore IOError while writing If the wcache directory is non writable for some reason (eg: belong to root). Trying to write to it currently crash Mercurial. Instead we ignore the error and skip writing that cache. We should probably improve the user experience of multiple users interacting with the same repository. However this is not an adventure for stable. Differential Revision: https://phab.mercurial-scm.org/D8512

copy: give better error message when no source paths found with --at-rev The new error message matches what we show when marking copies in the working copy. Differential Revision: https://phab.mercurial-scm.org/D8496

tests: show poor error message for `hg cp -A --at-rev . non-existent dst` Differential Revision: https://phab.mercurial-scm.org/D8495

copy: to find copy source, walk parent of revision we're marking copies in As shown in the previous patch, `hg cp --after --at-rev . src dst` fails if `src` is not in `.`. It seems obvious that you should always walk the *parent* of the revision you're marking copies in, but that's not how it was done for the working copy, and I didn't think to change it when marking copies in a non-working-copy commit. This patch fixes that by walking the parent commit instead, but only if we're marking copies for a non-working-copy commit. We need to leave the working-copy code unchanged because it depends on the weird behavior of `workingctx.walk()`. With these changes, there's very little overlap between the working-copy version and the non-working-copy version of `walkpats()`, but I've refrained from cleaning that up on the stable branch. Differential Revision: https://phab.mercurial-scm.org/D8494

tests: show that `hg cp -A --at-rev .` doesn't work for renames I clearly forgot to implement (and test) support for marking of renames when I added support for marking of copies :( Differential Revision: https://phab.mercurial-scm.org/D8493

formatting: add missing newline Differential Revision: https://phab.mercurial-scm.org/D8509

fastexport: downgrade message about already exported changesets to debug The old warning level is just too noisy for incremental processing. Differential Revision: https://phab.mercurial-scm.org/D8487

revisions: parse "x123" as "nodeid starting with 123" without prefixhexnode `experimental.revisions.prefixhexnode` makes it so the template function `shortest()` uses an "x" prefix to disambiguate between short nodeids and revnums. That config has so far also been used for enabling parsing of "x123" unambiguously as a nodeid. That makes it a little annoying for people who have prefixhexnode=yes to share such nodeids with people who have prefixhexnode=no ("x123" will be considered invalid for them). There seems to be little harm in allowing that parsing for everyone. We still let e.g. bookmark names like "x123" take precedence over the nodeid, so that's not a concern. The only thing I can think of is that people get used to the "x" prefix being valid, making it impossible for us to change to a different prefix if we wanted to do that when graduating the feature. Differential Revision: https://phab.mercurial-scm.org/D8514

status: use cmdutil.check_at_most_one_arg() for checking --rev/--change There are apparently no tests for this either. Differential Revision: https://phab.mercurial-scm.org/D8511

diff: use cmdutil.check_at_most_one_arg() for checking --rev/--change The same check was done in extdiff as well, so I fixed that too. There are apparently no tests for this. Differential Revision: https://phab.mercurial-scm.org/D8510

copy: give better error message when no source paths found with --at-rev The new error message matches what we show when marking copies in the working copy. Differential Revision: https://phab.mercurial-scm.org/D8496

tests: show poor error message for `hg cp -A --at-rev . non-existent dst` Differential Revision: https://phab.mercurial-scm.org/D8495

copy: to find copy source, walk parent of revision we're marking copies in As shown in the previous patch, `hg cp --after --at-rev . src dst` fails if `src` is not in `.`. It seems obvious that you should always walk the *parent* of the revision you're marking copies in, but that's not how it was done for the working copy, and I didn't think to change it when marking copies in a non-working-copy commit. This patch fixes that by walking the parent commit instead, but only if we're marking copies for a non-working-copy commit. We need to leave the working-copy code unchanged because it depends on the weird behavior of `workingctx.walk()`. With these changes, there's very little overlap between the working-copy version and the non-working-copy version of `walkpats()`, but I've refrained from cleaning that up on the stable branch. Differential Revision: https://phab.mercurial-scm.org/D8494

tests: show that `hg cp -A --at-rev .` doesn't work for renames I clearly forgot to implement (and test) support for marking of renames when I added support for marking of copies :( Differential Revision: https://phab.mercurial-scm.org/D8493

rust-matchers: add TODO about incomplete `Display` for `IncludeMatcher` This is purely for future reference, I don't think this is a problem right now, since the `Display` is *only* used to ease debugging and has no real users. Differential Revision: https://phab.mercurial-scm.org/D8492

rust-filepatterns: match exact `rootglob`s with a `HashSet`, not in the regex This optimization yields some very interesting results in `rootglob`-heavy repositories. I build a test repository of the following structure: ``` root /<uuid>/build/empty_file ... repeat for 4000 entries ``` and a `.hgignore` containing the corresponding 4000 `rootglob` entries pointing to all `build/` folders. Rust+c `hg status` goes from 350ms down to 110ms. Differential Revision: https://phab.mercurial-scm.org/D8491

dirstate: force _checkexec to return a bool posix.checkexec can return True, False, or None. The rust status implementation expects a boolean, so make sure _checkexec returns a boolean. Differential Revision: https://phab.mercurial-scm.org/D8432

locking: wait for locks in `hg cp` and `hg mv` I traced this back to revision 1822 (64df422), and there's no explanation why we would prefer to error out instead of waiting for the locks. Differential Revision: https://phab.mercurial-scm.org/D8464

nodemap: teach `hg debugformat` about the persistent nodemap option We have a new requirement, we should display it. Differential Revision: https://phab.mercurial-scm.org/D8430

upgrade: support the --quiet flag The command is currently very verbose with a various bit of output being time sensitive or randomized. The make invocation bulky and hard to match in the test. We move various message from `ui.write` to `ui.status` in order for the `--quiet` flag to have effect and helps the situation. As a side benefit, we can replace the various redirection to > /dev/null with the --quiet flag. Differential Revision: https://phab.mercurial-scm.org/D8429

upgrade: clearly list optimisations This makes the command operation clearer. And this will be necessary to have a short version of this information with the next changesets, teaching `hg debugupgraderepo` about `--quiet`. Differential Revision: https://phab.mercurial-scm.org/D8428

nodemap: move the mode option to storage.revlog.nodemap.mode The option stay experimental as long as the main feature is. Differential Revision: https://phab.mercurial-scm.org/D8422

nodemap: move the option for mmap usage to storage.revlog.nodemap.mmap The option is stay experimental as long as the main feature is. Differential Revision: https://phab.mercurial-scm.org/D8421

nodemap: move and update the commend about persistence being experimental The comment was at the wrong place (on the developer option instead of the activation switch). So we move it at the right location and update it. Differential Revision: https://phab.mercurial-scm.org/D8420

nodemap: move the main switch to the `format` section The config to enable persistent nodemap is now `format.use-persistent-nodemap`. However the option remain marked as experimental because it only improve performance for people using the rust extensions. Differential Revision: https://phab.mercurial-scm.org/D8419

nodemap: drop the 'exp-' prefix for internal opener option The feature is now in a descent shape and we can consider having it "less" experimental. We won't be able to make it "totally" non-experimental, because its benefit rely on rust, which is totally experimental. Differential Revision: https://phab.mercurial-scm.org/D8418

nodemap: gate the feature behind a new requirement Now that the feature is working smoothly, a question was still open, should we gate the feature behind a new requirement or just treat it as a cache to be warmed by those who can and ignored by other. The advantage of using the cache approach is a transparent upgrade/downgrade story, making the feature easier to move to. However having out of date cache can come with a significant performance hit for process who expect an up to date cache but found none. In this case the file needs to be stored under `.hg/cache`. The "requirement" approach guarantee that the persistent nodemap is up to date. However, it comes with a less flexible activation story since an explicite upgrade is required. In this case the file can be stored in `.hg/store`. This wiki page is relevant to this questions: https://www.mercurial-scm.org/wiki/ComputedIndexPlan So which one should we take? Another element came into plan, the persistent nodemap use the `add` method of the transaction, it is used to keep track of a file content before a transaction in case we need to rollback it back. It turns out that the `transaction.add` API does not support file stored anywhere than `.hg/store`. Making it support file stored elsewhere is possible, require a change in on disk transaction format. Updating on disk file requires… introducing a new requirements. As a result, we pick the second option "gating the persistent nodemap behind a new requirements". Differential Revision: https://phab.mercurial-scm.org/D8417

nodemap: move on disk file to version 1 The current format contains the information we need, lets freeze it before the release. Differential Revision: https://phab.mercurial-scm.org/D8416

nodemap: add a new mode value, "strict" When "strict" is set, operation will refuse to use the slow path and abort if they would. This is useful for testing, benchmarking and server operation. Differential Revision: https://phab.mercurial-scm.org/D8415

nodemap: add a new mode option, with an optional "warn" value When "warn" is set, user will get notified when the slow code, used for compatibility is used. This can help people to detect situation were using that feature will give them a slowdown instead of a speedup. Differential Revision: https://phab.mercurial-scm.org/D8414

nodemap: also warm manifest nodemap with other caches The `hg debugupdatecache` command now also warm the persistent nodemap for the manifest (when applicable). Differential Revision: https://phab.mercurial-scm.org/D8411

nodemap: also use persistent nodemap for manifest The manifest as a different usage pattern than the changelog. First, while the lookup in changelog are not garanteed to match, the lookup in the manifest nodemap come from changelog and will exist in the manifest. In addition, looking up a manifest almost always result in unpacking a manifest an operation that rarely come cheap. Nevertheless, using a persistent nodemap provide a significant gain for some operations. For our measurementw, we use `hg cat --rev REV FILE` on the our reference mozilla-try. On this repository the persistent nodemap cache is about 29 MB in side for a total store side of 11,988 MB File with large history (file: b2g/config/gaia.json, revision: 195a1146daa0) no optimisation: 0.358s using mmap for index: 0.297s (-0.061s) persistent nodemap for changelog only: 0.275s (-0.024s) persistent nodemap for manifest too: 0.258s (-0.017s) File with small history (file: .hgignore, revision: 195a1146daa0) no optimisation: 0.377s using mmap for index: 0.296s (-0.061s) persistent nodemap for changelog only: 0.274s (-0.022s) persistent nodemap for manifest too: 0.257s (-0.017s) Same file but using a revision (8ba995b74e18) with a smaller manifest (3944829 bytes vs 10 bytes) no optimisation: 0.192s (-0.185s) using mmap for index: 0.131s (-0.061s) persistent nodemap for changelog only: 0.106s (-0.025s) persistent nodemap for manifest too: 0.087s (-0.019s) Differential Revision: https://phab.mercurial-scm.org/D8410

nodemap: create files in the repository used in the test We need a manifest with more content to test persistent nodemap for manifest. This change the repository content and affect all the hashes. Differential Revision: https://phab.mercurial-scm.org/D8409

rust-matchers: add timing tracing to regex compilation This might be useful to diagnose later performance issues or just to show the difference between engines. Differential Revision: https://phab.mercurial-scm.org/D8498

url: fix a bytes vs str crash in processing proxy headers (issue6249) I have no idea how to make a test for this, so if somebody knows, feel free to add one or follow up on this. The bug reporter reported that it worked for them, so there may not be other hidden issues here. Differential Revision: https://phab.mercurial-scm.org/D8485

pullbundles: use unfiltered repo for head/base matching The unfiltered view works even when changeset transistion from draft to hidden phase. The normal visibility is already ensured by discovery as invisible heads would have been filtered out before. Skipping the filtering has a positive impact on performance, too. Differential Revision: https://phab.mercurial-scm.org/D8481

procutil: always waiting on child processes to prevent zombies with 'hg serve' When runbgcommand is invoked by an extension with ensurestart=False, we never called waitpid - which is fine in most cases, except if that's happening on a command server (e.g. chg), in which case the child defunct process will just sit there for as long as the server is running. The actual semantics of SIGCHLD signal handling is a lot more complex than it seems, and the POSIX standard *seems* to read that it's ignored by default and everything would just work without the waitpid if we're not listening for it, but the truth is that it's only ignored if we *explicitly* set it to SIG_IGN. We further cannot set it to SIG_IGN or to a catch-all handler across all of 'hg serve', because Python's suprocess.Popen relies on that signal, and a few specific parts of hg also set custom handlers, so instead we wait for specific PIDs in dedicated threads. I did a poor-man's benchmark of the thread creation and it seems to take about 1ms, which is way better than the 20+ms from ensurestart=True. Differential Revision: https://phab.mercurial-scm.org/D8497

tests: use regular POSIX shell wait-on-file requires one POSIX extension (sleep with non-integral argument), but it doesn't require any bash extensions, so just require a normal POSIX shell. While here, use consistent formatting without redundant ; Differential Revision: https://phab.mercurial-scm.org/D8500

rust-regex: increase the DFA size limit for the `regex` crate `re2`'s DFA limit is already increased in `rust/hg-core/src/re2/rust_re2.cpp`, the same has to be done for the `regex` crate. Big repositories with big `.hgignore`s will sometimes hit this limit and face extreme performance regressions (I've seen one take *minutes* for `hg status`). Differential Revision: https://phab.mercurial-scm.org/D8499

merge with stable

Added signature for changeset cf3e07d7648a

Added tag 5.4 for changeset cf3e07d7648a

tests: clarify a comment describing a phabricator test scenario Per review feedback. Differential Revision: https://phab.mercurial-scm.org/D8455

phabricator: ensure that `phabsend` is given a contiguous, linear commit range Supplying a non-linear range was another orphan factory. While in theory there could be a use case for skipping over garbage commits (like adding debugging) and getting the valuable commits extracted out at the same time as posting a review, it seems far more likely that specifying a non-linear range is a user error. This is another case of issue6045, but predates both 0680b8a1992a and 601ce5392cb0. Neither the `--no-amend` case nor resubmitting a previously submitted commit would cause orphans. But for the sake of simplicity and to keep the parents tracked on Phabricator in the proper state, ban missing commits unconditionally. Differential Revision: https://phab.mercurial-scm.org/D8454

merge with stable

automation: support building Python 3 MSI installers This is very similar to what we just did for Inno. Differential Revision: https://phab.mercurial-scm.org/D8484

automation: support building Python 3 Inno installers The core packaging code now supports building Python 3 installers using PyOxidizer. Let's teach the automation code to invoke it so that we produce both Python 2 and Python 3 based exe installers. When publishing the artifacts, the Python 3 versions are preferred over the Python 2 versions given their higher weight (10 versus 9). This may be a controversial change. But I think making Python 3 the default is warranted, as it is the future. The Python 2 installers are still fully supported and can be installed should issues with Python 3 arise. Differential Revision: https://phab.mercurial-scm.org/D8483

automation: add extra arguments when building Inno These were being fed into the template expansion but not being used. This meant --version was not getting set when it should have been. Differential Revision: https://phab.mercurial-scm.org/D8482

packaging: add -python2 to Windows installer filenames We just taught the Windows installers to produce Python 3 variants built with PyOxidizer. Our plan is to publish both Python 2 and Python 3 versions of the installers for Mercurial 5.4. This commit teaches the Inno and WiX installers to add an optional string suffix to the installer name. On Python 2, that suffix is "-python2." We reserve the existing name for the Python 3 installers, which we want to make the default. Differential Revision: https://phab.mercurial-scm.org/D8479

automation: support building Windows wheels for Python 3.7 and 3.8 The time has come to support Python 3 on Windows. Let's teach our automation code to produce Windows wheels for Python 3.7 and 3.8. We could theoretically support 3.5 and 3.6. But I don't think it is worth it. People on Windows generally use the Mercurial installers, not wheels. And I'd prefer we limit variability and not have to worry about supporting earlier Python versions if it can be helped. As part of this, we change the invocation of pip to `python.exe -m pip`, as this is what is being recommended in Python docs these days. And it seemed to be required to avoid a weird build error. Why, I'm not sure. But it looks like pip was having trouble finding a Visual Studio files when invoked as `pip.exe` but not when using `python.exe -m pip`. Who knows. Differential Revision: https://phab.mercurial-scm.org/D8478

packaging: support building WiX installers with PyOxidizer We initially implemented PyOxidizer support for Inno installers. That did most of the heavy work of integrating PyOxidizer into the packaging system. Implementing WiX installer support was pretty straightforward. Aspects of this patch look very similar to Inno's. The main difference is the handling of the Visual C++ Redistributable Runtime files. The WiX installer was formerly using merge modules to install the VC++ 9.0 runtime because this feature is supported by the WiX installer (it isn't easily available to Inno installers). Our strategy for the runtime files is to install the vcruntime140.dll file next to hg.exe just like any other file. While we could leverage WiX's functionality for invoking a VCRedist installer, I don't want to deal with the complexity at this juncture. So, we let run_pyoxidizer() copy vcruntime140.dll into the staging directory (like it does for Inno) and our dynamic WiX XML generator picks it up as a regular file and installs it. We did, however, have to teach mercurial.wxs how to conditionally use the merge modules. But this was rather straightforward. Comparing the file layout of the WiX installers before and after: * Various lib/*.{pyd, dll} files no longer exist * python27.dll was replaced by python37.dll * vcruntime140.dll was added All these changes are expected due to the transition to Python 3 and to PyOxidizer, which embeded the .pyd and .dll files in hg.exe. Differential Revision: https://phab.mercurial-scm.org/D8477

packaging: move version derivation to run_wix_packaging() With the previous commit moving signing inline, we no longer need to compute the version string in build_installer() and can instead move this logic to run_wix_packaging(). This makes the logic in build_installer() simpler, which makes it easier to implement alternate building mechanisms. Differential Revision: https://phab.mercurial-scm.org/D8476

packaging: integrate signing into run_wix_packaging() Previously, signing was implemented via a separate function which called build_installer() and then called signing functionality. In this model, in order to implement an alternative build mechanism, we would have to invent a new variant to handle signing as well. This commit merges the signing logic into the function invoking wix. If we pass an argument holding metadata about how to sign, we sign hg.exe and the installer. This means all we have to do is pass in signing info and the signing just works. A slight change here is that signing of hg.exe happens in the staging directory as opposed to before the staging directory is populated. I don't think this matters. Differential Revision: https://phab.mercurial-scm.org/D8475

packaging: isolate invocation of WiX to own function Like we did for Inno, we want to split out the building of Mercurial from invoking the packaging tool so that we can introduce an alternate build mechanism. As part of this refactor, there are inconsequential changes to file layouts. Before, some shared files such as the WiX binaries and merge modules would be installed under build/. Now, they are installed under build/wix-*. This is to keep implementation simpler. But it also helps keep build state more isolated. Differential Revision: https://phab.mercurial-scm.org/D8474

packaging: support building Inno installer with PyOxidizer We want to start distributing Mercurial on Python 3 on Windows. PyOxidizer will be our vehicle for achieving that. This commit implements basic support for producing Inno installers using PyOxidizer. While it is an eventual goal of PyOxidizer to produce installers, those features aren't yet implemented. So our strategy for producing Mercurial installers is similar to what we've been doing with py2exe: invoke a build system to produce files then stage those files into a directory so they can be turned into an installer. We had to make significant alterations to the pyoxidizer.bzl config file to get it to produce the files that we desire for a Windows install. This meant differentiating the build targets so we can target Windows specifically. We've added a new module to hgpackaging to deal with interacting with PyOxidizer. It is similar to pyexe: we invoke a build process then copy files to a staging directory. Ideally these extra files would be defined in pyoxidizer.bzl. But I don't think it is worth doing at this time, as PyOxidizer's config files are lacking some features to make this turnkey. The rest of the change is introducing a variant of the Inno installer code that invokes PyOxidizer instead of py2exe. Comparing the Python 2.7 based Inno installers with this one, the following changes were observed: * No lib/*.{pyd, dll} files * No Microsoft.VC90.CRT.manifest * No msvc{m,p,r}90.dll files * python27.dll replaced with python37.dll * Add vcruntime140.dll file The disappearance of the .pyd and .dll files is acceptable, as PyOxidizer has embedded these in hg.exe and loads them from memory. The disappearance of the *90* files is acceptable because those provide the Visual C++ 9 runtime, as required by Python 2.7. Similarly, the appearance of vcruntime140.dll is a requirement of Python 3.7. Differential Revision: https://phab.mercurial-scm.org/D8473