rust-matchers: add timing tracing to regex compilation This might be useful to diagnose later performance issues or just to show the difference between engines. Differential Revision: https://phab.mercurial-scm.org/D8498

url: fix a bytes vs str crash in processing proxy headers (issue6249) I have no idea how to make a test for this, so if somebody knows, feel free to add one or follow up on this. The bug reporter reported that it worked for them, so there may not be other hidden issues here. Differential Revision: https://phab.mercurial-scm.org/D8485

pullbundles: use unfiltered repo for head/base matching The unfiltered view works even when changeset transistion from draft to hidden phase. The normal visibility is already ensured by discovery as invisible heads would have been filtered out before. Skipping the filtering has a positive impact on performance, too. Differential Revision: https://phab.mercurial-scm.org/D8481

procutil: always waiting on child processes to prevent zombies with 'hg serve' When runbgcommand is invoked by an extension with ensurestart=False, we never called waitpid - which is fine in most cases, except if that's happening on a command server (e.g. chg), in which case the child defunct process will just sit there for as long as the server is running. The actual semantics of SIGCHLD signal handling is a lot more complex than it seems, and the POSIX standard *seems* to read that it's ignored by default and everything would just work without the waitpid if we're not listening for it, but the truth is that it's only ignored if we *explicitly* set it to SIG_IGN. We further cannot set it to SIG_IGN or to a catch-all handler across all of 'hg serve', because Python's suprocess.Popen relies on that signal, and a few specific parts of hg also set custom handlers, so instead we wait for specific PIDs in dedicated threads. I did a poor-man's benchmark of the thread creation and it seems to take about 1ms, which is way better than the 20+ms from ensurestart=True. Differential Revision: https://phab.mercurial-scm.org/D8497

tests: use regular POSIX shell wait-on-file requires one POSIX extension (sleep with non-integral argument), but it doesn't require any bash extensions, so just require a normal POSIX shell. While here, use consistent formatting without redundant ; Differential Revision: https://phab.mercurial-scm.org/D8500

rust-regex: increase the DFA size limit for the `regex` crate `re2`'s DFA limit is already increased in `rust/hg-core/src/re2/rust_re2.cpp`, the same has to be done for the `regex` crate. Big repositories with big `.hgignore`s will sometimes hit this limit and face extreme performance regressions (I've seen one take *minutes* for `hg status`). Differential Revision: https://phab.mercurial-scm.org/D8499

merge with stable

Added signature for changeset cf3e07d7648a

Added tag 5.4 for changeset cf3e07d7648a

tests: clarify a comment describing a phabricator test scenario Per review feedback. Differential Revision: https://phab.mercurial-scm.org/D8455

phabricator: ensure that `phabsend` is given a contiguous, linear commit range Supplying a non-linear range was another orphan factory. While in theory there could be a use case for skipping over garbage commits (like adding debugging) and getting the valuable commits extracted out at the same time as posting a review, it seems far more likely that specifying a non-linear range is a user error. This is another case of issue6045, but predates both 0680b8a1992a and 601ce5392cb0. Neither the `--no-amend` case nor resubmitting a previously submitted commit would cause orphans. But for the sake of simplicity and to keep the parents tracked on Phabricator in the proper state, ban missing commits unconditionally. Differential Revision: https://phab.mercurial-scm.org/D8454

merge with stable

automation: support building Python 3 MSI installers This is very similar to what we just did for Inno. Differential Revision: https://phab.mercurial-scm.org/D8484

automation: support building Python 3 Inno installers The core packaging code now supports building Python 3 installers using PyOxidizer. Let's teach the automation code to invoke it so that we produce both Python 2 and Python 3 based exe installers. When publishing the artifacts, the Python 3 versions are preferred over the Python 2 versions given their higher weight (10 versus 9). This may be a controversial change. But I think making Python 3 the default is warranted, as it is the future. The Python 2 installers are still fully supported and can be installed should issues with Python 3 arise. Differential Revision: https://phab.mercurial-scm.org/D8483

automation: add extra arguments when building Inno These were being fed into the template expansion but not being used. This meant --version was not getting set when it should have been. Differential Revision: https://phab.mercurial-scm.org/D8482

packaging: add -python2 to Windows installer filenames We just taught the Windows installers to produce Python 3 variants built with PyOxidizer. Our plan is to publish both Python 2 and Python 3 versions of the installers for Mercurial 5.4. This commit teaches the Inno and WiX installers to add an optional string suffix to the installer name. On Python 2, that suffix is "-python2." We reserve the existing name for the Python 3 installers, which we want to make the default. Differential Revision: https://phab.mercurial-scm.org/D8479

automation: support building Windows wheels for Python 3.7 and 3.8 The time has come to support Python 3 on Windows. Let's teach our automation code to produce Windows wheels for Python 3.7 and 3.8. We could theoretically support 3.5 and 3.6. But I don't think it is worth it. People on Windows generally use the Mercurial installers, not wheels. And I'd prefer we limit variability and not have to worry about supporting earlier Python versions if it can be helped. As part of this, we change the invocation of pip to `python.exe -m pip`, as this is what is being recommended in Python docs these days. And it seemed to be required to avoid a weird build error. Why, I'm not sure. But it looks like pip was having trouble finding a Visual Studio files when invoked as `pip.exe` but not when using `python.exe -m pip`. Who knows. Differential Revision: https://phab.mercurial-scm.org/D8478

packaging: support building WiX installers with PyOxidizer We initially implemented PyOxidizer support for Inno installers. That did most of the heavy work of integrating PyOxidizer into the packaging system. Implementing WiX installer support was pretty straightforward. Aspects of this patch look very similar to Inno's. The main difference is the handling of the Visual C++ Redistributable Runtime files. The WiX installer was formerly using merge modules to install the VC++ 9.0 runtime because this feature is supported by the WiX installer (it isn't easily available to Inno installers). Our strategy for the runtime files is to install the vcruntime140.dll file next to hg.exe just like any other file. While we could leverage WiX's functionality for invoking a VCRedist installer, I don't want to deal with the complexity at this juncture. So, we let run_pyoxidizer() copy vcruntime140.dll into the staging directory (like it does for Inno) and our dynamic WiX XML generator picks it up as a regular file and installs it. We did, however, have to teach mercurial.wxs how to conditionally use the merge modules. But this was rather straightforward. Comparing the file layout of the WiX installers before and after: * Various lib/*.{pyd, dll} files no longer exist * python27.dll was replaced by python37.dll * vcruntime140.dll was added All these changes are expected due to the transition to Python 3 and to PyOxidizer, which embeded the .pyd and .dll files in hg.exe. Differential Revision: https://phab.mercurial-scm.org/D8477

packaging: move version derivation to run_wix_packaging() With the previous commit moving signing inline, we no longer need to compute the version string in build_installer() and can instead move this logic to run_wix_packaging(). This makes the logic in build_installer() simpler, which makes it easier to implement alternate building mechanisms. Differential Revision: https://phab.mercurial-scm.org/D8476

packaging: integrate signing into run_wix_packaging() Previously, signing was implemented via a separate function which called build_installer() and then called signing functionality. In this model, in order to implement an alternative build mechanism, we would have to invent a new variant to handle signing as well. This commit merges the signing logic into the function invoking wix. If we pass an argument holding metadata about how to sign, we sign hg.exe and the installer. This means all we have to do is pass in signing info and the signing just works. A slight change here is that signing of hg.exe happens in the staging directory as opposed to before the staging directory is populated. I don't think this matters. Differential Revision: https://phab.mercurial-scm.org/D8475

packaging: isolate invocation of WiX to own function Like we did for Inno, we want to split out the building of Mercurial from invoking the packaging tool so that we can introduce an alternate build mechanism. As part of this refactor, there are inconsequential changes to file layouts. Before, some shared files such as the WiX binaries and merge modules would be installed under build/. Now, they are installed under build/wix-*. This is to keep implementation simpler. But it also helps keep build state more isolated. Differential Revision: https://phab.mercurial-scm.org/D8474

packaging: support building Inno installer with PyOxidizer We want to start distributing Mercurial on Python 3 on Windows. PyOxidizer will be our vehicle for achieving that. This commit implements basic support for producing Inno installers using PyOxidizer. While it is an eventual goal of PyOxidizer to produce installers, those features aren't yet implemented. So our strategy for producing Mercurial installers is similar to what we've been doing with py2exe: invoke a build system to produce files then stage those files into a directory so they can be turned into an installer. We had to make significant alterations to the pyoxidizer.bzl config file to get it to produce the files that we desire for a Windows install. This meant differentiating the build targets so we can target Windows specifically. We've added a new module to hgpackaging to deal with interacting with PyOxidizer. It is similar to pyexe: we invoke a build process then copy files to a staging directory. Ideally these extra files would be defined in pyoxidizer.bzl. But I don't think it is worth doing at this time, as PyOxidizer's config files are lacking some features to make this turnkey. The rest of the change is introducing a variant of the Inno installer code that invokes PyOxidizer instead of py2exe. Comparing the Python 2.7 based Inno installers with this one, the following changes were observed: * No lib/*.{pyd, dll} files * No Microsoft.VC90.CRT.manifest * No msvc{m,p,r}90.dll files * python27.dll replaced with python37.dll * Add vcruntime140.dll file The disappearance of the .pyd and .dll files is acceptable, as PyOxidizer has embedded these in hg.exe and loads them from memory. The disappearance of the *90* files is acceptable because those provide the Visual C++ 9 runtime, as required by Python 2.7. Similarly, the appearance of vcruntime140.dll is a requirement of Python 3.7. Differential Revision: https://phab.mercurial-scm.org/D8473

packaging: split Inno installer building from Mercurial building We want to make the logic for producing the installer agnostic about how Mercurial is built to allow for alternate build methods (like PyOxidizer). Differential Revision: https://phab.mercurial-scm.org/D8472

packaging: remove pyoxidizer.bzl from packaging directory We have another version in rust/hgcli that is more modern and is already associated with our Rust CLI project. Differential Revision: https://phab.mercurial-scm.org/D8471

contrib: install PyOxidizer in Linux and Windows environments For Linux, this was trivial. For Windows, we need to teach the powershell script to install Rust as well. This was also pretty straightforward. Differential Revision: https://phab.mercurial-scm.org/D8468

diff: re-establish linear runtime performance The previous method with sum() and list() creates a new list object for every hunk. Then sum() is used to flatten out this sequence of lists. The sum() function is not "lazy", but creates a new list object for every "+" operation and so this code had quadratic runtime behaviour.

copies: fix the changeset based algorithm regarding merge In 99ebde4fec99, we changed the list of files stored into the `files` field. This lead to the changeset centric copy algorithm to break in various merge situation involving merge. Older information could reach the merge through `p1`, and while information from `p2` was strictly fresher, it would get overwritten anyway. We update the situation with more details about which revision introduces rename information. This help use making the right decision in case of merge. We are now running a more comprehensive suite of test with include this kind of situation. The behavior differ slightly from the filelog based in a couple of instance. There is mostly two distinct cases: 1) there are conflicting rename information in a merge (different rename history on each side). In this case the filelog based implementation arbitrarily pick a side based on the file-revision-number. So it depends on a local factor. The changeset centric algorithm will use a deterministic approach, by picking the information coming from the first parent of the merge. This is stable across different clone. 2) rename information related to file that exist in both source and destination. The filelog based implementation do not even try to detect these, however the changeset centric one get them for "free" (it is simpler to detect them than not). The new implementation focus on correctness. Performance improvement will come later. Differential Revision: https://phab.mercurial-scm.org/D8244

merge with stable

rust-chg: clean up excessive indents Differential Revision: https://phab.mercurial-scm.org/D8450

rust-chg: do not terminate tokio runtime until pager exits We no longer need to spawn a task just to keep the pager handle. The pager handle can be held by ChgUiHandler since the handler itself is not consumed and recreated across async calls. Differential Revision: https://phab.mercurial-scm.org/D8449

rust-chg: modernize entry function Finally the entire build passes. There's a bug that run() no longer waits for the spawned pager, which will be fixed by the next patch. Differential Revision: https://phab.mercurial-scm.org/D8448

rust-chg: reimplement locator by using async/await and tokio-0.2 connect_spawned() is rewritten from scratch by using std::process. Before, it would select completion of either connection or server process. New code could be implemented as such, but it's much simpler to occasionally run try_wait() to detect server death. Differential Revision: https://phab.mercurial-scm.org/D8447

rust-chg: reimplement ChgClientExt as ChgClient wrapper ChgClient is no longer an extension trait because: a. Client object is not consumed and recreated in future-0.3 world, which unblocks writing a simple wrapper struct. b. async fn isn't allowed in trait. Overall, the API should become simpler. Differential Revision: https://phab.mercurial-scm.org/D8446

rust-chg: reimplement run_command operation as async function The crafted state machine is no longer needed thanks to async/await. The state machine is basically rewritten as follows: - Ready(..) -> return .. - PollAgain(..) -> run .. and await - Err(..) -> return Err(..) Differential Revision: https://phab.mercurial-scm.org/D8445

rust-chg: reimplement uihandler by using async-trait and tokio-0.2 We no longer have to consume self and arguments. Differential Revision: https://phab.mercurial-scm.org/D8444

rust-chg: have attach_io() simply take reference of AsRawFd object We no longer have to deal with the restriction of the Future type. Before, these file objects couldn't be references and that's the only reason why we had to make stderr an Option<T>. This fixes future type deduction issue of stderr = None, where rustc would complain that T of Option<T> couldn't be deduced. Differential Revision: https://phab.mercurial-scm.org/D8443

rust-status: check for '.hg' regardless of file type (issue6300) '.hg' would show up in `hg status` if were a symlink. Differential Revision: https://phab.mercurial-scm.org/D8461

rust: remove extra empty line This is a gratuitous cleanup. Differential Revision: https://phab.mercurial-scm.org/D8460

upgrade: properly filter action depending on planned work The `determineactions` function filters out deficiency that are not scheduled to be fixed by the target repository configuration. However it only did so for requirement we currently support, letting other actions for unsupported requirement through even if the target repo configuration did not request it. As a result the output of the command was easily polluted by experimental feature with no upgrade support. We rework the code to still filter out requirement based action without the faulty filtering. Differential Revision: https://phab.mercurial-scm.org/D8427

nodemap: skip persistent nodemap warming for revlog not using it Before this patch, the usual checking (especially, inline-ess) were not performed when warming the cache through `hg debugupdatecache`. This is now fixed. Differential Revision: https://phab.mercurial-scm.org/D8408

wait-on-file: adjust the timer counter The wait performed in increment of 0.01 second, but the timer was expressed in second. So we did not wait 20s, we waited 0.2 second. Internally we adjust the counter to be expressed in centisecond. This prevent some flackyness in the test. Differential Revision: https://phab.mercurial-scm.org/D8453

packaging: add docutils as dependency The previous commit revealed that attempting to run `python setup.py build_doc` from the packaging virtualenv failed due to missing docutils package. We didn't notice this before because py2exe Windows packaging appears to use a Python from another virtualenv (which does include docutils) to invoke setup.py. I discovered this as part of implementing packaging outside of that virtualenv environment. Differential Revision: https://phab.mercurial-scm.org/D8470

setup: use sysstr() on process output Otherwise we get a str-bytes mismatch on Python 3 if an error occurs. Differential Revision: https://phab.mercurial-scm.org/D8469

automation: install latest Python versions in Linux Staying up to date. Keeping parity with the Windows environment. Differential Revision: https://phab.mercurial-scm.org/D8467

contrib: update to latest Python 2.7, 3.7, and 3.8 We would ideally update the 3.5 and 3.6 versions as well. But Python didn't generate exe installers for newer versions for some reason. Differential Revision: https://phab.mercurial-scm.org/D8466

automation: always use latest Windows AMI The old AMI isn't available any more. We seem to run into this problem every few months. Amazon (or Microsoft) appears to be removing the old AMIs when they are superseded or something. Let's give up on tracking known images and switch the image selection logic to use the latest published image. Differential Revision: https://phab.mercurial-scm.org/D8465

rust-chg: reimplement attach_io operation as async function In short, MessageLoop<Connection> was redesigned as Protocol<Connection>, and the protocol methods no longer consume self. API changes are briefly documented in the following page: https://docs.rs/tokio-hglib/0.3.0/tokio_hglib/struct.Protocol.html Differential Revision: https://phab.mercurial-scm.org/D8442

rust-chg: upgrade to futures-0.3 based libraries And do some trivial fixes: - BytesMut::put_u32_be() -> put_u32() - tokio_process -> tokio::process, CommandExt -> Command, spawn_async() -> spawn(), stdin() -> stdin - tokio_timer::sleep() -> tokio::time::delay_for() Differential Revision: https://phab.mercurial-scm.org/D8441

rust-chg: exclude futures-dependent modules from build and break things It's impractical to upgrade the codebase incrementally since futures 0.1 and 0.3 APIs are fundamentally different. So this patch temporarily excludes futures-dependent modules from the build. These modules will be upgraded and re-enabled one by one. Differential Revision: https://phab.mercurial-scm.org/D8440

tests: stabilize test-log.t on Windows I think this is because Windows doesn't recognize single quote. Other ssh based clones use this clunky style too. Differential Revision: https://phab.mercurial-scm.org/D8459

tests: stabilize test-convert-hg-source.t on Windows This was a missing update in 1b8fd4af3318. Differential Revision: https://phab.mercurial-scm.org/D8458

commit: tell user what to do with .hg/last-message.txt I have always assumed that the message will be reused by the next `hg commit`, but it seems it's just silently dropped on the next commit. Let's try to be more helpful by telling the user that they have to manually tell hg to reuse it. The file will still be lost if the user runs some other operation in between (like a non-in-memory rebase). That will be fixed once we've switched all operations to be in-memory :) I didn't include `$(hg root)/` in the path in the message to the user because that would have made the message too long. Hopefully the user will figure that part out themselves. Differential Revision: https://phab.mercurial-scm.org/D8463

test-check-rust-format: specify --edition=2018 rustfmt doesn't read Cargo.toml unless it's executed by cargo. https://github.com/rust-lang/rustfmt#rusts-editions

Added signature for changeset 26ce8e751503

Added tag 5.4rc0 for changeset 26ce8e751503

merge default into stable for 5.4 release

templatekw: fix shownames() to check if namespace exists in repo (issue6301) Namespace registration is dynamic, but the corresponding template keyword is registered statically.

wait-on-file: use proper variable in math This seems better and safer to be explicit. Differential Revision: https://phab.mercurial-scm.org/D8426

wait-on-file: don't quote arithmetic argument This is unnecessary and Mac OS X choke on them. Differential Revision: https://phab.mercurial-scm.org/D8425

graft: exit 1 on conflicts, like merge It's more consistent, and makes it nicer to script around hg if you don't have to ignore exit code 255, which is the error code for basically everything in hg. Differential Revision: https://phab.mercurial-scm.org/D8423