packaging: split Inno installer building from Mercurial building We want to make the logic for producing the installer agnostic about how Mercurial is built to allow for alternate build methods (like PyOxidizer). Differential Revision: https://phab.mercurial-scm.org/D8472

packaging: remove pyoxidizer.bzl from packaging directory We have another version in rust/hgcli that is more modern and is already associated with our Rust CLI project. Differential Revision: https://phab.mercurial-scm.org/D8471

contrib: install PyOxidizer in Linux and Windows environments For Linux, this was trivial. For Windows, we need to teach the powershell script to install Rust as well. This was also pretty straightforward. Differential Revision: https://phab.mercurial-scm.org/D8468

diff: re-establish linear runtime performance The previous method with sum() and list() creates a new list object for every hunk. Then sum() is used to flatten out this sequence of lists. The sum() function is not "lazy", but creates a new list object for every "+" operation and so this code had quadratic runtime behaviour.

copies: fix the changeset based algorithm regarding merge In 99ebde4fec99, we changed the list of files stored into the `files` field. This lead to the changeset centric copy algorithm to break in various merge situation involving merge. Older information could reach the merge through `p1`, and while information from `p2` was strictly fresher, it would get overwritten anyway. We update the situation with more details about which revision introduces rename information. This help use making the right decision in case of merge. We are now running a more comprehensive suite of test with include this kind of situation. The behavior differ slightly from the filelog based in a couple of instance. There is mostly two distinct cases: 1) there are conflicting rename information in a merge (different rename history on each side). In this case the filelog based implementation arbitrarily pick a side based on the file-revision-number. So it depends on a local factor. The changeset centric algorithm will use a deterministic approach, by picking the information coming from the first parent of the merge. This is stable across different clone. 2) rename information related to file that exist in both source and destination. The filelog based implementation do not even try to detect these, however the changeset centric one get them for "free" (it is simpler to detect them than not). The new implementation focus on correctness. Performance improvement will come later. Differential Revision: https://phab.mercurial-scm.org/D8244

merge with stable

rust-chg: clean up excessive indents Differential Revision: https://phab.mercurial-scm.org/D8450

rust-chg: do not terminate tokio runtime until pager exits We no longer need to spawn a task just to keep the pager handle. The pager handle can be held by ChgUiHandler since the handler itself is not consumed and recreated across async calls. Differential Revision: https://phab.mercurial-scm.org/D8449

rust-chg: modernize entry function Finally the entire build passes. There's a bug that run() no longer waits for the spawned pager, which will be fixed by the next patch. Differential Revision: https://phab.mercurial-scm.org/D8448

rust-chg: reimplement locator by using async/await and tokio-0.2 connect_spawned() is rewritten from scratch by using std::process. Before, it would select completion of either connection or server process. New code could be implemented as such, but it's much simpler to occasionally run try_wait() to detect server death. Differential Revision: https://phab.mercurial-scm.org/D8447

rust-chg: reimplement ChgClientExt as ChgClient wrapper ChgClient is no longer an extension trait because: a. Client object is not consumed and recreated in future-0.3 world, which unblocks writing a simple wrapper struct. b. async fn isn't allowed in trait. Overall, the API should become simpler. Differential Revision: https://phab.mercurial-scm.org/D8446

rust-chg: reimplement run_command operation as async function The crafted state machine is no longer needed thanks to async/await. The state machine is basically rewritten as follows: - Ready(..) -> return .. - PollAgain(..) -> run .. and await - Err(..) -> return Err(..) Differential Revision: https://phab.mercurial-scm.org/D8445

rust-chg: reimplement uihandler by using async-trait and tokio-0.2 We no longer have to consume self and arguments. Differential Revision: https://phab.mercurial-scm.org/D8444

rust-chg: have attach_io() simply take reference of AsRawFd object We no longer have to deal with the restriction of the Future type. Before, these file objects couldn't be references and that's the only reason why we had to make stderr an Option<T>. This fixes future type deduction issue of stderr = None, where rustc would complain that T of Option<T> couldn't be deduced. Differential Revision: https://phab.mercurial-scm.org/D8443

rust-status: check for '.hg' regardless of file type (issue6300) '.hg' would show up in `hg status` if were a symlink. Differential Revision: https://phab.mercurial-scm.org/D8461

rust: remove extra empty line This is a gratuitous cleanup. Differential Revision: https://phab.mercurial-scm.org/D8460

upgrade: properly filter action depending on planned work The `determineactions` function filters out deficiency that are not scheduled to be fixed by the target repository configuration. However it only did so for requirement we currently support, letting other actions for unsupported requirement through even if the target repo configuration did not request it. As a result the output of the command was easily polluted by experimental feature with no upgrade support. We rework the code to still filter out requirement based action without the faulty filtering. Differential Revision: https://phab.mercurial-scm.org/D8427

nodemap: skip persistent nodemap warming for revlog not using it Before this patch, the usual checking (especially, inline-ess) were not performed when warming the cache through `hg debugupdatecache`. This is now fixed. Differential Revision: https://phab.mercurial-scm.org/D8408

wait-on-file: adjust the timer counter The wait performed in increment of 0.01 second, but the timer was expressed in second. So we did not wait 20s, we waited 0.2 second. Internally we adjust the counter to be expressed in centisecond. This prevent some flackyness in the test. Differential Revision: https://phab.mercurial-scm.org/D8453

packaging: add docutils as dependency The previous commit revealed that attempting to run `python setup.py build_doc` from the packaging virtualenv failed due to missing docutils package. We didn't notice this before because py2exe Windows packaging appears to use a Python from another virtualenv (which does include docutils) to invoke setup.py. I discovered this as part of implementing packaging outside of that virtualenv environment. Differential Revision: https://phab.mercurial-scm.org/D8470

setup: use sysstr() on process output Otherwise we get a str-bytes mismatch on Python 3 if an error occurs. Differential Revision: https://phab.mercurial-scm.org/D8469

automation: install latest Python versions in Linux Staying up to date. Keeping parity with the Windows environment. Differential Revision: https://phab.mercurial-scm.org/D8467

contrib: update to latest Python 2.7, 3.7, and 3.8 We would ideally update the 3.5 and 3.6 versions as well. But Python didn't generate exe installers for newer versions for some reason. Differential Revision: https://phab.mercurial-scm.org/D8466

automation: always use latest Windows AMI The old AMI isn't available any more. We seem to run into this problem every few months. Amazon (or Microsoft) appears to be removing the old AMIs when they are superseded or something. Let's give up on tracking known images and switch the image selection logic to use the latest published image. Differential Revision: https://phab.mercurial-scm.org/D8465

rust-chg: reimplement attach_io operation as async function In short, MessageLoop<Connection> was redesigned as Protocol<Connection>, and the protocol methods no longer consume self. API changes are briefly documented in the following page: https://docs.rs/tokio-hglib/0.3.0/tokio_hglib/struct.Protocol.html Differential Revision: https://phab.mercurial-scm.org/D8442

rust-chg: upgrade to futures-0.3 based libraries And do some trivial fixes: - BytesMut::put_u32_be() -> put_u32() - tokio_process -> tokio::process, CommandExt -> Command, spawn_async() -> spawn(), stdin() -> stdin - tokio_timer::sleep() -> tokio::time::delay_for() Differential Revision: https://phab.mercurial-scm.org/D8441

rust-chg: exclude futures-dependent modules from build and break things It's impractical to upgrade the codebase incrementally since futures 0.1 and 0.3 APIs are fundamentally different. So this patch temporarily excludes futures-dependent modules from the build. These modules will be upgraded and re-enabled one by one. Differential Revision: https://phab.mercurial-scm.org/D8440

tests: stabilize test-log.t on Windows I think this is because Windows doesn't recognize single quote. Other ssh based clones use this clunky style too. Differential Revision: https://phab.mercurial-scm.org/D8459

tests: stabilize test-convert-hg-source.t on Windows This was a missing update in 1b8fd4af3318. Differential Revision: https://phab.mercurial-scm.org/D8458

commit: tell user what to do with .hg/last-message.txt I have always assumed that the message will be reused by the next `hg commit`, but it seems it's just silently dropped on the next commit. Let's try to be more helpful by telling the user that they have to manually tell hg to reuse it. The file will still be lost if the user runs some other operation in between (like a non-in-memory rebase). That will be fixed once we've switched all operations to be in-memory :) I didn't include `$(hg root)/` in the path in the message to the user because that would have made the message too long. Hopefully the user will figure that part out themselves. Differential Revision: https://phab.mercurial-scm.org/D8463

test-check-rust-format: specify --edition=2018 rustfmt doesn't read Cargo.toml unless it's executed by cargo. https://github.com/rust-lang/rustfmt#rusts-editions

Added signature for changeset 26ce8e751503

Added tag 5.4rc0 for changeset 26ce8e751503

merge default into stable for 5.4 release

templatekw: fix shownames() to check if namespace exists in repo (issue6301) Namespace registration is dynamic, but the corresponding template keyword is registered statically.

wait-on-file: use proper variable in math This seems better and safer to be explicit. Differential Revision: https://phab.mercurial-scm.org/D8426

wait-on-file: don't quote arithmetic argument This is unnecessary and Mac OS X choke on them. Differential Revision: https://phab.mercurial-scm.org/D8425

graft: exit 1 on conflicts, like merge It's more consistent, and makes it nicer to script around hg if you don't have to ignore exit code 255, which is the error code for basically everything in hg. Differential Revision: https://phab.mercurial-scm.org/D8423

tests: deal with "ls" vs "ls -A" difference on 2BSD derived systems BSD-derived systems will use "ls -A" when running as root. Adjust the test cases to always use the flag and include .hg and related output as appropiately. Differential Revision: https://phab.mercurial-scm.org/D8397

tests: skip non-readable check for root Trying to check for errors on non-readable hgrc requires UNIX permissions, but still won't work for root. So adjust the check. Differential Revision: https://phab.mercurial-scm.org/D8396

tests: skip CVS tests for root It is not uncommon for cvs to check for root and to refuse work in that case. Differential Revision: https://phab.mercurial-scm.org/D8395

make: drop the `-c` arg to `install` in the documentation makefile This arg caused `gmake install` on OpenIndiana 2019.10 (illumos) fail with: install: The -c, -f, -n options each require a directory following! install: The -c, -f, -n options each require a directory following! install: The -c, -f, -n options each require a directory following! gmake[1]: *** [Makefile:41: install] Error 2 gmake[1]: Leaving directory '/usr/local/share/mercurial/doc' The workaround is to run `gmake install-bin`. The man page for 10.14 says this is to copy the file and is only for compatability, as it is the default. The CentOS 7 man page says it is ignored. The top level makefile doesn't use this argument at all, so I'm not sure why it's here. Differential Revision: https://phab.mercurial-scm.org/D8439

phabricator: restack any new orphans created by phabsend (issue6045) Previously, posting a new review for a non head commit would orphan the head. The general case is any descendant of the selected revisions got orphaned if this was the first time the selected revisions were submitted. It doesn't happen when resubmitting. I've already had coworkers hit this a few times and get confused. Since posting a review isn't generally thought of as an editing operation, it would probably be easier for new users if we just restacked. This avoids restacking existing orphans around the submission because that may involve merge conflict resolution. Users who already have orphans should know how to stabilize them anyway. Differential Revision: https://phab.mercurial-scm.org/D8438

phabricator: prevent posting obsolete commits I don't see why this would be useful in the first place. But I had a coworker submit a single commit that was not a branch head, and the result was to orphan its child and keep the original commit visible. He then did up arrow + Enter, and it happily created a new review (since the URL isn't amended into the original commit specified on the command line) and a new successor, resulting in a local divergence. I'd like to fix the issue with creating orphans, but this is simple enough to prevent on its own. Differential Revision: https://phab.mercurial-scm.org/D8437

phabricator: avoid creating unstable children within the review stack The instability occurred when rebasing something that has already been submitted onto something that hasn't, and then resubmitting the stack. Or as the test shows, just resubmitting and including something earlier that wasn't previously submitted. There's a general case here where any children (not just the ones in the range of commits posted for review) should be re-stabilized. But handling the selected commits here will cause the `local:commit` node values that are tracked on Phabricator to be properly kept in sync. Differential Revision: https://phab.mercurial-scm.org/D8436

phabricator: add an option to fold several commits into one review (issue6244) Now that all of the pieces are in place, alter the user facing command to allow it. This is the default behavior when using `arc`, but I much prefer the 1:1 approach, and I'm tempted to mark this advanced to limit its abuse. I started out calling this `--no-stack` like the feature request suggested, but I found it less obvious (especially when writing the code), so I went with the `hg fold` analogue. This will populate the `Commits` tab in the web UI with the hash of each commit folded into the review. From experimentation, it seems to list them in the order they are received from the extension instead of the actual parent/child relationship. The extension sends them in sorted order, thanks to `templatefilters.json()`. Since there's enough info there for them to put things in the right order, JSON is unordered aside from lists (IIUC), and there doesn't seem to be any harmful side effects, I guess we write this off as their bug. It is simple enough to workaround by putting a check for `util.sortdict` into `templatefilters.json()`, and don't resort in that case. There are a handful of restrictions that are documented in the code, which somebody could probably fix if they're interested. Notably, this requires the (default) `--amend` option, because there's not an easy way to apply a local tag across several commits. This also doesn't do preflight checking to ensure that all previous commits that were part of a single review are selected when updating. That seems expensive. What happens is the excluded commit is dropped from the review, but it keeps the Differential Revision line in the commit message. Not everything can be edited, so it doesn't seem worth making the code even more complicated to handle this edge case. There are a couple of "obsolete feature not enabled but X markers found!" messages that appeared on Windows but not macOS. I have no idea what's going on here, but that's an unrelated issue, so I conditionalized those lines. Differential Revision: https://phab.mercurial-scm.org/D8314

tests: move the phabricator auth token to the global config file The next commit introduces a new repo to simplify its development. This value needs to be modified to record tests, so it doesn't make sense to have to do that twice. The callsign and URL are *not* moved because there are tests that fallback to the .arcconfig file when those aren't present. Differential Revision: https://phab.mercurial-scm.org/D8390

phabricator: add debug logging to show previous node values in `phabsend` This isn't real useful here, but was very useful showing how `phabsend --fold` handles commits created by `hg fold` and `hg split`. It introduces a new debug function and flag instead of using `ui.debug()`, because `--debug` prints out all of the API chatter. Differential Revision: https://phab.mercurial-scm.org/D8389

rust: update README with new information about the Re2 path Differential Revision: https://phab.mercurial-scm.org/D8452

rust: add option of static linking a local Re2 install Previously, only dynamically linking the system-wide install was possible. We force the user to provide one to prevent hard-to-track errors. Differential Revision: https://phab.mercurial-scm.org/D8451

templatekw: cache mergestate even if merge is not ongoing While playing with eBPF, I noticed .hg/merge/state{,2} files were tried to open() for each revision. That's not healthy. Let's cache the "inactive" state as well.

revset: remove explicit sort() from unstable() The instability should be dealt with by baseset.__init__(), and the returned smartset is ordered by 'subset &' anyway.

revset: implement a simple 'foo#generations' expression The result of this expression is equivalent to 'foo#g[:]': all reachable ancestors and descendants of a revset foo. While not very useful functionality on its own, its implementation can be tested and revset.relations dict can be easily used by extensions to add other non-subscript relations. Differential Revision: https://phab.mercurial-scm.org/D8394

revset: rename generationsrel() to generationssubrel() This highlights that this is an implementation of 'x#y[z]' relation subscript operator as opposed to 'x#y' relation operator. Differential Revision: https://phab.mercurial-scm.org/D8393

manifest: leave a TODO where we may have more work for sha1 portability Differential Revision: https://phab.mercurial-scm.org/D8375

manifest: add test for long hashes Differential Revision: https://phab.mercurial-scm.org/D8373

manifest: remove a final 40-byte assumption from pure-python parser Differential Revision: https://phab.mercurial-scm.org/D8372

manifest: fix yet another 20-byte-hash assumption Differential Revision: https://phab.mercurial-scm.org/D8371

manifest: fix another pure-parsing 20-byte assumption Differential Revision: https://phab.mercurial-scm.org/D8370

manifest: teach treemanifest about long hashes Differential Revision: https://phab.mercurial-scm.org/D8369

manifest: start removing 40-byte hash restrictions from C code Differential Revision: https://phab.mercurial-scm.org/D8368

manifest: remove a 20-byte-hash assumption from pure manifest parsing This is almost certainly slower, but it shouldn't be by much. We need to do this for the eventual hash transition, no matter what that looks like. Differential Revision: https://phab.mercurial-scm.org/D8367

revset: mark `extinct()` experimental Differential Revision: https://phab.mercurial-scm.org/D8407

revset: mark `successors()` experimental Differential Revision: https://phab.mercurial-scm.org/D8406

revset: mark `obsolete()` experimental I assume not marking it was unintentional, since the obsolete concept it still experimental. Differential Revision: https://phab.mercurial-scm.org/D8405

nodemap: access the mmap opener option using bytes The key is bytes and need to be accessed s such. Differential Revision: https://phab.mercurial-scm.org/D8413

nodemap: rename a variable to prevent shawoding "_" This will make pyflakes happy with the next changeset. Differential Revision: https://phab.mercurial-scm.org/D8412

rust-chg: indent process_message() to prepare mass rewrite to futures-0.3 I'll start upgrading the codebase to modern async/await-based implementation, which cannot be done incrementally. This is the last non-breaking patch to prepare for the rewrite. Differential Revision: https://phab.mercurial-scm.org/D8403

rust-chg: move get_umask() call out of run() function run() will be an async function, but get_umask() isn't thread safe. Differential Revision: https://phab.mercurial-scm.org/D8402

rust-chg: leverage impl trait at argument position Differential Revision: https://phab.mercurial-scm.org/D8401

setup: link osutil.so to libsocket on Solaris/illumos (issue6299)

revset: import `unstable()` from the evolve extension Differential Revision: https://phab.mercurial-scm.org/D8404

fastexport: make a diagnostics message more localizable Differential Revision: https://phab.mercurial-scm.org/D8398

rust-chg: use "crate::" to import local modules I feel it's easier to follow to resolve modules from the crate root than relative path from self module. Differential Revision: https://phab.mercurial-scm.org/D8400

rust-chg: upgrade to 2018 edition and remove useless extern crates Differential Revision: https://phab.mercurial-scm.org/D8399

mergestate: store about files resolved in favour of other Committing a merge sometimes wrongly creates a new filenode where it can re-use an existing one. This happens because the commit code does it's own calculation and does not know what happened on merge. This starts storing information in mergestate about files which were automatically merged and the other/remote version of file was used. We need this information at commit to pick the filenode parent for the new commit. This issue was found by Pierre-Yves David and idea to store the relevant parts in mergestate is also suggested by him. Somethings which can be further investigated are: 1) refactoring of commit logic more to depend on this information 2) maybe a more generic solution? Differential Revision: https://phab.mercurial-scm.org/D8392

histedit: add test that an invalid editor doesn't bork client state We had a report of a situation like this borking a user at Google, but I can't reproduce it in a test. Let's at least backstop the issue with a test, so we don't accidentally introduce such a bug... Differential Revision: https://phab.mercurial-scm.org/D8391

rust-chg: silence warning about dated coding style Differential Revision: https://phab.mercurial-scm.org/D8384

rust-chg: update name of the server process This is a copy of updateprocname() of hgclient.c. At this point, rust-chg is basically functional. I did dogfooding for a couple of weeks in 2018. There are a few remaining tasks: a. loop detection by CHGINTERNALMARK b. forward unsupported commands (notably serve -d) to real hg c. better handling of early server exception d. modernize codebase (2018 edition, impl trait, async/await) For (d), we'll probably want to switch to async-std, but I'm thinking of upgrading to Tokio 0.2 as an intermediate step since process API isn't ported to async-std yet. I'm pretty sure future migration to async-std will be painless compared to the mass rewrite from futures-0.1 to 0.3. https://github.com/async-rs/async-std/issues/22 Differential Revision: https://phab.mercurial-scm.org/D8383

rust-chg: send client side umask to server This is equivalent to forwardumask() of hgclient.c. Differential Revision: https://phab.mercurial-scm.org/D8382

rust-chg: add config validation and process returned instructions This is the reimplementation of runinstructions() and main() in chg.c. In Rust version, we only pass in early arguments to the server as the locator doesn't know the full arguments. This should be fine since these arguments are just passed in to _earlyparseopts() and _parseconfig(), which means the server doesn't need full arguments. Another difference is the handling of the "exit <code>" instruction. In Rust version, we can simply reuse the connection instead of "exit(code)" as the command error isn't displayed yet. That's because the client-side stdio is not attached until the connection is validated. This behavior is cleaner than C, but it also means that the early server exception wouldn't be propagated to client because stderr isn't attached. So we might have to reconsider when to attach/detach the server stdio. Differential Revision: https://phab.mercurial-scm.org/D8381

rust-chg: collect server flags from command arguments This is the reimplementation of testsensitiveflag() and setcmdserverargs() of chg.c. Differential Revision: https://phab.mercurial-scm.org/D8380

rust-chg: add interface to run "validate" request Differential Revision: https://phab.mercurial-scm.org/D8379

rust-chg: add helper to parse instructions sent from server This is well structured version of runinstructions() of chg.c. Differential Revision: https://phab.mercurial-scm.org/D8378

tests: move verification closer to setup in test-copies-chain-merge.t I found this test case really hard to read because it requires scrolling back and forth between the setup and the verification. This patch moves them closer together. There was also duplicated descriptions for each test case (duplicated between setup and verification). They had already become out of sync. I picked the description from the verification, partly so the differences would be highlighted in this patch. Differential Revision: https://phab.mercurial-scm.org/D8377

tests: collect all branch creation in one place in test-copies-chain-merge.t I found this test case really hard to read because it requires scrolling back and forth between the setup and the verification. The next patch will move the verification close to the merge commits they test. This patch prepares for that by moving all branch creation first so they are separate from the merge commits (because many merge commits reuse the same branches). Differential Revision: https://phab.mercurial-scm.org/D8376

hgk: remove a "b" used on a kwargs expansion, the keys are strs Differential Revision: https://phab.mercurial-scm.org/D8385

rust-chg: send client-side environment variables to server This is also needed to run config validation. The "validate" request and its response handling will be implemented in the next batch. Differential Revision: https://phab.mercurial-scm.org/D8364

rust-chg: add helper to pack environment variables On my machine, "printenv | wc -c" says 3422. That's the only reason why the initial buffer capacity is set to 4kB. Differential Revision: https://phab.mercurial-scm.org/D8363

rust-chg: move set_current_dir() to Locator This is necessary to run config validation in proper environment. Differential Revision: https://phab.mercurial-scm.org/D8362

rust-chg: abort if server doesn't have required capabilities It's checked at Locator::connect() since we'll have to do more stuff in this function. Differential Revision: https://phab.mercurial-scm.org/D8361

rust-chg: spawn server process if not running This is the minimal reimplementation of gethgcmd(), execcmdserver(), retryconnectcmdserver(), and connectcmdserver() in chg.c. No config validation is implemented yet. And some Py3 workarounds would be missing as this is the code I wrote in 2018. Differential Revision: https://phab.mercurial-scm.org/D8360

rust-chg: update dependencies Differential Revision: https://phab.mercurial-scm.org/D8359

rust-chg: depend on released version of tokio-process Still it is futures-0.1 based. Differential Revision: https://phab.mercurial-scm.org/D8358

rust-chg: add brief comment about initial capacity of temp_sock_path() I don't know if it can be expressed as a compile-time constant, so it's a comment for now. About this series: This is quite old patches for rust-chg. I heard from Octobus people that there's a plan to do an experiment on merging hgcli + chg + some Rust?, so I decided to respin the rust-chg series. Maybe we'll rewrite the core to leverage the recent async/await functionality, but I want to first make my old patches in so the rust-chg can be a drop-in replacement for the chg of C. Compiler warnings will be removed later, and the codebase will be upgraded to the 2018 edition later. Differential Revision: https://phab.mercurial-scm.org/D8357

discovery: avoid wrongly saying there are nothing to pull We can get in a situation where a revision passed through `hg pull --rev REV` are available on the server, but not a descendant of the advertised server heads. For example the server could lying be during heads advertisement, to hide some pull request. Or obsolete/hidden content could be explicitly pulled. So in this case the lookup associated to `REV` returned successfully, but the normal discovery will find all advertised heads already known locally. This flip a special boolean `anyinc` that will prevent any fetch attempt, preventing `REV` to be pulled over. We add three line of code to detect this case and make sure a pull actually happens. My main target is to make some third party extensions happy (I expect the associated test to move upstream with the extension). However this fix already make some of the `infinitepush` test happier.