rust-dirstate: add rust implementation of `parse_dirstate` and `pack_dirstate` Working towards the goal of having a complete Rust implementation of `hg status`, these two utils are a first step of many to be taken to improve performance and code maintainability. Two dependencies have been added: `memchr` and `byteorder`. Both of them have been written by reputable community members and are very mature crates. The Rust code will often need to use their byte-oriented functions. A few unit tests have been added and may help future development and debugging. In a future patch that uses `parse_dirstate` to stat the working tree in parallel - which neither the Python nor the C implementations do - actual performance improvements will be seen for larger repositories. Differential Revision: https://phab.mercurial-scm.org/D6348

changelog: define changelogrevision.p[12]copies for null revision Looks like I missed these in 5382d8f8530b (changelog: parse copy metadata if available in extras, 2017-12-27). `hg debugp[12]copies -r null` fails before this patch. Differential Revision: https://phab.mercurial-scm.org/D6376

copies: write empty entries in changeset when also writing to filelog When writing to both changeset and filelog (during transition), we don't want the reader to waste time by falling back to reading from the filelog when there is no copy metadata. Let's write out empty copy metadata instead (the read path is already prepared for this case). Thanks to Greg for pointing this out. Differential Revision: https://phab.mercurial-scm.org/D6306

rebase: hide help for revisions.Predicates._destautoorphanrebase

unshelve: add space to help

context: default to using branch from dirstate only in workingctx Same reasoning as previous commits: only the workingctx should know about the dirstate. committablectx now seems free of dirstate references. Differential Revision: https://phab.mercurial-scm.org/D6374

context: let caller pass in branch to committablectx.__init__() committablectx.__init__() currently looks up the branch from the dirstate unless it's passed in the extras. memctx.__init__() has a branch argument, but since committablectx.__init__() doesn't accept it, it lets that constructor look up the branch from the dirstate before it overwrites it, which seems awkward. Differential Revision: https://phab.mercurial-scm.org/D6366

context: move contents of committablectx.markcommitted() to workingctx Same reasoning as previous commits: this function updates the dirstate. By not updating the dirstate here, we also fix the close-head test. Differential Revision: https://phab.mercurial-scm.org/D6365

tests: demonstrate that close-head command updates working copy The help text for the command says "...it doesn't change the working directory", so I don't think this is intentional. Differential Revision: https://phab.mercurial-scm.org/D6364

context: move walk() and match() overrides from committablectx to workingctx Same reasoning as previous commit: these functions update the dirstate. Differential Revision: https://phab.mercurial-scm.org/D6363

context: move flags overrides from committablectx to workingctx These read from the dirstate, so they shouldn't be used in other subclasses. Differential Revision: https://phab.mercurial-scm.org/D6362

context: reuse changectx._copies() in all but workingctx This moves the dirstate-specific _copies() implementation from committablectx into workingctx where it should be (I think all dirstate-specific stuff should be moved into workingctx). The part of changectx._copies() that is for producing changeset-wide copy dicts from the filectxs is moved into basectx so it's reused by the other subclasses. The part of changectx._copies() that's about reading copy information from the changeset remains there. This fixes in-memory rebase (and makes `hg convert` able to write copies to changesets). Differential Revision: https://phab.mercurial-scm.org/D6219

overlayworkingctx: don't include added-then-deleted files in memctx If a file (such as a .orig file) is temporarily added to the overlayworkingctx and then deleted, it's still going to be in the _cache dict. In tomemctx(), we created the list of files from _cache.keys(), so the memctx.files() would include the temporary file. That was fine because the list of files was only used in localrepo.commitctx() (I think), where there's an extra filtering of incorrectly removed files (annotated with an inaccurate "update manifest" comment). I'd like to call memctx.files() in another case, but first we need to make it accurate. Differential Revision: https://phab.mercurial-scm.org/D6361

tests: demonstrate loss of changeset copy metadata on rebase Differential Revision: https://phab.mercurial-scm.org/D6360

overlaycontext: allow calling copydata() on clean context We should just report no copy if the context is clean. Differential Revision: https://phab.mercurial-scm.org/D6358

tests: demonstrate another failure with in-memory rebase and copies This is a similar to dd1ab72be983 (test: demonstrate crash with in-memory rebase and copies, 2019-03-14). The new failure started with 57203e0210f8 (copies: calculate mergecopies() based on pathcopies(), 2019-04-11). It happens in the call to mergemod.update() on rebase.py:1268 where we call mergemod.update() to graft a node. Since the mergecopies() rewrite, that calls _related() with the filectx from the overlaywctx instead of a filectx from the changectx where the file was last modified. Either should be fine, so I don't think that's a bug. Differential Revision: https://phab.mercurial-scm.org/D6357

commit: fix a typo ("form p1" -> "from p1") Differential Revision: https://phab.mercurial-scm.org/D6375

automation: initial support for running Linux tests Building on top of our Windows automation support, this commit implements support for performing automated tasks on remote Linux machines. Specifically, we implement support for running tests on ephemeral EC2 instances. This seems to be a worthwhile place to start, as building packages on Linux is more or less a solved problem because we already have facilities for building in Docker containers, which provide "good enough" reproducibility guarantees. The new `run-tests-linux` command works similarly to `run-tests-windows`: it ensures an AMI with hg dependencies is available, provisions a temporary EC2 instance with this AMI, pushes local changes to that instance via SSH, then invokes `run-tests.py`. Using this new command, I am able to run the entire test harness substantially faster then I am on my local machine courtesy of access to massive core EC2 instances: wall: 16:20 ./run-tests.py -l (i7-6700K) wall: 14:00 automation.py run-tests-linux --ec2-instance c5.2xlarge wall: 8:30 automation.py run-tests-linux --ec2-instance m5.4xlarge wall: 8:04 automation.py run-tests-linux --ec2-instance c5.4xlarge wall: 4:30 automation.py run-tests-linux --ec2-instance c5.9xlarge wall: 3:57 automation.py run-tests-linux --ec2-instance m5.12xlarge wall: 3:05 automation.py run-tests-linux --ec2-instance m5.24xlarge wall: 3:02 automation.py run-tests-linux --ec2-instance c5.18xlarge ~3 minute wall time to run pretty much the entire test harness is not too bad! The AMIs install multiple versions of Python. And the run-tests-linux command specifies which one to use: automation.py run-tests-linux --python system3 automation.py run-tests-linux --python 3.5 automation.py run-tests-linux --python pypy2.7 By default, the system Python 2.7 is used. Using this functionality, I was able to identity some unexpected test failures on PyPy! Included in the feature is support for running with alternate filesystems. You can simply pass --filesystem to the command to specify the type of filesystem to run tests on. When the ephemeral instance is started, a new filesystem will be created and tests will run from it: wall: 4:30 automation.py run-tests-linux --ec2-instance c5.9xlarge wall: 4:20 automation.py run-tests-linux --ec2-instance c5d.9xlarge --filesystem xfs wall: 4:24 automation.py run-tests-linux --ec2-instance c5d.9xlarge --filesystem tmpfs wall: 4:26 automation.py run-tests-linux --ec2-instance c5d.9xlarge --filesystem ext4 We also support multiple Linux distributions: $ automation.py run-tests-linux --distro debian9 total time: 298.1s; setup: 60.7s; tests: 237.5s; setup overhead: 20.4% $ automation.py run-tests-linux --distro ubuntu18.04 total time: 286.1s; setup: 61.3s; tests: 224.7s; setup overhead: 21.4% $ automation.py run-tests-linux --distro ubuntu18.10 total time: 278.5s; setup: 58.2s; tests: 220.3s; setup overhead: 20.9% $ automation.py run-tests-linux --distro ubuntu19.04 total time: 265.8s; setup: 42.5s; tests: 223.3s; setup overhead: 16.0% Debian and Ubuntu are supported because those are what I use and am most familiar with. It should be easy enough to add support for other distros. Unlike the Windows AMIs, Linux EC2 instances bill per second. So the cost to instantiating an ephemeral instance isn't as severe. That being said, there is some overhead, as it takes several dozen seconds for the instance to boot, push local changes, and build Mercurial. During this time, the instance is largely CPU idle and wasting money. Even with this inefficiency, running tests is relatively cheap: $0.15-$0.25 per full test run. A machine running tests as efficiently as these EC2 instances would cost say $6,000, so you can run the test harness a >20,000 times for the cost of an equivalent machine. Running tests in EC2 is almost certainly cheaper than buying a beefy machine for developers to use :) # no-check-commit because foo_bar function names Differential Revision: https://phab.mercurial-scm.org/D6319

automation: move image operations to own functions An upcoming commit will need this functionality with slightly different values and it is enough code to not want to duplicate. Let's refactor into standalone functions so it can be reused. Differential Revision: https://phab.mercurial-scm.org/D6318

automation: add --version argument to build-all-windows-packages This lets us pass a version string through when building all Windows packages, just like we can do with the individual commands which produce installers. Differential Revision: https://phab.mercurial-scm.org/D6317

automation: do a force push to synchronize We don't know what the state of the remote is. Force pushing will be more resilient. Differential Revision: https://phab.mercurial-scm.org/D6316

automation: add check that hg source directory is a repo Synchronizing from e.g. source distributions is not yet supported. Let's add a check so we fail with an error message indicating such. Differential Revision: https://phab.mercurial-scm.org/D6315

automation: shore up rebooting behavior There was a race condition in the old code. Use instance.stop()/instance.start() to eliminate it. As part of debugging this, I also found another race condition related to PowerShell permissions after the reboot. Unfortunately, I'm not sure the best way to work around it. I've added a comment for now. Differential Revision: https://phab.mercurial-scm.org/D6288

automation: wait longer for WinRM connection I got a few timeouts waiting for only 120s for the WinRM connection to become available. Increasing to 180s seems to fix. I guess AWS isn't as consistent as I would like :( Differential Revision: https://phab.mercurial-scm.org/D6287