Augie Fackler <augie@google.com> [Mon, 20 May 2019 11:40:47 -0400] rev 42371
merge with stable
Yuya Nishihara <yuya@tcha.org> [Mon, 20 May 2019 08:40:54 +0900] rev 42370
templatekw: change default value of 'requires' to ()
Since we dropped support for the old-style template keywords, we no longer
have to distinguish None (old-style) and an empty requirement (new-style).
Martin von Zweigbergk <martinvonz@google.com> [Tue, 14 May 2019 16:30:38 -0700] rev 42369
commit: move sorting of added and removed files list to lower level
localrepo.commitctx() has lists of all changed files, as well as lists
of added and removed files. The list of all files is unsorted and
changelog.add() will sort it. Let's also sort the lists of added and
removed files at a lower level (manifestrevlog.add()) for
consistency. It also seems safer to do it there, just before we write
them to the store. That way other callers won't be able to create
invalid commits (or whatever the consequence is) by passing in
unsorted lists. Also, alternative storages may not care that the lists
are sorted. I don't think this will be a performance problem (someone
should have fixed the sorting in changelog.add() if it were).
Differential Revision: https://phab.mercurial-scm.org/D6390
Martin von Zweigbergk <martinvonz@google.com> [Wed, 24 Apr 2019 09:39:40 -0700] rev 42368
match: drop unnecessary adding of '' to set of dirs
This breaks some tests for "rootfilesin:" in a pattern matcher even
more, but that just shows how broken that case is.
Differential Revision: https://phab.mercurial-scm.org/D6406
Martin von Zweigbergk <martinvonz@google.com> [Mon, 22 Apr 2019 22:43:00 -0700] rev 42367
narrowcommands: drop unnecessary adding of '' for root directory
It's now included by util.dirs().
Differential Revision: https://phab.mercurial-scm.org/D6405
Martin von Zweigbergk <martinvonz@google.com> [Wed, 17 Apr 2019 21:39:18 -0700] rev 42366
copies: remove hack for adding root dir to util.dirs object
Differential Revision: https://phab.mercurial-scm.org/D6404
Martin von Zweigbergk <martinvonz@google.com> [Tue, 16 May 2017 11:00:38 -0700] rev 42365
util: make util.dirs() and util.finddirs() include root directory (API)
This changes the behavior of test-origbackup-conflict.t so it no
longer errors out when the backup path points to an existing
file. Instead, it replaces the file by a directory. That seems
reasonable to me.
Differential Revision: https://phab.mercurial-scm.org/D6403
Martin von Zweigbergk <martinvonz@google.com> [Thu, 13 Jul 2017 23:43:16 -0700] rev 42364
dirstate: drop workaround for '.' matching root directory
The check was added in 31abcae33b4f (dirstate: do not ignore current
directory '.' (issue 1078), 2008-04-05) to fix issue1078. Funnily
enough, comment #2 on that issue mentions using '' instead of '.' to
represent the root directory, just like my previous patch did.
test-hgignore.t fails with this patch without the previous patch.
Differential Revision: https://phab.mercurial-scm.org/D6402
Martin von Zweigbergk <martinvonz@google.com> [Mon, 15 May 2017 00:12:19 -0700] rev 42363
match: use '' instead of '.' for root directory (API)
I think '' is generally a better value for the root directory than '.'
is. For example, os.path.join('', 'foo') => 'foo', while
os.path.join('.', 'foo') => './foo'.
This patch mostly makes it so we use '' internally in
match.py. However, it also affects the API in visitdir(),
visitchildrenset() and files(). The two former now also accept '' as
input. I've updated the callers of these methods. I've also added a
deprecation warning for passing '.' (for external callers). The only
caller I could find that was affected by files() returning '' instead
of '.' was in dirstate.walk(). I've updated that.
The next few patches show some workarounds we can remove by using ''
instead of '.'.
Differential Revision: https://phab.mercurial-scm.org/D6401
Martin von Zweigbergk <martinvonz@google.com> [Wed, 24 Apr 2019 09:32:29 -0700] rev 42362
dirstate: move special handling of files==['.'] together
I think it makes it a little clearer to have the two conditions for
files==['.'] near each other.
Differential Revision: https://phab.mercurial-scm.org/D6400
Martin von Zweigbergk <martinvonz@google.com> [Fri, 17 May 2019 00:57:57 -0700] rev 42361
convert: don't include file in "files" list if it's added in p2
If the file is from p2, we should clearly compare the flags to what
they were in p2.
Also note that manifest.flags('non-existent') unfortunately returns ''
instead of erroring out.
Differential Revision: https://phab.mercurial-scm.org/D6409
Martin von Zweigbergk <martinvonz@google.com> [Fri, 17 May 2019 11:32:48 -0700] rev 42360
convert: demonstrate broken {files} list in merge commits with file flags
When there is a merge in which the flags for a file from p2 is
non-empty, `hg convert` will incorrectly include that in the
changeset's files list.
Differential Revision: https://phab.mercurial-scm.org/D6408
Matt Harbison <matt_harbison@yahoo.com> [Sat, 18 May 2019 19:56:06 -0400] rev 42359
templater: drop support for old style keywords (API)
These changes originated from several commits over a period of time, so I'm
slightly unsure if this is correct. But the tests pass.
Matt Harbison <matt_harbison@yahoo.com> [Sat, 18 May 2019 19:38:47 -0400] rev 42358
commands: drop support for legacy ^cmd registration (API)
Matt Harbison <matt_harbison@yahoo.com> [Sat, 18 May 2019 19:33:48 -0400] rev 42357
extensions: drop support for extsetup() without `ui` argument (API)
Martin von Zweigbergk <martinvonz@google.com> [Fri, 17 May 2019 11:11:40 -0700] rev 42356
relnotes: mention removed support for mixed log graph lines
This adds release notes for 264a2cbb25d0 (graphmod: remove support for
graph lines mixing parent/grandparent styles (BC), 2018-10-16).
Differential Revision: https://phab.mercurial-scm.org/D6407
Augie Fackler <augie@google.com> [Fri, 17 May 2019 11:03:47 -0400] rev 42355
tests: fix test-clonebundles on recent openbsd
I guess openbsd feels like it needs to stringify this errno in
lowercase and omit the "host" part of "hostname. Okay.
Reported in a big test diff talking about libressl, see 6122. I'm not
flagging this because most of that issue is about a libressl string
change, so this doesn't really make a big difference there.
Differential Revision: https://phab.mercurial-scm.org/D6399
Georges Racinet <georges.racinet@octobus.net> [Thu, 16 May 2019 21:17:14 +0200] rev 42354
rust-python3: compatibility fix for integer conversion
On python3, `to_py_object()` on the usize gives us a PyLong,
whereas it is the generic `PyObject` already on python2, which fits
the `py.None()` default value.
Upcasting to `PyObject` explicitely in all cases solves the issue.
Differential Revision: https://phab.mercurial-scm.org/D6396
Augie Fackler <augie@google.com> [Fri, 17 May 2019 09:42:02 -0400] rev 42353
rust: sort dependencies entries in Cargo.toml
I should probably write a test to enforce this...
Differential Revision: https://phab.mercurial-scm.org/D6398
Pulkit Goyal <7895pulkit@gmail.com> [Fri, 17 May 2019 00:04:29 +0530] rev 42352
py3: make contrib/testparseutil.py to work on str(unicodes)
contrib/check-code work on unicodes and call functions from testparseutil.py
which before this patch used to work on bytes.
This path removes that inconsistency and make testparseutil.py work on unicodes.
This makes test-check-code.t and test-contrib-check-code.t work on Python 3
again.
Differential Revision: https://phab.mercurial-scm.org/D6391
Raphaël Gomès <rgomes@octobus.net> [Fri, 17 May 2019 09:36:29 -0400] rev 42351
rust-filepatterns: call new Rust implementations from Python
This change adds the import to the `rust-cpython` bindings and uses
them when appropriate.
A wrapper function has been defined in the case of `_regex` to
keep this patch simple.
Differential Revision: https://phab.mercurial-scm.org/D6273
Raphaël Gomès <rgomes@octobus.net> [Fri, 17 May 2019 09:36:29 -0400] rev 42350
rust-filepatterns: add `rust-cpython` bindings for `filepatterns`
This change adds the `rust-cpython` interface for top-level functions and
exceptions in the filepatterns module.
Contrary to the Python implementation, this tries to have finer-grained
exceptions to allow for better readability and flow control down the line.
Differential Revision: https://phab.mercurial-scm.org/D6272
Raphaël Gomès <rgomes@octobus.net> [Wed, 24 Apr 2019 11:34:09 +0200] rev 42349
rust-filepatterns: add a Rust implementation of pattern-related utils
This change introduces Rust implementations of two functions related to
pattern handling, all located in `match.py`:
- `_regex`
- `readpatternfile`
These utils are useful in the long-term effort to improve `hg status`'s
performance using Rust. Experimental work done by Valentin Gatien-Baron
shows very promising improvements, but is too different from the current
Mercurial core code structure to be used "as-is".
This is the first - albeit very small - step towards the code revamp
needed down the line.
Two dependencies were added: `regex` and `lazy_static`. Both of them
will be useful for a majority of the Rust code that will be written,
are well known and maintained either by the Rust core team, or by
very frequent contributors.
Differential Revision: https://phab.mercurial-scm.org/D6271
Martin von Zweigbergk <martinvonz@google.com> [Wed, 15 May 2019 22:11:41 -0700] rev 42348
exchange: don't take wlock if bookmarks are stored in .hg/store/
If bookmarks are stored in .hg/store/, there is no need for the
wlock().
Differential Revision: https://phab.mercurial-scm.org/D6388
Martin von Zweigbergk <martinvonz@google.com> [Wed, 15 May 2019 22:09:02 -0700] rev 42347
bookmarks: keep bookmarks in .hg/store if new config set
Bookmarks storage consists of two parts: (1) the set of bookmarks and
their positions, and (2) the current bookmark. The former can get
updated by exchange, while the latter cannot. However, they are both
stored in directly .hg/ and protected by repo.wlock(). As a result,
ugly workarounds were needed. This patch introduces a new config
option to store the set of bookmarks and their positions in .hg/store/
but still storing the current bookmark directory in .hg/. The config
option only takes effect at repo creation time. It results in a new
requirement being set.
Differential Revision: https://phab.mercurial-scm.org/D6387
Yuya Nishihara <yuya@tcha.org> [Thu, 16 May 2019 08:15:20 +0900] rev 42346
log: flag topo-sorted set as such
This isn't required right now, but revs.istopo() should return True.
Martin von Zweigbergk <martinvonz@google.com> [Wed, 09 Jan 2019 15:54:45 -0800] rev 42345
copies: fix duplicatecopies() with overlay context
The reasoning for this check is in 78d760aa3607 (duplicatecopies: do
not mark items not in the dirstate as copies, 2013-03-28). The check
was then moved to workingfilectx in 754b5117622f (context: add
workingfilectx.markcopied, 2017-10-15) and no corresponding check was
added later when overlayworkingfilectx was added. Rather than adding
the check there, this patch adds a more generic check on the callers
side and removes the check in workingfilectx.markcopied().
Differential Revision: https://phab.mercurial-scm.org/D6380
Martin von Zweigbergk <martinvonz@google.com> [Wed, 15 May 2019 16:10:52 -0700] rev 42344
tests: demonstrate crash when rebasing across copy with --collapse
As reported by timeless.
Differential Revision: https://phab.mercurial-scm.org/D6379
Augie Fackler <augie@google.com> [Wed, 15 May 2019 17:18:57 -0400] rev 42343
exthelper: add some semi-useful trace logs
It'd be nice to make the trace functions a little better-named in the output,
but I'm not sure how much better we can do without overhead. This at least
lets you see if a single reposetup function is eating all the time or if it's
spread over all of them. I needed this because Google's uber-extension has a
long load time and I wasn't sure where the problem was.
Differential Revision: https://phab.mercurial-scm.org/D6381
Martin von Zweigbergk <martinvonz@google.com> [Wed, 15 May 2019 23:26:05 -0700] rev 42342
help: add missing blank line, making "revlog-compression" show up
Differential Revision: https://phab.mercurial-scm.org/D6386
Martin von Zweigbergk <martinvonz@google.com> [Wed, 15 May 2019 11:53:22 -0700] rev 42341
tests: fix share test to actually share the repo
"repo2" is clearly meant to be a share from "repo1" but without
sharing bookmarks. However, `hg unshare` was called in the repo, so it
had become completely unrelated and thus not testing what it was
supposed to test.
Differential Revision: https://phab.mercurial-scm.org/D6385
Martin von Zweigbergk <martinvonz@google.com> [Wed, 15 May 2019 11:38:45 -0700] rev 42340
tests: separate out bookmarks tests from test-share.t
Differential Revision: https://phab.mercurial-scm.org/D6384
Martin von Zweigbergk <martinvonz@google.com> [Wed, 15 May 2019 10:19:36 -0700] rev 42339
bookmarks: use vfs.tryread() instead of reimplementing it
Differential Revision: https://phab.mercurial-scm.org/D6383
Martin von Zweigbergk <martinvonz@google.com> [Wed, 15 May 2019 10:13:29 -0700] rev 42338
bookmarks: use context manager when writing files
Differential Revision: https://phab.mercurial-scm.org/D6382
timeless <timeless@mozdev.org> [Wed, 15 May 2019 10:54:36 -0400] rev 42337
bisect: do not crash with rewritten commits
Martin von Zweigbergk <martinvonz@google.com> [Wed, 01 May 2019 09:34:47 -0700] rev 42336
log: add config for making `hg log -G` always topo-sorted
I (and everyone else at Google) have an log alias that adds graph mode
and templating. I have another one that builds on the first and also
restricts the set of revisions to only show those I'm most likely to
care about. This second alias also adds topological sorting. I still
sometimes use the first one. When I do, it very often bothers me that
it's not topologically sorted (branches are interleaved). This patch
adds a config option for always using topological sorting with graph
log.
The revision set is sorted eagerly, which seems like a bad idea, but
it doesn't seem to make a big difference in the hg repo (150ms). I
initially tried to instead wrap the user's revset in sort(...,topo),
but that seemed much harder.
Differential Revision: https://phab.mercurial-scm.org/D6331
Martin von Zweigbergk <martinvonz@google.com> [Tue, 14 May 2019 09:13:39 -0700] rev 42335
log: remove an unnecessary "and opts.get('rev')" condition
As Yuya pointed out, the condition is unnecessary since
revs.isdescending() would be true if --follow without --rev.
Differential Revision: https://phab.mercurial-scm.org/D6372
Kyle Lippincott <spectral@google.com> [Tue, 16 Oct 2018 04:59:36 -0700] rev 42334
graphmod: remove support for graph lines mixing parent/grandparent styles (BC)
Currently, if the configuration for a graph edge draw style has multiple bytes
(at least on python2), it is interpreted as "this is a request to draw the line
partially in the style of the parent, partially in the style of the
grandparent". This precludes the configuration handling unicode characters
(which trigger the `len > 1` check, at least on python2), and I believe was part
of the reason that beautifygraph was written the way it was.
Talking with the person who implemented this, it appears to have been to achieve
feature parity with the rendering of the smartlog extension. I suspect that this
isn't actually used outside of that situation, so I think that we can remove it
without much issue.
This will make it so that multi-character edges are possible, and render any
existing configuration that uses this feature with these multiple characters.
This is *not* going to adjust the width of everything to make it line up
correctly, please see the test that's being modified in this changeset for an
example of how the previous configuration now renders.
Note also that the previous configuration seems to have been broken, or at least
it was behaving in a really non-obvious way - it was avoiding the grandparent
character(s) when it should have been displaying them! This is why so many "!"
characters changed to "3."; I don't know if this was intentional.
Differential Revision: https://phab.mercurial-scm.org/D5112
Pulkit Goyal <7895pulkit@gmail.com> [Wed, 15 May 2019 21:02:32 +0300] rev 42333
py3: add 5 new passing tests
Differential Revision: https://phab.mercurial-scm.org/D6378
Pulkit Goyal <7895pulkit@gmail.com> [Wed, 15 May 2019 20:37:39 +0300] rev 42332
py3: add a r'' to prevent transformer adding b''
# skip-blame because just r'' prefix
Differential Revision: https://phab.mercurial-scm.org/D6377
Raphaël Gomès <rgomes@octobus.net> [Mon, 06 May 2019 22:51:10 +0200] rev 42331
rust-dirstate: call parse/pack bindings from Python
A future patch will need to address the issue of Rust module policy,
to avoid having ugly duplicate imports and conditionals all over the place.
As the rewrite of dirstate in Rust progresses, we will need fewer of those
"contact points".
Differential Revision: https://phab.mercurial-scm.org/D6350
Raphaël Gomès <rgomes@octobus.net> [Mon, 06 May 2019 22:50:34 +0200] rev 42330
rust-dirstate: add rust-cpython bindings to the new parse/pack functions
This allows for Python code to call `parse/pack_dirstate` transparently.
These bindings are heavy given the relatively simple task, as they are bound
to implementation details of both the C and Python code. They will be slimmed
down in future patches and eventually completely removed once more of the
dirstate code has been refactored/rewritten in Rust.
Both functions emulate the mutate-on-loop style of the Python and C
implementations by looping over changed items in the compatibility layer,
instead of at the core functions.
Differential Revision: https://phab.mercurial-scm.org/D6349
Raphaël Gomès <rgomes@octobus.net> [Mon, 06 May 2019 22:48:09 +0200] rev 42329
rust-dirstate: add rust implementation of `parse_dirstate` and `pack_dirstate`
Working towards the goal of having a complete Rust implementation of
`hg status`, these two utils are a first step of many to be taken
to improve performance and code maintainability.
Two dependencies have been added: `memchr` and `byteorder`.
Both of them have been written by reputable community members and are
very mature crates.
The Rust code will often need to use their byte-oriented functions.
A few unit tests have been added and may help future development and debugging.
In a future patch that uses `parse_dirstate` to stat the working tree in
parallel - which neither the Python nor the C implementations do - actual
performance improvements will be seen for larger repositories.
Differential Revision: https://phab.mercurial-scm.org/D6348
Martin von Zweigbergk <martinvonz@google.com> [Tue, 14 May 2019 22:56:58 -0700] rev 42328
changelog: define changelogrevision.p[12]copies for null revision
Looks like I missed these in 5382d8f8530b (changelog: parse copy
metadata if available in extras, 2017-12-27). `hg debugp[12]copies -r
null` fails before this patch.
Differential Revision: https://phab.mercurial-scm.org/D6376
Martin von Zweigbergk <martinvonz@google.com> [Tue, 23 Apr 2019 13:29:13 -0700] rev 42327
copies: write empty entries in changeset when also writing to filelog
When writing to both changeset and filelog (during transition), we
don't want the reader to waste time by falling back to reading from
the filelog when there is no copy metadata. Let's write out empty copy
metadata instead (the read path is already prepared for this
case). Thanks to Greg for pointing this out.
Differential Revision: https://phab.mercurial-scm.org/D6306
timeless <timeless@mozdev.org> [Mon, 13 May 2019 14:19:36 -0400] rev 42326
rebase: hide help for revisions.Predicates._destautoorphanrebase
timeless <timeless@mozdev.org> [Fri, 03 May 2019 16:07:57 -0400] rev 42325
unshelve: add space to help
Martin von Zweigbergk <martinvonz@google.com> [Fri, 10 May 2019 22:24:47 -0700] rev 42324
context: default to using branch from dirstate only in workingctx
Same reasoning as previous commits: only the workingctx should know
about the dirstate.
committablectx now seems free of dirstate references.
Differential Revision: https://phab.mercurial-scm.org/D6374
Martin von Zweigbergk <martinvonz@google.com> [Fri, 10 May 2019 22:51:33 -0700] rev 42323
context: let caller pass in branch to committablectx.__init__()
committablectx.__init__() currently looks up the branch from the
dirstate unless it's passed in the extras. memctx.__init__() has a
branch argument, but since committablectx.__init__() doesn't accept
it, it lets that constructor look up the branch from the dirstate
before it overwrites it, which seems awkward.
Differential Revision: https://phab.mercurial-scm.org/D6366
Martin von Zweigbergk <martinvonz@google.com> [Fri, 10 May 2019 21:55:59 -0700] rev 42322
context: move contents of committablectx.markcommitted() to workingctx
Same reasoning as previous commits: this function updates the
dirstate. By not updating the dirstate here, we also fix the
close-head test.
Differential Revision: https://phab.mercurial-scm.org/D6365
Martin von Zweigbergk <martinvonz@google.com> [Fri, 10 May 2019 22:18:11 -0700] rev 42321
tests: demonstrate that close-head command updates working copy
The help text for the command says "...it doesn't change the working
directory", so I don't think this is intentional.
Differential Revision: https://phab.mercurial-scm.org/D6364
Martin von Zweigbergk <martinvonz@google.com> [Fri, 10 May 2019 21:53:41 -0700] rev 42320
context: move walk() and match() overrides from committablectx to workingctx
Same reasoning as previous commit: these functions update the dirstate.
Differential Revision: https://phab.mercurial-scm.org/D6363
Martin von Zweigbergk <martinvonz@google.com> [Fri, 10 May 2019 21:35:30 -0700] rev 42319
context: move flags overrides from committablectx to workingctx
These read from the dirstate, so they shouldn't be used in other
subclasses.
Differential Revision: https://phab.mercurial-scm.org/D6362
Martin von Zweigbergk <martinvonz@google.com> [Fri, 10 May 2019 13:41:42 -0700] rev 42318
context: reuse changectx._copies() in all but workingctx
This moves the dirstate-specific _copies() implementation from
committablectx into workingctx where it should be (I think all
dirstate-specific stuff should be moved into workingctx). The part of
changectx._copies() that is for producing changeset-wide copy dicts
from the filectxs is moved into basectx so it's reused by the other
subclasses. The part of changectx._copies() that's about reading copy
information from the changeset remains there. This fixes in-memory
rebase (and makes `hg convert` able to write copies to changesets).
Differential Revision: https://phab.mercurial-scm.org/D6219
Martin von Zweigbergk <martinvonz@google.com> [Fri, 10 May 2019 14:27:22 -0700] rev 42317
overlayworkingctx: don't include added-then-deleted files in memctx
If a file (such as a .orig file) is temporarily added to the
overlayworkingctx and then deleted, it's still going to be in the
_cache dict. In tomemctx(), we created the list of files from
_cache.keys(), so the memctx.files() would include the temporary
file. That was fine because the list of files was only used in
localrepo.commitctx() (I think), where there's an extra filtering of
incorrectly removed files (annotated with an inaccurate "update
manifest" comment). I'd like to call memctx.files() in another case,
but first we need to make it accurate.
Differential Revision: https://phab.mercurial-scm.org/D6361
Martin von Zweigbergk <martinvonz@google.com> [Fri, 10 May 2019 10:23:46 -0700] rev 42316
tests: demonstrate loss of changeset copy metadata on rebase
Differential Revision: https://phab.mercurial-scm.org/D6360
Martin von Zweigbergk <martinvonz@google.com> [Fri, 10 May 2019 11:03:54 -0700] rev 42315
overlaycontext: allow calling copydata() on clean context
We should just report no copy if the context is clean.
Differential Revision: https://phab.mercurial-scm.org/D6358
Martin von Zweigbergk <martinvonz@google.com> [Fri, 10 May 2019 10:23:08 -0700] rev 42314
tests: demonstrate another failure with in-memory rebase and copies
This is a similar to dd1ab72be983 (test: demonstrate crash with
in-memory rebase and copies, 2019-03-14). The new failure started with
57203e0210f8 (copies: calculate mergecopies() based on pathcopies(),
2019-04-11). It happens in the call to mergemod.update() on
rebase.py:1268 where we call mergemod.update() to graft a node. Since
the mergecopies() rewrite, that calls _related() with the filectx from
the overlaywctx instead of a filectx from the changectx where the file
was last modified. Either should be fine, so I don't think that's
a bug.
Differential Revision: https://phab.mercurial-scm.org/D6357
Martin von Zweigbergk <martinvonz@google.com> [Tue, 14 May 2019 16:40:49 -0700] rev 42313
commit: fix a typo ("form p1" -> "from p1")
Differential Revision: https://phab.mercurial-scm.org/D6375
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 27 Apr 2019 11:48:26 -0700] rev 42312
automation: initial support for running Linux tests
Building on top of our Windows automation support, this commit
implements support for performing automated tasks on remote Linux
machines. Specifically, we implement support for running tests
on ephemeral EC2 instances. This seems to be a worthwhile place
to start, as building packages on Linux is more or less a solved
problem because we already have facilities for building in Docker
containers, which provide "good enough" reproducibility guarantees.
The new `run-tests-linux` command works similarly to
`run-tests-windows`: it ensures an AMI with hg dependencies is
available, provisions a temporary EC2 instance with this AMI, pushes
local changes to that instance via SSH, then invokes `run-tests.py`.
Using this new command, I am able to run the entire test harness
substantially faster then I am on my local machine courtesy of
access to massive core EC2 instances:
wall: 16:20 ./run-tests.py -l (i7-6700K)
wall: 14:00 automation.py run-tests-linux --ec2-instance c5.2xlarge
wall: 8:30 automation.py run-tests-linux --ec2-instance m5.4xlarge
wall: 8:04 automation.py run-tests-linux --ec2-instance c5.4xlarge
wall: 4:30 automation.py run-tests-linux --ec2-instance c5.9xlarge
wall: 3:57 automation.py run-tests-linux --ec2-instance m5.12xlarge
wall: 3:05 automation.py run-tests-linux --ec2-instance m5.24xlarge
wall: 3:02 automation.py run-tests-linux --ec2-instance c5.18xlarge
~3 minute wall time to run pretty much the entire test harness is
not too bad!
The AMIs install multiple versions of Python. And the run-tests-linux
command specifies which one to use:
automation.py run-tests-linux --python system3
automation.py run-tests-linux --python 3.5
automation.py run-tests-linux --python pypy2.7
By default, the system Python 2.7 is used. Using this functionality,
I was able to identity some unexpected test failures on PyPy!
Included in the feature is support for running with alternate
filesystems. You can simply pass --filesystem to the command to
specify the type of filesystem to run tests on. When the ephemeral
instance is started, a new filesystem will be created and tests
will run from it:
wall: 4:30 automation.py run-tests-linux --ec2-instance c5.9xlarge
wall: 4:20 automation.py run-tests-linux --ec2-instance c5d.9xlarge --filesystem xfs
wall: 4:24 automation.py run-tests-linux --ec2-instance c5d.9xlarge --filesystem tmpfs
wall: 4:26 automation.py run-tests-linux --ec2-instance c5d.9xlarge --filesystem ext4
We also support multiple Linux distributions:
$ automation.py run-tests-linux --distro debian9
total time: 298.1s; setup: 60.7s; tests: 237.5s; setup overhead: 20.4%
$ automation.py run-tests-linux --distro ubuntu18.04
total time: 286.1s; setup: 61.3s; tests: 224.7s; setup overhead: 21.4%
$ automation.py run-tests-linux --distro ubuntu18.10
total time: 278.5s; setup: 58.2s; tests: 220.3s; setup overhead: 20.9%
$ automation.py run-tests-linux --distro ubuntu19.04
total time: 265.8s; setup: 42.5s; tests: 223.3s; setup overhead: 16.0%
Debian and Ubuntu are supported because those are what I use and am
most familiar with. It should be easy enough to add support for other
distros.
Unlike the Windows AMIs, Linux EC2 instances bill per second. So
the cost to instantiating an ephemeral instance isn't as severe.
That being said, there is some overhead, as it takes several dozen
seconds for the instance to boot, push local changes, and build
Mercurial. During this time, the instance is largely CPU idle and
wasting money. Even with this inefficiency, running tests is
relatively cheap: $0.15-$0.25 per full test run. A machine running
tests as efficiently as these EC2 instances would cost say $6,000, so
you can run the test harness a >20,000 times for the cost of an
equivalent machine. Running tests in EC2 is almost certainly cheaper
than buying a beefy machine for developers to use :)
# no-check-commit because foo_bar function names
Differential Revision: https://phab.mercurial-scm.org/D6319