annotate contrib/automation/hgautomation/linux.py @ 48843:d953a42b157d

automation: run hg with python3 Python 2.7 support will go away soon. Let's use Python 3 as part of the automation. Differential Revision: https://phab.mercurial-scm.org/D12260
author Gregory Szorc <gregory.szorc@gmail.com>
date Sat, 19 Feb 2022 18:04:49 -0700
parents 0de5eefbe4b2
children 5c8148cd7f13
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
42285
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
1 # linux.py - Linux specific automation functionality
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
2 #
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
3 # Copyright 2019 Gregory Szorc <gregory.szorc@gmail.com>
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
4 #
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
5 # This software may be used and distributed according to the terms of the
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
6 # GNU General Public License version 2 or any later version.
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
7
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
8 # no-check-code because Python 3 native.
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
9
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
10 import os
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
11 import pathlib
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
12 import shlex
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
13 import subprocess
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
14 import tempfile
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
15
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43018
diff changeset
16 from .ssh import exec_command
42285
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
17
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
18
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
19 # Linux distributions that are supported.
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
20 DISTROS = {
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
21 'debian9',
43018
d1d919f679f7 automation: support and use Debian Buster by default
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43015
diff changeset
22 'debian10',
42285
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
23 'ubuntu18.04',
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
24 'ubuntu19.04',
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
25 }
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
26
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
27 INSTALL_PYTHONS = r'''
43315
388ada1c07c6 automation: install Python 2.7.17, 3.7.5, and PyPy 7.2.0
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43235
diff changeset
28 PYENV2_VERSIONS="2.7.17 pypy2.7-7.2.0"
47196
743522fa2455 automation: install latest Python versions in Linux environment
Gregory Szorc <gregory.szorc@gmail.com>
parents: 47195
diff changeset
29 PYENV3_VERSIONS="3.5.10 3.6.13 3.7.10 3.8.10 3.9.5 pypy3.5-7.0.0 pypy3.6-7.3.3 pypy3.7-7.3.3"
42285
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
30
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
31 git clone https://github.com/pyenv/pyenv.git /hgdev/pyenv
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
32 pushd /hgdev/pyenv
47196
743522fa2455 automation: install latest Python versions in Linux environment
Gregory Szorc <gregory.szorc@gmail.com>
parents: 47195
diff changeset
33 git checkout 328fd42c3a2fbf14ae46dae2021a087fe27ba7e2
42285
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
34 popd
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
35
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
36 export PYENV_ROOT="/hgdev/pyenv"
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
37 export PATH="$PYENV_ROOT/bin:$PATH"
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
38
43014
cbd94ee3a72e automation: upgrade packages in Linux environment
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43013
diff changeset
39 # pip 19.2.3.
cbd94ee3a72e automation: upgrade packages in Linux environment
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43013
diff changeset
40 PIP_SHA256=57e3643ff19f018f8a00dfaa6b7e4620e3c1a7a2171fd218425366ec006b3bfe
cbd94ee3a72e automation: upgrade packages in Linux environment
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43013
diff changeset
41 wget -O get-pip.py --progress dot:mega https://github.com/pypa/get-pip/raw/309a56c5fd94bd1134053a541cb4657a4e47e09d/get-pip.py
42285
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
42 echo "${PIP_SHA256} get-pip.py" | sha256sum --check -
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
43
43014
cbd94ee3a72e automation: upgrade packages in Linux environment
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43013
diff changeset
44 VIRTUALENV_SHA256=f78d81b62d3147396ac33fc9d77579ddc42cc2a98dd9ea38886f616b33bc7fb2
cbd94ee3a72e automation: upgrade packages in Linux environment
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43013
diff changeset
45 VIRTUALENV_TARBALL=virtualenv-16.7.5.tar.gz
cbd94ee3a72e automation: upgrade packages in Linux environment
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43013
diff changeset
46 wget -O ${VIRTUALENV_TARBALL} --progress dot:mega https://files.pythonhosted.org/packages/66/f0/6867af06d2e2f511e4e1d7094ff663acdebc4f15d4a0cb0fed1007395124/${VIRTUALENV_TARBALL}
42285
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
47 echo "${VIRTUALENV_SHA256} ${VIRTUALENV_TARBALL}" | sha256sum --check -
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
48
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
49 for v in ${PYENV2_VERSIONS}; do
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
50 pyenv install -v ${v}
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
51 ${PYENV_ROOT}/versions/${v}/bin/python get-pip.py
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
52 ${PYENV_ROOT}/versions/${v}/bin/pip install ${VIRTUALENV_TARBALL}
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
53 ${PYENV_ROOT}/versions/${v}/bin/pip install -r /hgdev/requirements-py2.txt
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
54 done
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
55
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
56 for v in ${PYENV3_VERSIONS}; do
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
57 pyenv install -v ${v}
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
58 ${PYENV_ROOT}/versions/${v}/bin/python get-pip.py
47195
546e812a1c2d automation: create Python 3.5 variant of requirements.txt
Gregory Szorc <gregory.szorc@gmail.com>
parents: 46353
diff changeset
59
546e812a1c2d automation: create Python 3.5 variant of requirements.txt
Gregory Szorc <gregory.szorc@gmail.com>
parents: 46353
diff changeset
60 case ${v} in
546e812a1c2d automation: create Python 3.5 variant of requirements.txt
Gregory Szorc <gregory.szorc@gmail.com>
parents: 46353
diff changeset
61 3.5.*)
546e812a1c2d automation: create Python 3.5 variant of requirements.txt
Gregory Szorc <gregory.szorc@gmail.com>
parents: 46353
diff changeset
62 REQUIREMENTS=requirements-py3.5.txt
546e812a1c2d automation: create Python 3.5 variant of requirements.txt
Gregory Szorc <gregory.szorc@gmail.com>
parents: 46353
diff changeset
63 ;;
546e812a1c2d automation: create Python 3.5 variant of requirements.txt
Gregory Szorc <gregory.szorc@gmail.com>
parents: 46353
diff changeset
64 pypy3.5*)
546e812a1c2d automation: create Python 3.5 variant of requirements.txt
Gregory Szorc <gregory.szorc@gmail.com>
parents: 46353
diff changeset
65 REQUIREMENTS=requirements-py3.5.txt
546e812a1c2d automation: create Python 3.5 variant of requirements.txt
Gregory Szorc <gregory.szorc@gmail.com>
parents: 46353
diff changeset
66 ;;
546e812a1c2d automation: create Python 3.5 variant of requirements.txt
Gregory Szorc <gregory.szorc@gmail.com>
parents: 46353
diff changeset
67 *)
546e812a1c2d automation: create Python 3.5 variant of requirements.txt
Gregory Szorc <gregory.szorc@gmail.com>
parents: 46353
diff changeset
68 REQUIREMENTS=requirements-py3.txt
546e812a1c2d automation: create Python 3.5 variant of requirements.txt
Gregory Szorc <gregory.szorc@gmail.com>
parents: 46353
diff changeset
69 ;;
546e812a1c2d automation: create Python 3.5 variant of requirements.txt
Gregory Szorc <gregory.szorc@gmail.com>
parents: 46353
diff changeset
70 esac
546e812a1c2d automation: create Python 3.5 variant of requirements.txt
Gregory Szorc <gregory.szorc@gmail.com>
parents: 46353
diff changeset
71
546e812a1c2d automation: create Python 3.5 variant of requirements.txt
Gregory Szorc <gregory.szorc@gmail.com>
parents: 46353
diff changeset
72 ${PYENV_ROOT}/versions/${v}/bin/pip install -r /hgdev/${REQUIREMENTS}
42285
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
73 done
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
74
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
75 pyenv global ${PYENV2_VERSIONS} ${PYENV3_VERSIONS} system
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43018
diff changeset
76 '''.lstrip().replace(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43018
diff changeset
77 '\r\n', '\n'
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43018
diff changeset
78 )
42285
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
79
47202
0de5eefbe4b2 contrib: install PyOxidizer 0.16.0
Gregory Szorc <gregory.szorc@gmail.com>
parents: 47197
diff changeset
80 INSTALL_PYOXIDIZER = r'''
0de5eefbe4b2 contrib: install PyOxidizer 0.16.0
Gregory Szorc <gregory.szorc@gmail.com>
parents: 47197
diff changeset
81 PYOXIDIZER_VERSION=0.16.0
0de5eefbe4b2 contrib: install PyOxidizer 0.16.0
Gregory Szorc <gregory.szorc@gmail.com>
parents: 47197
diff changeset
82 PYOXIDIZER_SHA256=8875471c270312fbb934007fd30f65f1904cc0f5da6188d61c90ed2129b9f9c1
0de5eefbe4b2 contrib: install PyOxidizer 0.16.0
Gregory Szorc <gregory.szorc@gmail.com>
parents: 47197
diff changeset
83 PYOXIDIZER_URL=https://github.com/indygreg/PyOxidizer/releases/download/pyoxidizer%2F${PYOXIDIZER_VERSION}/pyoxidizer-${PYOXIDIZER_VERSION}-linux_x86_64.zip
0de5eefbe4b2 contrib: install PyOxidizer 0.16.0
Gregory Szorc <gregory.szorc@gmail.com>
parents: 47197
diff changeset
84
0de5eefbe4b2 contrib: install PyOxidizer 0.16.0
Gregory Szorc <gregory.szorc@gmail.com>
parents: 47197
diff changeset
85 wget -O pyoxidizer.zip --progress dot:mega ${PYOXIDIZER_URL}
0de5eefbe4b2 contrib: install PyOxidizer 0.16.0
Gregory Szorc <gregory.szorc@gmail.com>
parents: 47197
diff changeset
86 echo "${PYOXIDIZER_SHA256} pyoxidizer.zip" | sha256sum --check -
0de5eefbe4b2 contrib: install PyOxidizer 0.16.0
Gregory Szorc <gregory.szorc@gmail.com>
parents: 47197
diff changeset
87
0de5eefbe4b2 contrib: install PyOxidizer 0.16.0
Gregory Szorc <gregory.szorc@gmail.com>
parents: 47197
diff changeset
88 unzip pyoxidizer.zip
0de5eefbe4b2 contrib: install PyOxidizer 0.16.0
Gregory Szorc <gregory.szorc@gmail.com>
parents: 47197
diff changeset
89 chmod +x pyoxidizer
0de5eefbe4b2 contrib: install PyOxidizer 0.16.0
Gregory Szorc <gregory.szorc@gmail.com>
parents: 47197
diff changeset
90 sudo mv pyoxidizer /usr/local/bin/pyoxidizer
0de5eefbe4b2 contrib: install PyOxidizer 0.16.0
Gregory Szorc <gregory.szorc@gmail.com>
parents: 47197
diff changeset
91 '''
42285
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
92
42696
89ba81771fc9 automation: install Rust in Linux environment
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42694
diff changeset
93 INSTALL_RUST = r'''
89ba81771fc9 automation: install Rust in Linux environment
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42694
diff changeset
94 RUSTUP_INIT_SHA256=a46fe67199b7bcbbde2dcbc23ae08db6f29883e260e23899a88b9073effc9076
89ba81771fc9 automation: install Rust in Linux environment
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42694
diff changeset
95 wget -O rustup-init --progress dot:mega https://static.rust-lang.org/rustup/archive/1.18.3/x86_64-unknown-linux-gnu/rustup-init
89ba81771fc9 automation: install Rust in Linux environment
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42694
diff changeset
96 echo "${RUSTUP_INIT_SHA256} rustup-init" | sha256sum --check -
89ba81771fc9 automation: install Rust in Linux environment
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42694
diff changeset
97
89ba81771fc9 automation: install Rust in Linux environment
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42694
diff changeset
98 chmod +x rustup-init
89ba81771fc9 automation: install Rust in Linux environment
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42694
diff changeset
99 sudo -H -u hg -g hg ./rustup-init -y
47197
2df655f682a6 automation: update rust in Linux environment
Gregory Szorc <gregory.szorc@gmail.com>
parents: 47196
diff changeset
100 sudo -H -u hg -g hg /home/hg/.cargo/bin/rustup install 1.41.1 1.52.0
42696
89ba81771fc9 automation: install Rust in Linux environment
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42694
diff changeset
101 sudo -H -u hg -g hg /home/hg/.cargo/bin/rustup component add clippy
89ba81771fc9 automation: install Rust in Linux environment
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42694
diff changeset
102 '''
89ba81771fc9 automation: install Rust in Linux environment
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42694
diff changeset
103
89ba81771fc9 automation: install Rust in Linux environment
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42694
diff changeset
104
42285
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
105 BOOTSTRAP_VIRTUALENV = r'''
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
106 /usr/bin/virtualenv /hgdev/venv-bootstrap
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
107
43014
cbd94ee3a72e automation: upgrade packages in Linux environment
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43013
diff changeset
108 HG_SHA256=35fc8ba5e0379c1b3affa2757e83fb0509e8ac314cbd9f1fd133cf265d16e49f
cbd94ee3a72e automation: upgrade packages in Linux environment
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43013
diff changeset
109 HG_TARBALL=mercurial-5.1.1.tar.gz
42285
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
110
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
111 wget -O ${HG_TARBALL} --progress dot:mega https://www.mercurial-scm.org/release/${HG_TARBALL}
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
112 echo "${HG_SHA256} ${HG_TARBALL}" | sha256sum --check -
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
113
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
114 /hgdev/venv-bootstrap/bin/pip install ${HG_TARBALL}
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43018
diff changeset
115 '''.lstrip().replace(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43018
diff changeset
116 '\r\n', '\n'
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43018
diff changeset
117 )
42285
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
118
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
119
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43018
diff changeset
120 BOOTSTRAP_DEBIAN = (
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43018
diff changeset
121 r'''
42285
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
122 #!/bin/bash
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
123
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
124 set -ex
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
125
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
126 DISTRO=`grep DISTRIB_ID /etc/lsb-release | awk -F= '{{print $2}}'`
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
127 DEBIAN_VERSION=`cat /etc/debian_version`
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
128 LSB_RELEASE=`lsb_release -cs`
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
129
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
130 sudo /usr/sbin/groupadd hg
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
131 sudo /usr/sbin/groupadd docker
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
132 sudo /usr/sbin/useradd -g hg -G sudo,docker -d /home/hg -m -s /bin/bash hg
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
133 sudo mkdir /home/hg/.ssh
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
134 sudo cp ~/.ssh/authorized_keys /home/hg/.ssh/authorized_keys
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
135 sudo chown -R hg:hg /home/hg/.ssh
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
136 sudo chmod 700 /home/hg/.ssh
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
137 sudo chmod 600 /home/hg/.ssh/authorized_keys
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
138
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
139 cat << EOF | sudo tee /etc/sudoers.d/90-hg
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
140 hg ALL=(ALL) NOPASSWD:ALL
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
141 EOF
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
142
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
143 sudo apt-get update
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
144 sudo DEBIAN_FRONTEND=noninteractive apt-get -yq dist-upgrade
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
145
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
146 # Install packages necessary to set up Docker Apt repo.
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
147 sudo DEBIAN_FRONTEND=noninteractive apt-get -yq install --no-install-recommends \
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
148 apt-transport-https \
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
149 gnupg
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
150
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
151 cat > docker-apt-key << EOF
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
152 -----BEGIN PGP PUBLIC KEY BLOCK-----
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
153
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
154 mQINBFit2ioBEADhWpZ8/wvZ6hUTiXOwQHXMAlaFHcPH9hAtr4F1y2+OYdbtMuth
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
155 lqqwp028AqyY+PRfVMtSYMbjuQuu5byyKR01BbqYhuS3jtqQmljZ/bJvXqnmiVXh
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
156 38UuLa+z077PxyxQhu5BbqntTPQMfiyqEiU+BKbq2WmANUKQf+1AmZY/IruOXbnq
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
157 L4C1+gJ8vfmXQt99npCaxEjaNRVYfOS8QcixNzHUYnb6emjlANyEVlZzeqo7XKl7
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
158 UrwV5inawTSzWNvtjEjj4nJL8NsLwscpLPQUhTQ+7BbQXAwAmeHCUTQIvvWXqw0N
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
159 cmhh4HgeQscQHYgOJjjDVfoY5MucvglbIgCqfzAHW9jxmRL4qbMZj+b1XoePEtht
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
160 ku4bIQN1X5P07fNWzlgaRL5Z4POXDDZTlIQ/El58j9kp4bnWRCJW0lya+f8ocodo
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
161 vZZ+Doi+fy4D5ZGrL4XEcIQP/Lv5uFyf+kQtl/94VFYVJOleAv8W92KdgDkhTcTD
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
162 G7c0tIkVEKNUq48b3aQ64NOZQW7fVjfoKwEZdOqPE72Pa45jrZzvUFxSpdiNk2tZ
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
163 XYukHjlxxEgBdC/J3cMMNRE1F4NCA3ApfV1Y7/hTeOnmDuDYwr9/obA8t016Yljj
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
164 q5rdkywPf4JF8mXUW5eCN1vAFHxeg9ZWemhBtQmGxXnw9M+z6hWwc6ahmwARAQAB
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
165 tCtEb2NrZXIgUmVsZWFzZSAoQ0UgZGViKSA8ZG9ja2VyQGRvY2tlci5jb20+iQI3
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
166 BBMBCgAhBQJYrefAAhsvBQsJCAcDBRUKCQgLBRYCAwEAAh4BAheAAAoJEI2BgDwO
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
167 v82IsskP/iQZo68flDQmNvn8X5XTd6RRaUH33kXYXquT6NkHJciS7E2gTJmqvMqd
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
168 tI4mNYHCSEYxI5qrcYV5YqX9P6+Ko+vozo4nseUQLPH/ATQ4qL0Zok+1jkag3Lgk
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
169 jonyUf9bwtWxFp05HC3GMHPhhcUSexCxQLQvnFWXD2sWLKivHp2fT8QbRGeZ+d3m
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
170 6fqcd5Fu7pxsqm0EUDK5NL+nPIgYhN+auTrhgzhK1CShfGccM/wfRlei9Utz6p9P
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
171 XRKIlWnXtT4qNGZNTN0tR+NLG/6Bqd8OYBaFAUcue/w1VW6JQ2VGYZHnZu9S8LMc
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
172 FYBa5Ig9PxwGQOgq6RDKDbV+PqTQT5EFMeR1mrjckk4DQJjbxeMZbiNMG5kGECA8
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
173 g383P3elhn03WGbEEa4MNc3Z4+7c236QI3xWJfNPdUbXRaAwhy/6rTSFbzwKB0Jm
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
174 ebwzQfwjQY6f55MiI/RqDCyuPj3r3jyVRkK86pQKBAJwFHyqj9KaKXMZjfVnowLh
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
175 9svIGfNbGHpucATqREvUHuQbNnqkCx8VVhtYkhDb9fEP2xBu5VvHbR+3nfVhMut5
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
176 G34Ct5RS7Jt6LIfFdtcn8CaSas/l1HbiGeRgc70X/9aYx/V/CEJv0lIe8gP6uDoW
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
177 FPIZ7d6vH+Vro6xuWEGiuMaiznap2KhZmpkgfupyFmplh0s6knymuQINBFit2ioB
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
178 EADneL9S9m4vhU3blaRjVUUyJ7b/qTjcSylvCH5XUE6R2k+ckEZjfAMZPLpO+/tF
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
179 M2JIJMD4SifKuS3xck9KtZGCufGmcwiLQRzeHF7vJUKrLD5RTkNi23ydvWZgPjtx
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
180 Q+DTT1Zcn7BrQFY6FgnRoUVIxwtdw1bMY/89rsFgS5wwuMESd3Q2RYgb7EOFOpnu
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
181 w6da7WakWf4IhnF5nsNYGDVaIHzpiqCl+uTbf1epCjrOlIzkZ3Z3Yk5CM/TiFzPk
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
182 z2lLz89cpD8U+NtCsfagWWfjd2U3jDapgH+7nQnCEWpROtzaKHG6lA3pXdix5zG8
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
183 eRc6/0IbUSWvfjKxLLPfNeCS2pCL3IeEI5nothEEYdQH6szpLog79xB9dVnJyKJb
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
184 VfxXnseoYqVrRz2VVbUI5Blwm6B40E3eGVfUQWiux54DspyVMMk41Mx7QJ3iynIa
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
185 1N4ZAqVMAEruyXTRTxc9XW0tYhDMA/1GYvz0EmFpm8LzTHA6sFVtPm/ZlNCX6P1X
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
186 zJwrv7DSQKD6GGlBQUX+OeEJ8tTkkf8QTJSPUdh8P8YxDFS5EOGAvhhpMBYD42kQ
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
187 pqXjEC+XcycTvGI7impgv9PDY1RCC1zkBjKPa120rNhv/hkVk/YhuGoajoHyy4h7
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
188 ZQopdcMtpN2dgmhEegny9JCSwxfQmQ0zK0g7m6SHiKMwjwARAQABiQQ+BBgBCAAJ
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
189 BQJYrdoqAhsCAikJEI2BgDwOv82IwV0gBBkBCAAGBQJYrdoqAAoJEH6gqcPyc/zY
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
190 1WAP/2wJ+R0gE6qsce3rjaIz58PJmc8goKrir5hnElWhPgbq7cYIsW5qiFyLhkdp
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
191 YcMmhD9mRiPpQn6Ya2w3e3B8zfIVKipbMBnke/ytZ9M7qHmDCcjoiSmwEXN3wKYI
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
192 mD9VHONsl/CG1rU9Isw1jtB5g1YxuBA7M/m36XN6x2u+NtNMDB9P56yc4gfsZVES
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
193 KA9v+yY2/l45L8d/WUkUi0YXomn6hyBGI7JrBLq0CX37GEYP6O9rrKipfz73XfO7
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
194 JIGzOKZlljb/D9RX/g7nRbCn+3EtH7xnk+TK/50euEKw8SMUg147sJTcpQmv6UzZ
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
195 cM4JgL0HbHVCojV4C/plELwMddALOFeYQzTif6sMRPf+3DSj8frbInjChC3yOLy0
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
196 6br92KFom17EIj2CAcoeq7UPhi2oouYBwPxh5ytdehJkoo+sN7RIWua6P2WSmon5
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
197 U888cSylXC0+ADFdgLX9K2zrDVYUG1vo8CX0vzxFBaHwN6Px26fhIT1/hYUHQR1z
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
198 VfNDcyQmXqkOnZvvoMfz/Q0s9BhFJ/zU6AgQbIZE/hm1spsfgvtsD1frZfygXJ9f
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
199 irP+MSAI80xHSf91qSRZOj4Pl3ZJNbq4yYxv0b1pkMqeGdjdCYhLU+LZ4wbQmpCk
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
200 SVe2prlLureigXtmZfkqevRz7FrIZiu9ky8wnCAPwC7/zmS18rgP/17bOtL4/iIz
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
201 QhxAAoAMWVrGyJivSkjhSGx1uCojsWfsTAm11P7jsruIL61ZzMUVE2aM3Pmj5G+W
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
202 9AcZ58Em+1WsVnAXdUR//bMmhyr8wL/G1YO1V3JEJTRdxsSxdYa4deGBBY/Adpsw
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
203 24jxhOJR+lsJpqIUeb999+R8euDhRHG9eFO7DRu6weatUJ6suupoDTRWtr/4yGqe
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
204 dKxV3qQhNLSnaAzqW/1nA3iUB4k7kCaKZxhdhDbClf9P37qaRW467BLCVO/coL3y
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
205 Vm50dwdrNtKpMBh3ZpbB1uJvgi9mXtyBOMJ3v8RZeDzFiG8HdCtg9RvIt/AIFoHR
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
206 H3S+U79NT6i0KPzLImDfs8T7RlpyuMc4Ufs8ggyg9v3Ae6cN3eQyxcK3w0cbBwsh
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
207 /nQNfsA6uu+9H7NhbehBMhYnpNZyrHzCmzyXkauwRAqoCbGCNykTRwsur9gS41TQ
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
208 M8ssD1jFheOJf3hODnkKU+HKjvMROl1DK7zdmLdNzA1cvtZH/nCC9KPj1z8QC47S
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
209 xx+dTZSx4ONAhwbS/LN3PoKtn8LPjY9NP9uDWI+TWYquS2U+KHDrBDlsgozDbs/O
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
210 jCxcpDzNmXpWQHEtHU7649OXHP7UeNST1mCUCH5qdank0V1iejF6/CfTFU4MfcrG
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
211 YT90qFF93M3v01BbxP+EIY2/9tiIPbrd
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
212 =0YYh
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
213 -----END PGP PUBLIC KEY BLOCK-----
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
214 EOF
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
215
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
216 sudo apt-key add docker-apt-key
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
217
43011
198b51d453fe automation: use LSB_RELEASE instead of DEBIAN_VERSION
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42696
diff changeset
218 if [ "$LSB_RELEASE" = "stretch" ]; then
42285
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
219 cat << EOF | sudo tee -a /etc/apt/sources.list
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
220 # Need backports for clang-format-6.0
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
221 deb http://deb.debian.org/debian stretch-backports main
43018
d1d919f679f7 automation: support and use Debian Buster by default
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43015
diff changeset
222 EOF
d1d919f679f7 automation: support and use Debian Buster by default
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43015
diff changeset
223 fi
42285
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
224
43018
d1d919f679f7 automation: support and use Debian Buster by default
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43015
diff changeset
225 if [ "$LSB_RELEASE" = "stretch" -o "$LSB_RELEASE" = "buster" ]; then
d1d919f679f7 automation: support and use Debian Buster by default
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43015
diff changeset
226 cat << EOF | sudo tee -a /etc/apt/sources.list
42285
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
227 # Sources are useful if we want to compile things locally.
43018
d1d919f679f7 automation: support and use Debian Buster by default
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43015
diff changeset
228 deb-src http://deb.debian.org/debian $LSB_RELEASE main
d1d919f679f7 automation: support and use Debian Buster by default
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43015
diff changeset
229 deb-src http://security.debian.org/debian-security $LSB_RELEASE/updates main
d1d919f679f7 automation: support and use Debian Buster by default
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43015
diff changeset
230 deb-src http://deb.debian.org/debian $LSB_RELEASE-updates main
d1d919f679f7 automation: support and use Debian Buster by default
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43015
diff changeset
231 deb-src http://deb.debian.org/debian $LSB_RELEASE-backports main
42285
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
232
43018
d1d919f679f7 automation: support and use Debian Buster by default
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43015
diff changeset
233 deb [arch=amd64] https://download.docker.com/linux/debian $LSB_RELEASE stable
42285
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
234 EOF
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
235
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
236 elif [ "$DISTRO" = "Ubuntu" ]; then
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
237 cat << EOF | sudo tee -a /etc/apt/sources.list
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
238 deb [arch=amd64] https://download.docker.com/linux/ubuntu $LSB_RELEASE stable
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
239 EOF
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
240
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
241 fi
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
242
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
243 sudo apt-get update
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
244
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
245 PACKAGES="\
43015
dffa9d3c2491 automation: install awscli and python3-boto3 packages
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43014
diff changeset
246 awscli \
42285
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
247 btrfs-progs \
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
248 build-essential \
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
249 bzr \
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
250 clang-format-6.0 \
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
251 cvs \
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
252 darcs \
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
253 debhelper \
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
254 devscripts \
43012
136c2536d83f automation: always install docker-ce
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43011
diff changeset
255 docker-ce \
42285
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
256 dpkg-dev \
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
257 dstat \
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
258 emacs \
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
259 gettext \
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
260 git \
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
261 htop \
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
262 iotop \
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
263 jfsutils \
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
264 libbz2-dev \
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
265 libexpat1-dev \
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
266 libffi-dev \
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
267 libgdbm-dev \
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
268 liblzma-dev \
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
269 libncurses5-dev \
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
270 libnss3-dev \
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
271 libreadline-dev \
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
272 libsqlite3-dev \
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
273 libssl-dev \
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
274 netbase \
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
275 ntfs-3g \
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
276 nvme-cli \
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
277 pyflakes \
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
278 pyflakes3 \
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
279 pylint \
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
280 pylint3 \
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
281 python-all-dev \
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
282 python-dev \
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
283 python-docutils \
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
284 python-fuzzywuzzy \
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
285 python-pygments \
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
286 python-subversion \
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
287 python-vcr \
43015
dffa9d3c2491 automation: install awscli and python3-boto3 packages
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43014
diff changeset
288 python3-boto3 \
42285
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
289 python3-dev \
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
290 python3-docutils \
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
291 python3-fuzzywuzzy \
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
292 python3-pygments \
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
293 python3-vcr \
43413
acdd4f28f6a3 automation: install python3-venv Debian package
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43315
diff changeset
294 python3-venv \
42285
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
295 rsync \
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
296 sqlite3 \
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
297 subversion \
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
298 tcl-dev \
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
299 tk-dev \
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
300 tla \
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
301 unzip \
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
302 uuid-dev \
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
303 vim \
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
304 virtualenv \
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
305 wget \
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
306 xfsprogs \
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
307 zip \
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
308 zlib1g-dev"
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
309
43011
198b51d453fe automation: use LSB_RELEASE instead of DEBIAN_VERSION
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42696
diff changeset
310 if [ "LSB_RELEASE" = "stretch" ]; then
42285
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
311 PACKAGES="$PACKAGES linux-perf"
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
312 elif [ "$DISTRO" = "Ubuntu" ]; then
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
313 PACKAGES="$PACKAGES linux-tools-common"
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
314 fi
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
315
43018
d1d919f679f7 automation: support and use Debian Buster by default
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43015
diff changeset
316 # Monotone only available in older releases.
d1d919f679f7 automation: support and use Debian Buster by default
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43015
diff changeset
317 if [ "$LSB_RELEASE" = "stretch" -o "$LSB_RELEASE" = "xenial" ]; then
42285
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
318 PACKAGES="$PACKAGES monotone"
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
319 fi
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
320
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
321 sudo DEBIAN_FRONTEND=noninteractive apt-get -yq install --no-install-recommends $PACKAGES
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
322
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
323 # Create clang-format symlink so test harness finds it.
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
324 sudo update-alternatives --install /usr/bin/clang-format clang-format \
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
325 /usr/bin/clang-format-6.0 1000
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
326
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
327 sudo mkdir /hgdev
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
328 # Will be normalized to hg:hg later.
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
329 sudo chown `whoami` /hgdev
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
330
42696
89ba81771fc9 automation: install Rust in Linux environment
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42694
diff changeset
331 {install_rust}
47202
0de5eefbe4b2 contrib: install PyOxidizer 0.16.0
Gregory Szorc <gregory.szorc@gmail.com>
parents: 47197
diff changeset
332 {install_pyoxidizer}
42696
89ba81771fc9 automation: install Rust in Linux environment
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42694
diff changeset
333
47195
546e812a1c2d automation: create Python 3.5 variant of requirements.txt
Gregory Szorc <gregory.szorc@gmail.com>
parents: 46353
diff changeset
334 cp requirements-*.txt /hgdev/
42285
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
335
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
336 # Disable the pip version check because it uses the network and can
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
337 # be annoying.
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
338 cat << EOF | sudo tee -a /etc/pip.conf
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
339 [global]
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
340 disable-pip-version-check = True
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
341 EOF
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
342
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
343 {install_pythons}
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
344 {bootstrap_virtualenv}
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
345
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
346 /hgdev/venv-bootstrap/bin/hg clone https://www.mercurial-scm.org/repo/hg /hgdev/src
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
347
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
348 # Mark the repo as non-publishing.
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
349 cat >> /hgdev/src/.hg/hgrc << EOF
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
350 [phases]
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
351 publish = false
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
352 EOF
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
353
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
354 sudo chown -R hg:hg /hgdev
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43018
diff changeset
355 '''.lstrip()
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43018
diff changeset
356 .format(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43018
diff changeset
357 install_rust=INSTALL_RUST,
47202
0de5eefbe4b2 contrib: install PyOxidizer 0.16.0
Gregory Szorc <gregory.szorc@gmail.com>
parents: 47197
diff changeset
358 install_pyoxidizer=INSTALL_PYOXIDIZER,
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43018
diff changeset
359 install_pythons=INSTALL_PYTHONS,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43018
diff changeset
360 bootstrap_virtualenv=BOOTSTRAP_VIRTUALENV,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43018
diff changeset
361 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43018
diff changeset
362 .replace('\r\n', '\n')
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43018
diff changeset
363 )
42285
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
364
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
365
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
366 # Prepares /hgdev for operations.
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
367 PREPARE_HGDEV = '''
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
368 #!/bin/bash
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
369
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
370 set -e
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
371
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
372 FS=$1
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
373
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
374 ensure_device() {
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
375 if [ -z "${DEVICE}" ]; then
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
376 echo "could not find block device to format"
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
377 exit 1
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
378 fi
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
379 }
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
380
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
381 # Determine device to partition for extra filesystem.
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
382 # If only 1 volume is present, it will be the root volume and
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
383 # should be /dev/nvme0. If multiple volumes are present, the
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
384 # root volume could be nvme0 or nvme1. Use whichever one doesn't have
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
385 # a partition.
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
386 if [ -e /dev/nvme1n1 ]; then
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
387 if [ -e /dev/nvme0n1p1 ]; then
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
388 DEVICE=/dev/nvme1n1
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
389 else
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
390 DEVICE=/dev/nvme0n1
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
391 fi
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
392 else
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
393 DEVICE=
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
394 fi
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
395
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
396 sudo mkdir /hgwork
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
397
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
398 if [ "${FS}" != "default" -a "${FS}" != "tmpfs" ]; then
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
399 ensure_device
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
400 echo "creating ${FS} filesystem on ${DEVICE}"
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
401 fi
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
402
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
403 if [ "${FS}" = "default" ]; then
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
404 :
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
405
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
406 elif [ "${FS}" = "btrfs" ]; then
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
407 sudo mkfs.btrfs ${DEVICE}
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
408 sudo mount ${DEVICE} /hgwork
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
409
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
410 elif [ "${FS}" = "ext3" ]; then
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
411 # lazy_journal_init speeds up filesystem creation at the expense of
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
412 # integrity if things crash. We are an ephemeral instance, so we don't
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
413 # care about integrity.
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
414 sudo mkfs.ext3 -E lazy_journal_init=1 ${DEVICE}
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
415 sudo mount ${DEVICE} /hgwork
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
416
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
417 elif [ "${FS}" = "ext4" ]; then
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
418 sudo mkfs.ext4 -E lazy_journal_init=1 ${DEVICE}
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
419 sudo mount ${DEVICE} /hgwork
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
420
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
421 elif [ "${FS}" = "jfs" ]; then
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
422 sudo mkfs.jfs ${DEVICE}
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
423 sudo mount ${DEVICE} /hgwork
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
424
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
425 elif [ "${FS}" = "tmpfs" ]; then
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
426 echo "creating tmpfs volume in /hgwork"
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
427 sudo mount -t tmpfs -o size=1024M tmpfs /hgwork
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
428
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
429 elif [ "${FS}" = "xfs" ]; then
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
430 sudo mkfs.xfs ${DEVICE}
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
431 sudo mount ${DEVICE} /hgwork
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
432
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
433 else
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
434 echo "unsupported filesystem: ${FS}"
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
435 exit 1
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
436 fi
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
437
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
438 echo "/hgwork ready"
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
439
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
440 sudo chown hg:hg /hgwork
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
441 mkdir /hgwork/tmp
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
442 chown hg:hg /hgwork/tmp
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
443
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
444 rsync -a /hgdev/src /hgwork/
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43018
diff changeset
445 '''.lstrip().replace(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43018
diff changeset
446 '\r\n', '\n'
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43018
diff changeset
447 )
42285
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
448
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
449
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
450 HG_UPDATE_CLEAN = '''
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
451 set -ex
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
452
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
453 HG=/hgdev/venv-bootstrap/bin/hg
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
454
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
455 cd /hgwork/src
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
456 ${HG} --config extensions.purge= purge --all
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
457 ${HG} update -C $1
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
458 ${HG} log -r .
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43018
diff changeset
459 '''.lstrip().replace(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43018
diff changeset
460 '\r\n', '\n'
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43018
diff changeset
461 )
42285
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
462
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
463
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
464 def prepare_exec_environment(ssh_client, filesystem='default'):
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
465 """Prepare an EC2 instance to execute things.
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
466
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
467 The AMI has an ``/hgdev`` bootstrapped with various Python installs
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
468 and a clone of the Mercurial repo.
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
469
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
470 In EC2, EBS volumes launched from snapshots have wonky performance behavior.
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
471 Notably, blocks have to be copied on first access, which makes volume
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
472 I/O extremely slow on fresh volumes.
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
473
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
474 Furthermore, we may want to run operations, tests, etc on alternative
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
475 filesystems so we examine behavior on different filesystems.
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
476
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
477 This function is used to facilitate executing operations on alternate
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
478 volumes.
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
479 """
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
480 sftp = ssh_client.open_sftp()
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
481
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
482 with sftp.open('/hgdev/prepare-hgdev', 'wb') as fh:
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
483 fh.write(PREPARE_HGDEV)
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
484 fh.chmod(0o0777)
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
485
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
486 command = 'sudo /hgdev/prepare-hgdev %s' % filesystem
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
487 chan, stdin, stdout = exec_command(ssh_client, command)
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
488 stdin.close()
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
489
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
490 for line in stdout:
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
491 print(line, end='')
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
492
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
493 res = chan.recv_exit_status()
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
494
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
495 if res:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43018
diff changeset
496 raise Exception('non-0 exit code updating working directory; %d' % res)
42285
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
497
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
498
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43018
diff changeset
499 def synchronize_hg(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43018
diff changeset
500 source_path: pathlib.Path, ec2_instance, revision: str = None
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43018
diff changeset
501 ):
42285
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
502 """Synchronize a local Mercurial source path to remote EC2 instance."""
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
503
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
504 with tempfile.TemporaryDirectory() as temp_dir:
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
505 temp_dir = pathlib.Path(temp_dir)
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
506
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
507 ssh_dir = temp_dir / '.ssh'
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
508 ssh_dir.mkdir()
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
509 ssh_dir.chmod(0o0700)
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
510
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
511 public_ip = ec2_instance.public_ip_address
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
512
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
513 ssh_config = ssh_dir / 'config'
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
514
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
515 with ssh_config.open('w', encoding='utf-8') as fh:
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
516 fh.write('Host %s\n' % public_ip)
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
517 fh.write(' User hg\n')
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
518 fh.write(' StrictHostKeyChecking no\n')
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
519 fh.write(' UserKnownHostsFile %s\n' % (ssh_dir / 'known_hosts'))
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
520 fh.write(' IdentityFile %s\n' % ec2_instance.ssh_private_key_path)
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
521
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
522 if not (source_path / '.hg').is_dir():
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43018
diff changeset
523 raise Exception(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43018
diff changeset
524 '%s is not a Mercurial repository; synchronization '
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43018
diff changeset
525 'not yet supported' % source_path
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43018
diff changeset
526 )
42285
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
527
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
528 env = dict(os.environ)
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
529 env['HGPLAIN'] = '1'
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
530 env['HGENCODING'] = 'utf-8'
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
531
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
532 hg_bin = source_path / 'hg'
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
533
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
534 res = subprocess.run(
48843
d953a42b157d automation: run hg with python3
Gregory Szorc <gregory.szorc@gmail.com>
parents: 47202
diff changeset
535 ['python3', str(hg_bin), 'log', '-r', revision, '-T', '{node}'],
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43018
diff changeset
536 cwd=str(source_path),
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43018
diff changeset
537 env=env,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43018
diff changeset
538 check=True,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43018
diff changeset
539 capture_output=True,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43018
diff changeset
540 )
42285
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
541
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
542 full_revision = res.stdout.decode('ascii')
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
543
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
544 args = [
48843
d953a42b157d automation: run hg with python3
Gregory Szorc <gregory.szorc@gmail.com>
parents: 47202
diff changeset
545 'python3',
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43018
diff changeset
546 str(hg_bin),
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43018
diff changeset
547 '--config',
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43018
diff changeset
548 'ui.ssh=ssh -F %s' % ssh_config,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43018
diff changeset
549 '--config',
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43018
diff changeset
550 'ui.remotecmd=/hgdev/venv-bootstrap/bin/hg',
42684
9e0f1c80cddb automation: push changes affecting .hgtags
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42660
diff changeset
551 # Also ensure .hgtags changes are present so auto version
9e0f1c80cddb automation: push changes affecting .hgtags
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42660
diff changeset
552 # calculation works.
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43018
diff changeset
553 'push',
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43018
diff changeset
554 '-f',
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43018
diff changeset
555 '-r',
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43018
diff changeset
556 full_revision,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43018
diff changeset
557 '-r',
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43018
diff changeset
558 'file(.hgtags)',
42285
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
559 'ssh://%s//hgwork/src' % public_ip,
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
560 ]
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
561
42660
24cd5b0ba5b3 automation: allow exit code of 1 for `hg push`
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42285
diff changeset
562 res = subprocess.run(args, cwd=str(source_path), env=env)
24cd5b0ba5b3 automation: allow exit code of 1 for `hg push`
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42285
diff changeset
563
24cd5b0ba5b3 automation: allow exit code of 1 for `hg push`
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42285
diff changeset
564 # Allow 1 (no-op) to not trigger error.
24cd5b0ba5b3 automation: allow exit code of 1 for `hg push`
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42285
diff changeset
565 if res.returncode not in (0, 1):
24cd5b0ba5b3 automation: allow exit code of 1 for `hg push`
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42285
diff changeset
566 res.check_returncode()
42285
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
567
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
568 # TODO support synchronizing dirty working directory.
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
569
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
570 sftp = ec2_instance.ssh_client.open_sftp()
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
571
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
572 with sftp.open('/hgdev/hgup', 'wb') as fh:
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
573 fh.write(HG_UPDATE_CLEAN)
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
574 fh.chmod(0o0700)
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
575
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
576 chan, stdin, stdout = exec_command(
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43018
diff changeset
577 ec2_instance.ssh_client, '/hgdev/hgup %s' % full_revision
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43018
diff changeset
578 )
42285
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
579 stdin.close()
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
580
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
581 for line in stdout:
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
582 print(line, end='')
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
583
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
584 res = chan.recv_exit_status()
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
585
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
586 if res:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43018
diff changeset
587 raise Exception(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43018
diff changeset
588 'non-0 exit code updating working directory; %d' % res
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43018
diff changeset
589 )
42285
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
590
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
591
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
592 def run_tests(ssh_client, python_version, test_flags=None):
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
593 """Run tests on a remote Linux machine via an SSH client."""
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
594 test_flags = test_flags or []
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
595
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
596 print('running tests')
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
597
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
598 if python_version == 'system2':
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
599 python = '/usr/bin/python2'
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
600 elif python_version == 'system3':
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
601 python = '/usr/bin/python3'
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
602 elif python_version.startswith('pypy'):
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
603 python = '/hgdev/pyenv/shims/%s' % python_version
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
604 else:
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
605 python = '/hgdev/pyenv/shims/python%s' % python_version
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
606
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
607 test_flags = ' '.join(shlex.quote(a) for a in test_flags)
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
608
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
609 command = (
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
610 '/bin/sh -c "export TMPDIR=/hgwork/tmp; '
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43018
diff changeset
611 'cd /hgwork/src/tests && %s run-tests.py %s"' % (python, test_flags)
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43018
diff changeset
612 )
42285
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
613
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
614 chan, stdin, stdout = exec_command(ssh_client, command)
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
615
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
616 stdin.close()
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
617
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
618 for line in stdout:
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
619 print(line, end='')
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
620
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
621 return chan.recv_exit_status()