annotate contrib/automation/hgautomation/try_server.py @ 46233:172b294b6d65

setup: don't import distutils prior to checking FORCE_SETUPTOOLS I've seen warnings about importing distutils before setuptools, although I can't reproduce them at the moment. Differential Revision: https://phab.mercurial-scm.org/D9722
author Dan Villiom Podlaski Christiansen <danchr@gmail.com>
date Mon, 11 Jan 2021 13:48:13 +0100
parents c5c502bd1f70
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
43057
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
1 # try_server.py - Interact with Try server
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
2 #
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
3 # Copyright 2019 Gregory Szorc <gregory.szorc@gmail.com>
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
4 #
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
5 # This software may be used and distributed according to the terms of the
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
6 # GNU General Public License version 2 or any later version.
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
7
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
8 # no-check-code because Python 3 native.
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
9
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
10 import base64
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
11 import json
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
12 import os
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
13 import subprocess
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
14 import tempfile
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
15
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
16 from .aws import AWSConnection
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
17
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
18 LAMBDA_FUNCTION = "ci-try-server-upload"
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
19
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
20
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
21 def trigger_try(c: AWSConnection, rev="."):
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
22 """Trigger a new Try run."""
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
23 lambda_client = c.session.client("lambda")
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
24
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
25 cset, bundle = generate_bundle(rev=rev)
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
26
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
27 payload = {
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
28 "bundle": base64.b64encode(bundle).decode("utf-8"),
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
29 "node": cset["node"],
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
30 "branch": cset["branch"],
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
31 "user": cset["user"],
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
32 "message": cset["desc"],
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
33 }
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
34
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
35 print("resolved revision:")
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
36 print("node: %s" % cset["node"])
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
37 print("branch: %s" % cset["branch"])
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
38 print("user: %s" % cset["user"])
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
39 print("desc: %s" % cset["desc"].splitlines()[0])
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
40 print()
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
41
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
42 print("sending to Try...")
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
43 res = lambda_client.invoke(
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
44 FunctionName=LAMBDA_FUNCTION,
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
45 InvocationType="RequestResponse",
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
46 Payload=json.dumps(payload).encode("utf-8"),
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
47 )
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
48
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
49 body = json.load(res["Payload"])
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
50 for message in body:
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
51 print("remote: %s" % message)
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
52
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
53
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
54 def generate_bundle(rev="."):
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
55 """Generate a bundle suitable for use by the Try service.
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
56
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
57 Returns a tuple of revision metadata and raw Mercurial bundle data.
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
58 """
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
59 # `hg bundle` doesn't support streaming to stdout. So we use a temporary
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
60 # file.
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
61 path = None
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
62 try:
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
63 fd, path = tempfile.mkstemp(prefix="hg-bundle-", suffix=".hg")
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
64 os.close(fd)
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
65
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
66 args = [
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
67 "hg",
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
68 "bundle",
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
69 "--type",
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
70 "gzip-v2",
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
71 "--base",
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
72 "public()",
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
73 "--rev",
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
74 rev,
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
75 path,
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
76 ]
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
77
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
78 print("generating bundle...")
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
79 subprocess.run(args, check=True)
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
80
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
81 with open(path, "rb") as fh:
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
82 bundle_data = fh.read()
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
83
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
84 finally:
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
85 if path:
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
86 os.unlink(path)
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
87
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
88 args = [
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
89 "hg",
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
90 "log",
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
91 "-r",
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
92 rev,
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
93 # We have to upload as JSON, so it won't matter if we emit binary
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
94 # since we need to normalize to UTF-8.
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
95 "-T",
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
96 "json",
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
97 ]
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
98 res = subprocess.run(args, check=True, capture_output=True)
c5c502bd1f70 automation: add a command to submit to a Try server
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
99 return json.loads(res.stdout)[0], bundle_data