Mercurial > hg
annotate contrib/catapipe.py @ 44477:ad718271a9eb
git: skeleton of a new extension to _directly_ operate on git repos
This is based in part of work I did years ago in hgit, but it's mostly
new code since I'm using pygit2 instead of dulwich and the hg storage
interfaces have improved. Some cleanup of old hgit code by Pulkit,
which I greatly appreciate.
test-git-interop.t does not cover a whole lot of cases, but it
passes. It includes status, diff, making a new commit, and `hg annotate`
working on the git repository.
This is _not_ (yet) production quality code: this is an
experiment. Known technical debt lurking in this implementation:
* Writing bookmarks just totally ignores transactions.
* The way progress is threaded down into the gitstore is awful.
* Ideally we'd find a way to incrementally reindex DAGs. I'm not sure
how to do that efficiently, so we might need a "known only fast-forwards"
mode on the DAG indexer for use on `hg commit` and friends.
* We don't even _try_ to do anything reasonable for `hg pull` or `hg push`.
* Mercurial need an interface for the changelog type.
Tests currently require git 2.24 as far as I'm aware: `git status` has
some changed output that I didn't try and handle in a compatible way.
This patch has produced some interesting cleanups, most recently on
the manifest type. I expect continuing down this road will produce
other meritorious cleanups throughout our code.
Differential Revision: https://phab.mercurial-scm.org/D6734
author | Augie Fackler <augie@google.com> |
---|---|
date | Tue, 11 Feb 2020 00:44:59 -0500 |
parents | 2372284d9457 |
children | 6000f5b25c9b |
rev | line source |
---|---|
39252
9a81f126f9fa
contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
1 #!/usr/bin/env python3 |
9a81f126f9fa
contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
2 # |
9a81f126f9fa
contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
3 # Copyright 2018 Google LLC. |
9a81f126f9fa
contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
4 # |
9a81f126f9fa
contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
5 # This software may be used and distributed according to the terms of the |
9a81f126f9fa
contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
6 # GNU General Public License version 2 or any later version. |
9a81f126f9fa
contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
7 """Tool read primitive events from a pipe to produce a catapult trace. |
9a81f126f9fa
contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
8 |
40491
c311424ea579
catapult: add a bit more documentation on how to use catapult tracing
Kyle Lippincott <spectral@google.com>
parents:
39935
diff
changeset
|
9 Usage: |
c311424ea579
catapult: add a bit more documentation on how to use catapult tracing
Kyle Lippincott <spectral@google.com>
parents:
39935
diff
changeset
|
10 Terminal 1: $ catapipe.py /tmp/mypipe /tmp/trace.json |
c311424ea579
catapult: add a bit more documentation on how to use catapult tracing
Kyle Lippincott <spectral@google.com>
parents:
39935
diff
changeset
|
11 Terminal 2: $ HGCATAPULTSERVERPIPE=/tmp/mypipe hg root |
c311424ea579
catapult: add a bit more documentation on how to use catapult tracing
Kyle Lippincott <spectral@google.com>
parents:
39935
diff
changeset
|
12 <ctrl-c catapipe.py in Terminal 1> |
c311424ea579
catapult: add a bit more documentation on how to use catapult tracing
Kyle Lippincott <spectral@google.com>
parents:
39935
diff
changeset
|
13 $ catapult/tracing/bin/trace2html /tmp/trace.json # produce /tmp/trace.html |
c311424ea579
catapult: add a bit more documentation on how to use catapult tracing
Kyle Lippincott <spectral@google.com>
parents:
39935
diff
changeset
|
14 <open trace.html in your browser of choice; the WASD keys are very useful> |
c311424ea579
catapult: add a bit more documentation on how to use catapult tracing
Kyle Lippincott <spectral@google.com>
parents:
39935
diff
changeset
|
15 (catapult is located at https://github.com/catapult-project/catapult) |
c311424ea579
catapult: add a bit more documentation on how to use catapult tracing
Kyle Lippincott <spectral@google.com>
parents:
39935
diff
changeset
|
16 |
39252
9a81f126f9fa
contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
17 For now the event stream supports |
9a81f126f9fa
contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
18 |
9a81f126f9fa
contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
19 START $SESSIONID ... |
9a81f126f9fa
contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
20 |
9a81f126f9fa
contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
21 and |
9a81f126f9fa
contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
22 |
9a81f126f9fa
contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
23 END $SESSIONID ... |
9a81f126f9fa
contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
24 |
9a81f126f9fa
contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
25 events. Everything after the SESSIONID (which must not contain spaces) |
9a81f126f9fa
contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
26 is used as a label for the event. Events are timestamped as of when |
9a81f126f9fa
contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
27 they arrive in this process and are then used to produce catapult |
9a81f126f9fa
contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
28 traces that can be loaded in Chrome's about:tracing utility. It's |
9a81f126f9fa
contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
29 important that the event stream *into* this process stay simple, |
9a81f126f9fa
contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
30 because we have to emit it from the shell scripts produced by |
9a81f126f9fa
contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
31 run-tests.py. |
9a81f126f9fa
contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
32 |
9a81f126f9fa
contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
33 Typically you'll want to place the path to the named pipe in the |
9a81f126f9fa
contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
34 HGCATAPULTSERVERPIPE environment variable, which both run-tests and hg |
40491
c311424ea579
catapult: add a bit more documentation on how to use catapult tracing
Kyle Lippincott <spectral@google.com>
parents:
39935
diff
changeset
|
35 understand. To trace *only* run-tests, use HGTESTCATAPULTSERVERPIPE instead. |
39252
9a81f126f9fa
contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
36 """ |
9a81f126f9fa
contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
37 from __future__ import absolute_import, print_function |
9a81f126f9fa
contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
38 |
9a81f126f9fa
contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
39 import argparse |
9a81f126f9fa
contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
40 import json |
9a81f126f9fa
contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
41 import os |
39514
b29ec19748a7
contrib: use a monotonic timer in catapipe
Boris Feld <boris.feld@octobus.net>
parents:
39513
diff
changeset
|
42 import timeit |
39252
9a81f126f9fa
contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
43 |
9a81f126f9fa
contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
44 _TYPEMAP = { |
9a81f126f9fa
contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
45 'START': 'B', |
9a81f126f9fa
contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
46 'END': 'E', |
42475
ff562d711919
catapipe: add support for COUNTER events
Augie Fackler <augie@google.com>
parents:
40491
diff
changeset
|
47 'COUNTER': 'C', |
39252
9a81f126f9fa
contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
48 } |
9a81f126f9fa
contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
49 |
9a81f126f9fa
contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
50 _threadmap = {} |
9a81f126f9fa
contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
51 |
39514
b29ec19748a7
contrib: use a monotonic timer in catapipe
Boris Feld <boris.feld@octobus.net>
parents:
39513
diff
changeset
|
52 # Timeit already contains the whole logic about which timer to use based on |
b29ec19748a7
contrib: use a monotonic timer in catapipe
Boris Feld <boris.feld@octobus.net>
parents:
39513
diff
changeset
|
53 # Python version and OS |
b29ec19748a7
contrib: use a monotonic timer in catapipe
Boris Feld <boris.feld@octobus.net>
parents:
39513
diff
changeset
|
54 timer = timeit.default_timer |
b29ec19748a7
contrib: use a monotonic timer in catapipe
Boris Feld <boris.feld@octobus.net>
parents:
39513
diff
changeset
|
55 |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42475
diff
changeset
|
56 |
39252
9a81f126f9fa
contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
57 def main(): |
9a81f126f9fa
contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
58 parser = argparse.ArgumentParser() |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42475
diff
changeset
|
59 parser.add_argument( |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42475
diff
changeset
|
60 'pipe', |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42475
diff
changeset
|
61 type=str, |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42475
diff
changeset
|
62 nargs=1, |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42475
diff
changeset
|
63 help='Path of named pipe to create and listen on.', |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42475
diff
changeset
|
64 ) |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42475
diff
changeset
|
65 parser.add_argument( |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42475
diff
changeset
|
66 'output', |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42475
diff
changeset
|
67 default='trace.json', |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42475
diff
changeset
|
68 type=str, |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42475
diff
changeset
|
69 nargs='?', |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42475
diff
changeset
|
70 help='Path of json file to create where the traces ' 'will be stored.', |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42475
diff
changeset
|
71 ) |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42475
diff
changeset
|
72 parser.add_argument( |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42475
diff
changeset
|
73 '--debug', |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42475
diff
changeset
|
74 default=False, |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42475
diff
changeset
|
75 action='store_true', |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42475
diff
changeset
|
76 help='Print useful debug messages', |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42475
diff
changeset
|
77 ) |
39252
9a81f126f9fa
contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
78 args = parser.parse_args() |
9a81f126f9fa
contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
79 fn = args.pipe[0] |
9a81f126f9fa
contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
80 os.mkfifo(fn) |
9a81f126f9fa
contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
81 try: |
9a81f126f9fa
contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
82 with open(fn) as f, open(args.output, 'w') as out: |
9a81f126f9fa
contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
83 out.write('[\n') |
39514
b29ec19748a7
contrib: use a monotonic timer in catapipe
Boris Feld <boris.feld@octobus.net>
parents:
39513
diff
changeset
|
84 start = timer() |
39252
9a81f126f9fa
contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
85 while True: |
9a81f126f9fa
contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
86 ev = f.readline().strip() |
9a81f126f9fa
contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
87 if not ev: |
9a81f126f9fa
contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
88 continue |
39514
b29ec19748a7
contrib: use a monotonic timer in catapipe
Boris Feld <boris.feld@octobus.net>
parents:
39513
diff
changeset
|
89 now = timer() |
39252
9a81f126f9fa
contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
90 if args.debug: |
9a81f126f9fa
contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
91 print(ev) |
9a81f126f9fa
contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
92 verb, session, label = ev.split(' ', 2) |
9a81f126f9fa
contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
93 if session not in _threadmap: |
9a81f126f9fa
contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
94 _threadmap[session] = len(_threadmap) |
42475
ff562d711919
catapipe: add support for COUNTER events
Augie Fackler <augie@google.com>
parents:
40491
diff
changeset
|
95 if verb == 'COUNTER': |
ff562d711919
catapipe: add support for COUNTER events
Augie Fackler <augie@google.com>
parents:
40491
diff
changeset
|
96 amount, label = label.split(' ', 1) |
ff562d711919
catapipe: add support for COUNTER events
Augie Fackler <augie@google.com>
parents:
40491
diff
changeset
|
97 payload_args = {'value': int(amount)} |
ff562d711919
catapipe: add support for COUNTER events
Augie Fackler <augie@google.com>
parents:
40491
diff
changeset
|
98 else: |
ff562d711919
catapipe: add support for COUNTER events
Augie Fackler <augie@google.com>
parents:
40491
diff
changeset
|
99 payload_args = {} |
39252
9a81f126f9fa
contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
100 pid = _threadmap[session] |
39935
cb9b23ec093c
catapipe: fix bug introduced when switching to the timer api
Augie Fackler <augie@google.com>
parents:
39514
diff
changeset
|
101 ts_micros = (now - start) * 1000000 |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42475
diff
changeset
|
102 out.write( |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42475
diff
changeset
|
103 json.dumps( |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42475
diff
changeset
|
104 { |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42475
diff
changeset
|
105 "name": label, |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42475
diff
changeset
|
106 "cat": "misc", |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42475
diff
changeset
|
107 "ph": _TYPEMAP[verb], |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42475
diff
changeset
|
108 "ts": ts_micros, |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42475
diff
changeset
|
109 "pid": pid, |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42475
diff
changeset
|
110 "tid": 1, |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42475
diff
changeset
|
111 "args": payload_args, |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42475
diff
changeset
|
112 } |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42475
diff
changeset
|
113 ) |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42475
diff
changeset
|
114 ) |
39252
9a81f126f9fa
contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
115 out.write(',\n') |
9a81f126f9fa
contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
116 finally: |
9a81f126f9fa
contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
117 os.unlink(fn) |
9a81f126f9fa
contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
118 |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42475
diff
changeset
|
119 |
39252
9a81f126f9fa
contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
120 if __name__ == '__main__': |
9a81f126f9fa
contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
121 main() |