Mercurial > hg
annotate contrib/pull_logger.py @ 49508:791050360486
contrib: add pull_logger extension
This extension logs the pull parameters, i.e. the remote and common heads,
when pulling from the local repository.
The collected data should give an idea of the state of a pair of repositories
and allow replaying past synchronisations between them. This is particularly
useful for working on data exchange, bundling and caching-related
optimisations.
author | pacien <pacien.trangirard@pacien.net> |
---|---|
date | Mon, 25 Jul 2022 22:47:15 +0200 |
parents | |
children | 946c023212b8 |
rev | line source |
---|---|
49508
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
1 # pull_logger.py - Logs pulls to a JSON-line file in the repo's VFS. |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
2 # |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
3 # Copyright 2022 Pacien TRAN-GIRARD <pacien.trangirard@pacien.net> |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
4 # |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
5 # This software may be used and distributed according to the terms of the |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
6 # GNU General Public License version 2 or any later version. |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
7 |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
8 |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
9 '''logs pull parameters to a file |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
10 |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
11 This extension logs the pull parameters, i.e. the remote and common heads, |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
12 when pulling from the local repository. |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
13 |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
14 The collected data should give an idea of the state of a pair of repositories |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
15 and allow replaying past synchronisations between them. This is particularly |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
16 useful for working on data exchange, bundling and caching-related |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
17 optimisations. |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
18 |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
19 The record is a JSON-line file located in the repository's VFS at |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
20 .hg/pull_log.jsonl. |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
21 |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
22 Log write failures are not considered fatal: log writes may be skipped for any |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
23 reason such as insufficient storage or a timeout. |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
24 |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
25 The timeouts of the exclusive lock used when writing to the lock file can be |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
26 configured through the 'timeout.lock' and 'timeout.warn' options of this |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
27 plugin. Those are not expected to be held for a significant time in practice.:: |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
28 |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
29 [pull-logger] |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
30 timeout.lock = 300 |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
31 timeout.warn = 100 |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
32 |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
33 Note: there is no automatic log rotation and the size of the log is not capped. |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
34 ''' |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
35 |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
36 |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
37 import json |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
38 import time |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
39 |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
40 from mercurial.i18n import _ |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
41 from mercurial.utils import stringutil |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
42 from mercurial import ( |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
43 error, |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
44 extensions, |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
45 lock, |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
46 registrar, |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
47 wireprotov1server, |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
48 ) |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
49 |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
50 EXT_NAME = b'pull-logger' |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
51 EXT_VERSION_CODE = 0 |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
52 |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
53 LOG_FILE = b'pull_log.jsonl' |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
54 LOCK_NAME = LOG_FILE + b'.lock' |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
55 |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
56 configtable = {} |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
57 configitem = registrar.configitem(configtable) |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
58 configitem(EXT_NAME, b'timeout.lock', default=600) |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
59 configitem(EXT_NAME, b'timeout.warn', default=120) |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
60 |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
61 |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
62 def wrap_getbundle(orig, repo, proto, others, *args, **kwargs): |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
63 heads, common = extract_pull_heads(others) |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
64 log_entry = { |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
65 'timestamp': time.time(), |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
66 'logger_version': EXT_VERSION_CODE, |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
67 'heads': sorted(heads), |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
68 'common': sorted(common), |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
69 } |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
70 |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
71 try: |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
72 write_to_log(repo, log_entry) |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
73 except (IOError, error.LockError) as err: |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
74 msg = stringutil.forcebytestr(err) |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
75 repo.ui.warn(_(b'unable to append to pull log: %s\n') % msg) |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
76 |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
77 return orig(repo, proto, others, *args, **kwargs) |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
78 |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
79 |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
80 def extract_pull_heads(bundle_args): |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
81 opts = wireprotov1server.options( |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
82 b'getbundle', |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
83 wireprotov1server.wireprototypes.GETBUNDLE_ARGUMENTS.keys(), |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
84 bundle_args.copy(), # this call consumes the args destructively |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
85 ) |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
86 |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
87 heads = opts.get(b'heads', b'').decode('utf-8').split(' ') |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
88 common = opts.get(b'common', b'').decode('utf-8').split(' ') |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
89 return (heads, common) |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
90 |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
91 |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
92 def write_to_log(repo, entry): |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
93 locktimeout = repo.ui.configint(EXT_NAME, b'timeout.lock') |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
94 lockwarntimeout = repo.ui.configint(EXT_NAME, b'timeout.warn') |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
95 |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
96 with lock.trylock( |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
97 ui=repo.ui, |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
98 vfs=repo.vfs, |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
99 lockname=LOCK_NAME, |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
100 timeout=locktimeout, |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
101 warntimeout=lockwarntimeout, |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
102 ): |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
103 with repo.vfs.open(LOG_FILE, b'a+') as logfile: |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
104 serialised = json.dumps(entry, sort_keys=True) |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
105 logfile.write(serialised.encode('utf-8')) |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
106 logfile.write(b'\n') |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
107 logfile.flush() |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
108 |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
109 |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
110 def reposetup(ui, repo): |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
111 if repo.local(): |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
112 repo._wlockfreeprefix.add(LOG_FILE) |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
113 |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
114 |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
115 def uisetup(ui): |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
116 del wireprotov1server.commands[b'getbundle'] |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
117 decorator = wireprotov1server.wireprotocommand( |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
118 name=b'getbundle', |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
119 args=b'*', |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
120 permission=b'pull', |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
121 ) |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
122 |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
123 extensions.wrapfunction( |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
124 container=wireprotov1server, |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
125 funcname='getbundle', |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
126 wrapper=wrap_getbundle, |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
127 ) |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
128 |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
129 decorator(wireprotov1server.getbundle) |