Mercurial > hg
annotate contrib/pull_logger.py @ 49509:946c023212b8
pull_logger: add basic log file rotation based on size
author | pacien <pacien.trangirard@pacien.net> |
---|---|
date | Tue, 20 Sep 2022 14:04:54 +0200 |
parents | 791050360486 |
children |
rev | line source |
---|---|
49508
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
1 # pull_logger.py - Logs pulls to a JSON-line file in the repo's VFS. |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
2 # |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
3 # Copyright 2022 Pacien TRAN-GIRARD <pacien.trangirard@pacien.net> |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
4 # |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
5 # This software may be used and distributed according to the terms of the |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
6 # GNU General Public License version 2 or any later version. |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
7 |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
8 |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
9 '''logs pull parameters to a file |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
10 |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
11 This extension logs the pull parameters, i.e. the remote and common heads, |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
12 when pulling from the local repository. |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
13 |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
14 The collected data should give an idea of the state of a pair of repositories |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
15 and allow replaying past synchronisations between them. This is particularly |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
16 useful for working on data exchange, bundling and caching-related |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
17 optimisations. |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
18 |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
19 The record is a JSON-line file located in the repository's VFS at |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
20 .hg/pull_log.jsonl. |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
21 |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
22 Log write failures are not considered fatal: log writes may be skipped for any |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
23 reason such as insufficient storage or a timeout. |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
24 |
49509
946c023212b8
pull_logger: add basic log file rotation based on size
pacien <pacien.trangirard@pacien.net>
parents:
49508
diff
changeset
|
25 Some basic log file rotation can be enabled by setting 'rotate-size' to a value |
946c023212b8
pull_logger: add basic log file rotation based on size
pacien <pacien.trangirard@pacien.net>
parents:
49508
diff
changeset
|
26 greater than 0. This causes the current log file to be moved to |
946c023212b8
pull_logger: add basic log file rotation based on size
pacien <pacien.trangirard@pacien.net>
parents:
49508
diff
changeset
|
27 .hg/pull_log.jsonl.rotated when this threshold is met, discarding any |
946c023212b8
pull_logger: add basic log file rotation based on size
pacien <pacien.trangirard@pacien.net>
parents:
49508
diff
changeset
|
28 previously rotated log file. |
946c023212b8
pull_logger: add basic log file rotation based on size
pacien <pacien.trangirard@pacien.net>
parents:
49508
diff
changeset
|
29 |
49508
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
30 The timeouts of the exclusive lock used when writing to the lock file can be |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
31 configured through the 'timeout.lock' and 'timeout.warn' options of this |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
32 plugin. Those are not expected to be held for a significant time in practice.:: |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
33 |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
34 [pull-logger] |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
35 timeout.lock = 300 |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
36 timeout.warn = 100 |
49509
946c023212b8
pull_logger: add basic log file rotation based on size
pacien <pacien.trangirard@pacien.net>
parents:
49508
diff
changeset
|
37 rotate-size = 1kb |
49508
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
38 ''' |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
39 |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
40 |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
41 import json |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
42 import time |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
43 |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
44 from mercurial.i18n import _ |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
45 from mercurial.utils import stringutil |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
46 from mercurial import ( |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
47 error, |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
48 extensions, |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
49 lock, |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
50 registrar, |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
51 wireprotov1server, |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
52 ) |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
53 |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
54 EXT_NAME = b'pull-logger' |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
55 EXT_VERSION_CODE = 0 |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
56 |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
57 LOG_FILE = b'pull_log.jsonl' |
49509
946c023212b8
pull_logger: add basic log file rotation based on size
pacien <pacien.trangirard@pacien.net>
parents:
49508
diff
changeset
|
58 OLD_LOG_FILE = LOG_FILE + b'.rotated' |
49508
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
59 LOCK_NAME = LOG_FILE + b'.lock' |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
60 |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
61 configtable = {} |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
62 configitem = registrar.configitem(configtable) |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
63 configitem(EXT_NAME, b'timeout.lock', default=600) |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
64 configitem(EXT_NAME, b'timeout.warn', default=120) |
49509
946c023212b8
pull_logger: add basic log file rotation based on size
pacien <pacien.trangirard@pacien.net>
parents:
49508
diff
changeset
|
65 configitem(EXT_NAME, b'rotate-size', default=b'100MB') |
49508
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
66 |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
67 |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
68 def wrap_getbundle(orig, repo, proto, others, *args, **kwargs): |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
69 heads, common = extract_pull_heads(others) |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
70 log_entry = { |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
71 'timestamp': time.time(), |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
72 'logger_version': EXT_VERSION_CODE, |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
73 'heads': sorted(heads), |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
74 'common': sorted(common), |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
75 } |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
76 |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
77 try: |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
78 write_to_log(repo, log_entry) |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
79 except (IOError, error.LockError) as err: |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
80 msg = stringutil.forcebytestr(err) |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
81 repo.ui.warn(_(b'unable to append to pull log: %s\n') % msg) |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
82 |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
83 return orig(repo, proto, others, *args, **kwargs) |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
84 |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
85 |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
86 def extract_pull_heads(bundle_args): |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
87 opts = wireprotov1server.options( |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
88 b'getbundle', |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
89 wireprotov1server.wireprototypes.GETBUNDLE_ARGUMENTS.keys(), |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
90 bundle_args.copy(), # this call consumes the args destructively |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
91 ) |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
92 |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
93 heads = opts.get(b'heads', b'').decode('utf-8').split(' ') |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
94 common = opts.get(b'common', b'').decode('utf-8').split(' ') |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
95 return (heads, common) |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
96 |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
97 |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
98 def write_to_log(repo, entry): |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
99 locktimeout = repo.ui.configint(EXT_NAME, b'timeout.lock') |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
100 lockwarntimeout = repo.ui.configint(EXT_NAME, b'timeout.warn') |
49509
946c023212b8
pull_logger: add basic log file rotation based on size
pacien <pacien.trangirard@pacien.net>
parents:
49508
diff
changeset
|
101 rotatesize = repo.ui.configbytes(EXT_NAME, b'rotate-size') |
49508
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
102 |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
103 with lock.trylock( |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
104 ui=repo.ui, |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
105 vfs=repo.vfs, |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
106 lockname=LOCK_NAME, |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
107 timeout=locktimeout, |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
108 warntimeout=lockwarntimeout, |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
109 ): |
49509
946c023212b8
pull_logger: add basic log file rotation based on size
pacien <pacien.trangirard@pacien.net>
parents:
49508
diff
changeset
|
110 if rotatesize > 0 and repo.vfs.exists(LOG_FILE): |
946c023212b8
pull_logger: add basic log file rotation based on size
pacien <pacien.trangirard@pacien.net>
parents:
49508
diff
changeset
|
111 if repo.vfs.stat(LOG_FILE).st_size >= rotatesize: |
946c023212b8
pull_logger: add basic log file rotation based on size
pacien <pacien.trangirard@pacien.net>
parents:
49508
diff
changeset
|
112 repo.vfs.rename(LOG_FILE, OLD_LOG_FILE) |
946c023212b8
pull_logger: add basic log file rotation based on size
pacien <pacien.trangirard@pacien.net>
parents:
49508
diff
changeset
|
113 |
49508
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
114 with repo.vfs.open(LOG_FILE, b'a+') as logfile: |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
115 serialised = json.dumps(entry, sort_keys=True) |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
116 logfile.write(serialised.encode('utf-8')) |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
117 logfile.write(b'\n') |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
118 logfile.flush() |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
119 |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
120 |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
121 def reposetup(ui, repo): |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
122 if repo.local(): |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
123 repo._wlockfreeprefix.add(LOG_FILE) |
49509
946c023212b8
pull_logger: add basic log file rotation based on size
pacien <pacien.trangirard@pacien.net>
parents:
49508
diff
changeset
|
124 repo._wlockfreeprefix.add(OLD_LOG_FILE) |
49508
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
125 |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
126 |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
127 def uisetup(ui): |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
128 del wireprotov1server.commands[b'getbundle'] |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
129 decorator = wireprotov1server.wireprotocommand( |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
130 name=b'getbundle', |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
131 args=b'*', |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
132 permission=b'pull', |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
133 ) |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
134 |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
135 extensions.wrapfunction( |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
136 container=wireprotov1server, |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
137 funcname='getbundle', |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
138 wrapper=wrap_getbundle, |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
139 ) |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
140 |
791050360486
contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff
changeset
|
141 decorator(wireprotov1server.getbundle) |