Mercurial > hg
annotate contrib/memory.py @ 45095:8e04607023e5
procutil: ensure that procutil.std{out,err}.write() writes all bytes
Python 3 offers different kind of streams and it’s not guaranteed for all of
them that calling write() writes all bytes.
When Python is started in unbuffered mode, sys.std{out,err}.buffer are
instances of io.FileIO, whose write() can write less bytes for
platform-specific reasons (e.g. Linux has a 0x7ffff000 bytes maximum and could
write less if interrupted by a signal; when writing to Windows consoles, it’s
limited to 32767 bytes to avoid the "not enough space" error). This can lead to
silent loss of data, both when using sys.std{out,err}.buffer (which may in fact
not be a buffered stream) and when using the text streams sys.std{out,err}
(I’ve created a CPython bug report for that:
https://bugs.python.org/issue41221).
Python may fix the problem at some point. For now, we implement our own wrapper
for procutil.std{out,err} that calls the raw stream’s write() method until all
bytes have been written. We don’t use sys.std{out,err} for larger writes, so I
think it’s not worth the effort to patch them.
author | Manuel Jacob <me@manueljacob.de> |
---|---|
date | Fri, 10 Jul 2020 12:27:58 +0200 |
parents | 2372284d9457 |
children | d4ba4d51f85f |
rev | line source |
---|---|
10017
253d0da256b2
contrib: helper extension to track memory usage
Martin Geisler <mg@lazybytes.net>
parents:
diff
changeset
|
1 # memory.py - track memory usage |
253d0da256b2
contrib: helper extension to track memory usage
Martin Geisler <mg@lazybytes.net>
parents:
diff
changeset
|
2 # |
253d0da256b2
contrib: helper extension to track memory usage
Martin Geisler <mg@lazybytes.net>
parents:
diff
changeset
|
3 # Copyright 2009 Matt Mackall <mpm@selenic.com> and others |
253d0da256b2
contrib: helper extension to track memory usage
Martin Geisler <mg@lazybytes.net>
parents:
diff
changeset
|
4 # |
253d0da256b2
contrib: helper extension to track memory usage
Martin Geisler <mg@lazybytes.net>
parents:
diff
changeset
|
5 # This software may be used and distributed according to the terms of the |
10264 | 6 # GNU General Public License version 2 or any later version. |
10017
253d0da256b2
contrib: helper extension to track memory usage
Martin Geisler <mg@lazybytes.net>
parents:
diff
changeset
|
7 |
253d0da256b2
contrib: helper extension to track memory usage
Martin Geisler <mg@lazybytes.net>
parents:
diff
changeset
|
8 '''helper extension to measure memory usage |
253d0da256b2
contrib: helper extension to track memory usage
Martin Geisler <mg@lazybytes.net>
parents:
diff
changeset
|
9 |
253d0da256b2
contrib: helper extension to track memory usage
Martin Geisler <mg@lazybytes.net>
parents:
diff
changeset
|
10 Reads current and peak memory usage from ``/proc/self/status`` and |
253d0da256b2
contrib: helper extension to track memory usage
Martin Geisler <mg@lazybytes.net>
parents:
diff
changeset
|
11 prints it to ``stderr`` on exit. |
253d0da256b2
contrib: helper extension to track memory usage
Martin Geisler <mg@lazybytes.net>
parents:
diff
changeset
|
12 ''' |
253d0da256b2
contrib: helper extension to track memory usage
Martin Geisler <mg@lazybytes.net>
parents:
diff
changeset
|
13 |
28510
ade330deb39a
contrib: make memory.py use absolute_import
Pulkit Goyal <7895pulkit@gmail.com>
parents:
27795
diff
changeset
|
14 from __future__ import absolute_import |
10017
253d0da256b2
contrib: helper extension to track memory usage
Martin Geisler <mg@lazybytes.net>
parents:
diff
changeset
|
15 |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
31958
diff
changeset
|
16 |
10017
253d0da256b2
contrib: helper extension to track memory usage
Martin Geisler <mg@lazybytes.net>
parents:
diff
changeset
|
17 def memusage(ui): |
253d0da256b2
contrib: helper extension to track memory usage
Martin Geisler <mg@lazybytes.net>
parents:
diff
changeset
|
18 """Report memory usage of the current process.""" |
253d0da256b2
contrib: helper extension to track memory usage
Martin Geisler <mg@lazybytes.net>
parents:
diff
changeset
|
19 result = {'peak': 0, 'rss': 0} |
27795
3e0d27d298b7
with: use context manager for file I/O in memusage
Bryan O'Sullivan <bryano@fb.com>
parents:
10282
diff
changeset
|
20 with open('/proc/self/status', 'r') as status: |
10017
253d0da256b2
contrib: helper extension to track memory usage
Martin Geisler <mg@lazybytes.net>
parents:
diff
changeset
|
21 # This will only work on systems with a /proc file system |
253d0da256b2
contrib: helper extension to track memory usage
Martin Geisler <mg@lazybytes.net>
parents:
diff
changeset
|
22 # (like Linux). |
253d0da256b2
contrib: helper extension to track memory usage
Martin Geisler <mg@lazybytes.net>
parents:
diff
changeset
|
23 for line in status: |
253d0da256b2
contrib: helper extension to track memory usage
Martin Geisler <mg@lazybytes.net>
parents:
diff
changeset
|
24 parts = line.split() |
253d0da256b2
contrib: helper extension to track memory usage
Martin Geisler <mg@lazybytes.net>
parents:
diff
changeset
|
25 key = parts[0][2:-1].lower() |
253d0da256b2
contrib: helper extension to track memory usage
Martin Geisler <mg@lazybytes.net>
parents:
diff
changeset
|
26 if key in result: |
253d0da256b2
contrib: helper extension to track memory usage
Martin Geisler <mg@lazybytes.net>
parents:
diff
changeset
|
27 result[key] = int(parts[1]) |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
31958
diff
changeset
|
28 ui.write_err( |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
31958
diff
changeset
|
29 ", ".join( |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
31958
diff
changeset
|
30 ["%s: %.1f MiB" % (k, v / 1024.0) for k, v in result.iteritems()] |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
31958
diff
changeset
|
31 ) |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
31958
diff
changeset
|
32 + "\n" |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
31958
diff
changeset
|
33 ) |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
31958
diff
changeset
|
34 |
10017
253d0da256b2
contrib: helper extension to track memory usage
Martin Geisler <mg@lazybytes.net>
parents:
diff
changeset
|
35 |
253d0da256b2
contrib: helper extension to track memory usage
Martin Geisler <mg@lazybytes.net>
parents:
diff
changeset
|
36 def extsetup(ui): |
31958
de5c9d0e02ea
atexit: switch to home-grown implementation
Bryan O'Sullivan <bryano@fb.com>
parents:
30386
diff
changeset
|
37 ui.atexit(memusage, ui) |