annotate contrib/memory.py @ 45095:8e04607023e5

procutil: ensure that procutil.std{out,err}.write() writes all bytes Python 3 offers different kind of streams and it’s not guaranteed for all of them that calling write() writes all bytes. When Python is started in unbuffered mode, sys.std{out,err}.buffer are instances of io.FileIO, whose write() can write less bytes for platform-specific reasons (e.g. Linux has a 0x7ffff000 bytes maximum and could write less if interrupted by a signal; when writing to Windows consoles, it’s limited to 32767 bytes to avoid the "not enough space" error). This can lead to silent loss of data, both when using sys.std{out,err}.buffer (which may in fact not be a buffered stream) and when using the text streams sys.std{out,err} (I’ve created a CPython bug report for that: https://bugs.python.org/issue41221). Python may fix the problem at some point. For now, we implement our own wrapper for procutil.std{out,err} that calls the raw stream’s write() method until all bytes have been written. We don’t use sys.std{out,err} for larger writes, so I think it’s not worth the effort to patch them.
author Manuel Jacob <me@manueljacob.de>
date Fri, 10 Jul 2020 12:27:58 +0200
parents 2372284d9457
children d4ba4d51f85f
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
10017
253d0da256b2 contrib: helper extension to track memory usage
Martin Geisler <mg@lazybytes.net>
parents:
diff changeset
1 # memory.py - track memory usage
253d0da256b2 contrib: helper extension to track memory usage
Martin Geisler <mg@lazybytes.net>
parents:
diff changeset
2 #
253d0da256b2 contrib: helper extension to track memory usage
Martin Geisler <mg@lazybytes.net>
parents:
diff changeset
3 # Copyright 2009 Matt Mackall <mpm@selenic.com> and others
253d0da256b2 contrib: helper extension to track memory usage
Martin Geisler <mg@lazybytes.net>
parents:
diff changeset
4 #
253d0da256b2 contrib: helper extension to track memory usage
Martin Geisler <mg@lazybytes.net>
parents:
diff changeset
5 # This software may be used and distributed according to the terms of the
10264
d6512b3e9ac0 Merge with stable
Matt Mackall <mpm@selenic.com>
parents: 10017
diff changeset
6 # GNU General Public License version 2 or any later version.
10017
253d0da256b2 contrib: helper extension to track memory usage
Martin Geisler <mg@lazybytes.net>
parents:
diff changeset
7
253d0da256b2 contrib: helper extension to track memory usage
Martin Geisler <mg@lazybytes.net>
parents:
diff changeset
8 '''helper extension to measure memory usage
253d0da256b2 contrib: helper extension to track memory usage
Martin Geisler <mg@lazybytes.net>
parents:
diff changeset
9
253d0da256b2 contrib: helper extension to track memory usage
Martin Geisler <mg@lazybytes.net>
parents:
diff changeset
10 Reads current and peak memory usage from ``/proc/self/status`` and
253d0da256b2 contrib: helper extension to track memory usage
Martin Geisler <mg@lazybytes.net>
parents:
diff changeset
11 prints it to ``stderr`` on exit.
253d0da256b2 contrib: helper extension to track memory usage
Martin Geisler <mg@lazybytes.net>
parents:
diff changeset
12 '''
253d0da256b2 contrib: helper extension to track memory usage
Martin Geisler <mg@lazybytes.net>
parents:
diff changeset
13
28510
ade330deb39a contrib: make memory.py use absolute_import
Pulkit Goyal <7895pulkit@gmail.com>
parents: 27795
diff changeset
14 from __future__ import absolute_import
10017
253d0da256b2 contrib: helper extension to track memory usage
Martin Geisler <mg@lazybytes.net>
parents:
diff changeset
15
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 31958
diff changeset
16
10017
253d0da256b2 contrib: helper extension to track memory usage
Martin Geisler <mg@lazybytes.net>
parents:
diff changeset
17 def memusage(ui):
253d0da256b2 contrib: helper extension to track memory usage
Martin Geisler <mg@lazybytes.net>
parents:
diff changeset
18 """Report memory usage of the current process."""
253d0da256b2 contrib: helper extension to track memory usage
Martin Geisler <mg@lazybytes.net>
parents:
diff changeset
19 result = {'peak': 0, 'rss': 0}
27795
3e0d27d298b7 with: use context manager for file I/O in memusage
Bryan O'Sullivan <bryano@fb.com>
parents: 10282
diff changeset
20 with open('/proc/self/status', 'r') as status:
10017
253d0da256b2 contrib: helper extension to track memory usage
Martin Geisler <mg@lazybytes.net>
parents:
diff changeset
21 # This will only work on systems with a /proc file system
253d0da256b2 contrib: helper extension to track memory usage
Martin Geisler <mg@lazybytes.net>
parents:
diff changeset
22 # (like Linux).
253d0da256b2 contrib: helper extension to track memory usage
Martin Geisler <mg@lazybytes.net>
parents:
diff changeset
23 for line in status:
253d0da256b2 contrib: helper extension to track memory usage
Martin Geisler <mg@lazybytes.net>
parents:
diff changeset
24 parts = line.split()
253d0da256b2 contrib: helper extension to track memory usage
Martin Geisler <mg@lazybytes.net>
parents:
diff changeset
25 key = parts[0][2:-1].lower()
253d0da256b2 contrib: helper extension to track memory usage
Martin Geisler <mg@lazybytes.net>
parents:
diff changeset
26 if key in result:
253d0da256b2 contrib: helper extension to track memory usage
Martin Geisler <mg@lazybytes.net>
parents:
diff changeset
27 result[key] = int(parts[1])
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 31958
diff changeset
28 ui.write_err(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 31958
diff changeset
29 ", ".join(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 31958
diff changeset
30 ["%s: %.1f MiB" % (k, v / 1024.0) for k, v in result.iteritems()]
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 31958
diff changeset
31 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 31958
diff changeset
32 + "\n"
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 31958
diff changeset
33 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 31958
diff changeset
34
10017
253d0da256b2 contrib: helper extension to track memory usage
Martin Geisler <mg@lazybytes.net>
parents:
diff changeset
35
253d0da256b2 contrib: helper extension to track memory usage
Martin Geisler <mg@lazybytes.net>
parents:
diff changeset
36 def extsetup(ui):
31958
de5c9d0e02ea atexit: switch to home-grown implementation
Bryan O'Sullivan <bryano@fb.com>
parents: 30386
diff changeset
37 ui.atexit(memusage, ui)