util: add iterfile to workaround a fileobj.__iter__ issue with EINTR
The fileobj.__iter__ implementation in Python 2.7.12 (hg changeset
45d4cea97b04) is buggy: it cannot handle EINTR correctly.
In Objects/fileobject.c:
size_t Py_UniversalNewlineFread(....) {
....
if (!f->f_univ_newline)
return fread(buf, 1, n, stream);
....
}
According to the "fread" man page:
If an error occurs, or the end of the file is reached, the return value
is a short item count (or zero).
Therefore it's possible for "fread" (and "Py_UniversalNewlineFread") to
return a positive value while errno is set to EINTR and ferror(stream)
changes from zero to non-zero.
There are multiple "Py_UniversalNewlineFread": "file_read", "file_readinto",
"file_readlines", "readahead". While the first 3 have code to handle the
EINTR case, the last one "readahead" doesn't:
static int readahead(PyFileObject *f, Py_ssize_t bufsize) {
....
chunksize = Py_UniversalNewlineFread(
f->f_buf, bufsize, f->f_fp, (PyObject *)f);
....
if (chunksize == 0) {
if (ferror(f->f_fp)) {
PyErr_SetFromErrno(PyExc_IOError);
....
}
}
....
}
It means "readahead" could ignore EINTR, if "Py_UniversalNewlineFread"
returns a non-zero value. And at the next time "readahead" got executed, if
"Py_UniversalNewlineFread" returns 0, "readahead" would raise a Python error
without a incorrect errno - could be 0 - thus "IOError: [Errno 0] Error".
The only user of "readahead" is "readahead_get_line_skip".
The only user of "readahead_get_line_skip" is "file_iternext", aka.
"fileobj.__iter__", which should be avoided.
There are multiple places where the pattern "for x in fp" is used. This
patch adds a "iterfile" method in "util.py" so we can migrate our code from
"for x in fp" to "fox x in util.iterfile(fp)".
#!/usr/bin/env python
#
# Based on python's Tools/scripts/md5sum.py
#
# This software may be used and distributed according to the terms
# of the PYTHON SOFTWARE FOUNDATION LICENSE VERSION 2, which is
# GPL-compatible.
from __future__ import absolute_import
import os
import sys
try:
import hashlib
md5 = hashlib.md5
except ImportError:
import md5
md5 = md5.md5
try:
import msvcrt
msvcrt.setmode(sys.stdout.fileno(), os.O_BINARY)
msvcrt.setmode(sys.stderr.fileno(), os.O_BINARY)
except ImportError:
pass
for filename in sys.argv[1:]:
try:
fp = open(filename, 'rb')
except IOError as msg:
sys.stderr.write('%s: Can\'t open: %s\n' % (filename, msg))
sys.exit(1)
m = md5()
try:
for data in iter(lambda: fp.read(8192), ''):
m.update(data)
except IOError as msg:
sys.stderr.write('%s: I/O error: %s\n' % (filename, msg))
sys.exit(1)
sys.stdout.write('%s %s\n' % (m.hexdigest(), filename))
sys.exit(0)