Mercurial > hg
annotate tests/test-fix-pickle.t @ 45390:7d24201b6447
worker: don't expose readinto() on _blockingreader since pickle is picky
The `pickle` module expects the input to be buffered and a whole
object to be available when `pickle.load()` is called, which is not
necessarily true when we send data from workers back to the parent
process (i.e., it seems like a bad assumption for the `pickle` module
to make). We added a workaround for that in
https://phab.mercurial-scm.org/D8076, which made `read()` continue
until all the requested bytes have been read.
As we found out at work after a lot of investigation (I've spent the
last two days on this), the native version of `pickle.load()` has
started calling `readinto()` on the input since Python 3.8. That
started being called in
https://github.com/python/cpython/commit/91f4380cedbae32b49adbea2518014a5624c6523
(and only by the C version of `pickle.load()`)). Before that, it was
only `read()` and `readline()` that were called. The problem with that
was that `readinto()` on our `_blockingreader` was simply delegating
to the underlying, *unbuffered* object. The symptom we saw was that
`hg fix` started failing sometimes on Python 3.8 on Mac. It failed
very relyable in some cases. I still haven't figured out under what
circumstances it fails and I've been unable to reproduce it in test
cases (I've tried writing larger amounts of data, using different
numbers of workers, and making the formatters sleep). I have, however,
been able to reproduce it 3-4 times on Linux, but then it stopped
reproducing on the following few hundred attempts.
To fix the problem, we can simply remove the implementation of
`readinto()`, since the unpickler will then fall back to calling
`read()`. The fallback was added a bit later, in
https://github.com/python/cpython/commit/b19f7ecfa3adc6ba1544225317b9473649815b38. However,
that commit also added checking that what `read()` returns is a
`bytes`, so we also need to convert the `bytearray` we use into
that. I was able to add a test for that failure at least.
Differential Revision: https://phab.mercurial-scm.org/D8928
author | Martin von Zweigbergk <martinvonz@google.com> |
---|---|
date | Fri, 14 Aug 2020 20:45:49 -0700 |
parents | |
children |
rev | line source |
---|---|
45390
7d24201b6447
worker: don't expose readinto() on _blockingreader since pickle is picky
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
1 A script that implements uppercasing all letters in a file. |
7d24201b6447
worker: don't expose readinto() on _blockingreader since pickle is picky
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
2 |
7d24201b6447
worker: don't expose readinto() on _blockingreader since pickle is picky
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
3 $ UPPERCASEPY="$TESTTMP/uppercase.py" |
7d24201b6447
worker: don't expose readinto() on _blockingreader since pickle is picky
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
4 $ cat > $UPPERCASEPY <<EOF |
7d24201b6447
worker: don't expose readinto() on _blockingreader since pickle is picky
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
5 > import sys |
7d24201b6447
worker: don't expose readinto() on _blockingreader since pickle is picky
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
6 > from mercurial.utils.procutil import setbinary |
7d24201b6447
worker: don't expose readinto() on _blockingreader since pickle is picky
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
7 > setbinary(sys.stdin) |
7d24201b6447
worker: don't expose readinto() on _blockingreader since pickle is picky
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
8 > setbinary(sys.stdout) |
7d24201b6447
worker: don't expose readinto() on _blockingreader since pickle is picky
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
9 > sys.stdout.write(sys.stdin.read().upper()) |
7d24201b6447
worker: don't expose readinto() on _blockingreader since pickle is picky
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
10 > EOF |
7d24201b6447
worker: don't expose readinto() on _blockingreader since pickle is picky
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
11 $ TESTLINES="foo\nbar\nbaz\n" |
7d24201b6447
worker: don't expose readinto() on _blockingreader since pickle is picky
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
12 $ printf $TESTLINES | "$PYTHON" $UPPERCASEPY |
7d24201b6447
worker: don't expose readinto() on _blockingreader since pickle is picky
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
13 FOO |
7d24201b6447
worker: don't expose readinto() on _blockingreader since pickle is picky
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
14 BAR |
7d24201b6447
worker: don't expose readinto() on _blockingreader since pickle is picky
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
15 BAZ |
7d24201b6447
worker: don't expose readinto() on _blockingreader since pickle is picky
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
16 |
7d24201b6447
worker: don't expose readinto() on _blockingreader since pickle is picky
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
17 This file attempts to test our workarounds for pickle's lack of |
7d24201b6447
worker: don't expose readinto() on _blockingreader since pickle is picky
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
18 support for short reads. |
7d24201b6447
worker: don't expose readinto() on _blockingreader since pickle is picky
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
19 |
7d24201b6447
worker: don't expose readinto() on _blockingreader since pickle is picky
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
20 $ cat >> $HGRCPATH <<EOF |
7d24201b6447
worker: don't expose readinto() on _blockingreader since pickle is picky
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
21 > [extensions] |
7d24201b6447
worker: don't expose readinto() on _blockingreader since pickle is picky
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
22 > fix = |
7d24201b6447
worker: don't expose readinto() on _blockingreader since pickle is picky
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
23 > [fix] |
7d24201b6447
worker: don't expose readinto() on _blockingreader since pickle is picky
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
24 > uppercase-whole-file:command="$PYTHON" $UPPERCASEPY |
7d24201b6447
worker: don't expose readinto() on _blockingreader since pickle is picky
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
25 > uppercase-whole-file:pattern=set:** |
7d24201b6447
worker: don't expose readinto() on _blockingreader since pickle is picky
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
26 > EOF |
7d24201b6447
worker: don't expose readinto() on _blockingreader since pickle is picky
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
27 |
7d24201b6447
worker: don't expose readinto() on _blockingreader since pickle is picky
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
28 $ hg init repo |
7d24201b6447
worker: don't expose readinto() on _blockingreader since pickle is picky
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
29 $ cd repo |
7d24201b6447
worker: don't expose readinto() on _blockingreader since pickle is picky
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
30 |
7d24201b6447
worker: don't expose readinto() on _blockingreader since pickle is picky
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
31 # Create a file that's large enough that it seems to not fit in |
7d24201b6447
worker: don't expose readinto() on _blockingreader since pickle is picky
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
32 # pickle's buffer, making it use the code path that expects our |
7d24201b6447
worker: don't expose readinto() on _blockingreader since pickle is picky
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
33 # _blockingreader's read() method to return bytes. |
7d24201b6447
worker: don't expose readinto() on _blockingreader since pickle is picky
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
34 $ echo "some stuff" > file |
7d24201b6447
worker: don't expose readinto() on _blockingreader since pickle is picky
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
35 $ for i in $($TESTDIR/seq.py 13); do |
7d24201b6447
worker: don't expose readinto() on _blockingreader since pickle is picky
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
36 > cat file file > tmp |
7d24201b6447
worker: don't expose readinto() on _blockingreader since pickle is picky
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
37 > mv -f tmp file |
7d24201b6447
worker: don't expose readinto() on _blockingreader since pickle is picky
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
38 > done |
7d24201b6447
worker: don't expose readinto() on _blockingreader since pickle is picky
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
39 $ hg commit -Am "add large file" |
7d24201b6447
worker: don't expose readinto() on _blockingreader since pickle is picky
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
40 adding file |
7d24201b6447
worker: don't expose readinto() on _blockingreader since pickle is picky
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
41 |
7d24201b6447
worker: don't expose readinto() on _blockingreader since pickle is picky
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
42 Check that we don't get a crash |
7d24201b6447
worker: don't expose readinto() on _blockingreader since pickle is picky
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
43 |
7d24201b6447
worker: don't expose readinto() on _blockingreader since pickle is picky
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
44 $ hg fix -r . |
7d24201b6447
worker: don't expose readinto() on _blockingreader since pickle is picky
Martin von Zweigbergk <martinvonz@google.com>
parents:
diff
changeset
|
45 saved backup bundle to $TESTTMP/repo/.hg/strip-backup/*-fix.hg (glob) |