view mercurial/dummycert.pem @ 45390:7d24201b6447

worker: don't expose readinto() on _blockingreader since pickle is picky

The `pickle` module expects the input to be buffered and a whole
object to be available when `pickle.load()` is called, which is not
necessarily true when we send data from workers back to the parent
process (i.e., it seems like a bad assumption for the `pickle` module
to make). We added a workaround for that in
https://phab.mercurial-scm.org/D8076, which made `read()` continue
until all the requested bytes have been read.

As we found out at work after a lot of investigation (I've spent the
last two days on this), the native version of `pickle.load()` has
started calling `readinto()` on the input since Python 3.8. That
started being called in
https://github.com/python/cpython/commit/91f4380cedbae32b49adbea2518014a5624c6523
(and only by the C version of `pickle.load()`)). Before that, it was
only `read()` and `readline()` that were called. The problem with that
was that `readinto()` on our `_blockingreader` was simply delegating
to the underlying, *unbuffered* object. The symptom we saw was that
`hg fix` started failing sometimes on Python 3.8 on Mac. It failed
very relyable in some cases. I still haven't figured out under what
circumstances it fails and I've been unable to reproduce it in test
cases (I've tried writing larger amounts of data, using different
numbers of workers, and making the formatters sleep). I have, however,
been able to reproduce it 3-4 times on Linux, but then it stopped
reproducing on the following few hundred attempts.

To fix the problem, we can simply remove the implementation of
`readinto()`, since the unpickler will then fall back to calling
`read()`. The fallback was added a bit later, in
https://github.com/python/cpython/commit/b19f7ecfa3adc6ba1544225317b9473649815b38. However,
that commit also added checking that what `read()` returns is a
`bytes`, so we also need to convert the `bytearray` we use into
that. I was able to add a test for that failure at least.

Differential Revision: https://phab.mercurial-scm.org/D8928

A dummy certificate that will make OS X 10.6+ Python use the system CA
certificate store:

-----BEGIN CERTIFICATE-----
MIIBIzCBzgIJANjmj39sb3FmMA0GCSqGSIb3DQEBBQUAMBkxFzAVBgNVBAMTDmhn
LmV4YW1wbGUuY29tMB4XDTE0MDgzMDA4NDU1OVoXDTE0MDgyOTA4NDU1OVowGTEX
MBUGA1UEAxMOaGcuZXhhbXBsZS5jb20wXDANBgkqhkiG9w0BAQEFAANLADBIAkEA
mh/ZySGlcq0ALNLmA1gZqt61HruywPrRk6WyrLJRgt+X7OP9FFlEfl2tzHfzqvmK
CtSQoPINWOdAJMekBYFgKQIDAQABMA0GCSqGSIb3DQEBBQUAA0EAF9h49LkSqJ6a
IlpogZuUHtihXeKZBsiktVIDlDccYsNy0RSh9XxUfhk+XMLw8jBlYvcltSXdJ7We
aKdQRekuMQ==
-----END CERTIFICATE-----

This certificate was generated to be syntactically valid but never be usable;
it expired before it became valid.

Created as:

  $ cat > cn.conf << EOT
  > [req]
  > distinguished_name = req_distinguished_name
  > [req_distinguished_name]
  > commonName = Common Name
  > commonName_default = no.example.com
  > EOT
  $ openssl req -nodes -new -x509 -keyout /dev/null \
  >   -out dummycert.pem -days -1 -config cn.conf -subj '/CN=hg.example.com'

To verify the content of this certificate:

  $ openssl x509 -in dummycert.pem -noout -text
  Certificate:
      Data:
          Version: 1 (0x0)
          Serial Number: 15629337334278746470 (0xd8e68f7f6c6f7166)
      Signature Algorithm: sha1WithRSAEncryption
          Issuer: CN=hg.example.com
          Validity
              Not Before: Aug 30 08:45:59 2014 GMT
              Not After : Aug 29 08:45:59 2014 GMT
          Subject: CN=hg.example.com
          Subject Public Key Info:
              Public Key Algorithm: rsaEncryption
                  Public-Key: (512 bit)
                  Modulus:
                      00:9a:1f:d9:c9:21:a5:72:ad:00:2c:d2:e6:03:58:
                      19:aa:de:b5:1e:bb:b2:c0:fa:d1:93:a5:b2:ac:b2:
                      51:82:df:97:ec:e3:fd:14:59:44:7e:5d:ad:cc:77:
                      f3:aa:f9:8a:0a:d4:90:a0:f2:0d:58:e7:40:24:c7:
                      a4:05:81:60:29
                  Exponent: 65537 (0x10001)
      Signature Algorithm: sha1WithRSAEncryption
           17:d8:78:f4:b9:12:a8:9e:9a:22:5a:68:81:9b:94:1e:d8:a1:
           5d:e2:99:06:c8:a4:b5:52:03:94:37:1c:62:c3:72:d1:14:a1:
           f5:7c:54:7e:19:3e:5c:c2:f0:f2:30:65:62:f7:25:b5:25:dd:
           27:b5:9e:68:a7:50:45:e9:2e:31