view CONTRIBUTORS @ 44155:cb52e619c99e stable

worker: Use buffered input from the pickle stream

On Python 3, "pickle.load" will raise an exception ("_pickle.UnpicklingError:
pickle data was truncated") when it gets a short read, i.e. it receives fewer
bytes than it requested.

On our build machine, Mercurial seems to frequently hit this problem while
updating a mozilla-central clone iff it gets scheduled in batch mode. It is easy
to trigger with:

    #wipe the workdir
    rm -rf *
    hg update null

    chrt -b 0 hg update default

I've also written the following program, which demonstrates the core problem:

    from __future__ import print_function

    import io
    import os
    import pickle
    import time

    obj = {"a": 1, "b": 2}
    obj_data = pickle.dumps(obj)
    assert len(obj_data) > 10

    rfd, wfd = os.pipe()

    pid = os.fork()
    if pid == 0:
        os.close(rfd)

        for _ in range(4):
            time.sleep(0.5)
            print("First write")
            os.write(wfd, obj_data[:10])

            time.sleep(0.5)
            print("Second write")
            os.write(wfd, obj_data[10:])

        os._exit(0)

    try:
        os.close(wfd)

        rfile = os.fdopen(rfd, "rb", 0)

        print("Reading")
        while True:
            try:
                obj_copy = pickle.load(rfile)
                assert obj == obj_copy
            except EOFError:
                break
        print("Success")
    finally:
        os.kill(pid, 15)

The program reliably fails with Python 3.8 and succeeds with Python 2.7.

Providing the unpickler with a buffered reader fixes the issue, so let
"os.fdopen" create one.

https://bugzilla.mozilla.org/show_bug.cgi?id=1604486

Differential Revision: https://phab.mercurial-scm.org/D8051

[This file is here for historical purposes, all recent contributors
should appear in the changelog directly]

Andrea Arcangeli <andrea at suse.de>
Thomas Arendsen Hein <thomas at intevation.de>
Goffredo Baroncelli <kreijack at libero.it>
Muli Ben-Yehuda <mulix at mulix.org>
Mikael Berthe <mikael at lilotux.net>
Benoit Boissinot <bboissin at gmail.com>
Brendan Cully <brendan at kublai.com>
Vincent Danjean <vdanjean.ml at free.fr>
Jake Edge <jake at edge2.net>
Michael Fetterman <michael.fetterman at intel.com>
Edouard Gomez <ed.gomez at free.fr>
Eric Hopper <hopper at omnifarious.org>
Alecs King <alecsk at gmail.com>
Volker Kleinfeld <Volker.Kleinfeld at gmx.de>
Vadim Lebedev <vadim at mbdsys.com>
Christopher Li <hg at chrisli.org>
Chris Mason <mason at suse.com>
Colin McMillen <mcmillen at cs.cmu.edu>
Wojciech Milkowski <wmilkowski at interia.pl>
Chad Netzer <chad.netzer at gmail.com>
Bryan O'Sullivan <bos at serpentine.com>
Vicent Seguí Pascual <vseguip at gmail.com>
Sean Perry <shaleh at speakeasy.net>
Nguyen Anh Quynh <aquynh at gmail.com>
Ollivier Robert <roberto at keltia.freenix.fr>
Alexander Schremmer <alex at alexanderweb.de>
Arun Sharma <arun at sharma-home.net>
Josef "Jeff" Sipek <jeffpc at optonline.net>
Kevin Smith <yarcs at qualitycode.com>
TK Soh <teekaysoh at yahoo.com>
Radoslaw Szkodzinski <astralstorm at gorzow.mm.pl>
Samuel Tardieu <sam at rfc1149.net>
K Thananchayan <thananck at yahoo.com>
Andrew Thompson <andrewkt at aktzero.com>
Michael S. Tsirkin <mst at mellanox.co.il>
Rafael Villar Burke <pachi at mmn-arquitectos.com>
Tristan Wibberley <tristan at wibberley.org>
Mark Williamson <mark.williamson at cl.cam.ac.uk>