view hgeditor @ 44226:cb52e619c99e stable

worker: Use buffered input from the pickle stream

On Python 3, "pickle.load" will raise an exception ("_pickle.UnpicklingError:
pickle data was truncated") when it gets a short read, i.e. it receives fewer
bytes than it requested.

On our build machine, Mercurial seems to frequently hit this problem while
updating a mozilla-central clone iff it gets scheduled in batch mode. It is easy
to trigger with:

    #wipe the workdir
    rm -rf *
    hg update null

    chrt -b 0 hg update default

I've also written the following program, which demonstrates the core problem:

    from __future__ import print_function

    import io
    import os
    import pickle
    import time

    obj = {"a": 1, "b": 2}
    obj_data = pickle.dumps(obj)
    assert len(obj_data) > 10

    rfd, wfd = os.pipe()

    pid = os.fork()
    if pid == 0:
        os.close(rfd)

        for _ in range(4):
            time.sleep(0.5)
            print("First write")
            os.write(wfd, obj_data[:10])

            time.sleep(0.5)
            print("Second write")
            os.write(wfd, obj_data[10:])

        os._exit(0)

    try:
        os.close(wfd)

        rfile = os.fdopen(rfd, "rb", 0)

        print("Reading")
        while True:
            try:
                obj_copy = pickle.load(rfile)
                assert obj == obj_copy
            except EOFError:
                break
        print("Success")
    finally:
        os.kill(pid, 15)

The program reliably fails with Python 3.8 and succeeds with Python 2.7.

Providing the unpickler with a buffered reader fixes the issue, so let
"os.fdopen" create one.

https://bugzilla.mozilla.org/show_bug.cgi?id=1604486

Differential Revision: https://phab.mercurial-scm.org/D8051

#!/bin/sh
#
# This is an example of using HGEDITOR to create of diff to review the
# changes while committing.

# If you want to pass your favourite editor some other parameters
# only for Mercurial, modify this:
case "${EDITOR}" in
    "")
        EDITOR="vi"
        ;;
    emacs)
        EDITOR="$EDITOR -nw"
        ;;
    gvim|vim)
        EDITOR="$EDITOR -f -o"
        ;;
esac


HGTMP=""
cleanup_exit() {
    rm -rf "$HGTMP"
}

# Remove temporary files even if we get interrupted
trap "cleanup_exit" 0 # normal exit
trap "exit 255" HUP INT QUIT ABRT TERM

HGTMP=$(mktemp -d ${TMPDIR-/tmp}/hgeditor.XXXXXX)
[ x$HGTMP != x -a -d $HGTMP ] || {
  echo "Could not create temporary directory! Exiting." 1>&2
  exit 1
}

(
    grep '^HG: changed' "$1" | cut -b 13- | while read changed; do
        "$HG" diff "$changed" >> "$HGTMP/diff"
    done
)

cat "$1" > "$HGTMP/msg"

MD5=$(which md5sum 2>/dev/null) || \
    MD5=$(which md5 2>/dev/null)
[ -x "${MD5}" ] && CHECKSUM=`${MD5} "$HGTMP/msg"`
if [ -s "$HGTMP/diff" ]; then
    $EDITOR "$HGTMP/msg" "$HGTMP/diff" || exit $?
else
    $EDITOR "$HGTMP/msg" || exit $?
fi
[ -x "${MD5}" ] && (echo "$CHECKSUM" | ${MD5} -c >/dev/null 2>&1 && exit 13)

mv "$HGTMP/msg" "$1"

exit $?