contrib/fuzz/fncache.cc

procutil: avoid using os.fork() to implement runbgcommand

We ran into the following deadlock:

- some command creates an ssh peer, then raises without explicitly
  closing the peer (hg id + extension in our case)

- dispatch catches the exception, calls ui.log('commandfinish', ..)
  (the sshpeer is still not closed), which calls logtoprocess, which
  calls procutil.runbgcommand.

- in the child of runbgcommand's fork(), between the fork and the
  exec, the opening of file descriptors triggers a gc which runs the
  destructor for sshpeer, which waits on ssh's stderr being closed,
  which never happens since ssh's stderr is held open by the parent of
  the fork where said destructor hasn't run

Remotefilelog appears to have a hack around this deadlock as well.

I don't know if there's more subtlety to it, because even though the
problem is determistic, it is very fragile, so I didn't manage to
reduce it.

I can imagine three ways of tackling this problem:
1. don't run any python between fork and exec in runbgcommand
2. make the finalizer harmless after the fork
3. close the peer without relying on gc behavior

This commit goes with 1, as forking without exec'ing is tricky in
general in a language with gc finalizers. And maybe it's better in the
presence of rust threads. A future commit will try 2 or 3.

Performance wise: at low memory usage, it's an improvement. At higher
memory usage, it's about 2x faster than before when ensurestart=True,
but 2x slower when ensurestart=False. Not sure if that matters. The
reason for that last bit is that the subprocess.Popen always waits for
the execve to finish, and at high memory usage, execve is slow because
it deallocates the large page table. Numbers and script:

				 before after
    mem=1.0GB, ensurestart=True  52.1ms 26.0ms
    mem=1.0GB, ensurestart=False 14.7ms 26.0ms
    mem=0.5GB, ensurestart=True  23.2ms 11.2ms
    mem=0.5GB, ensurestart=False  6.2ms 11.3ms
    mem=0.2GB, ensurestart=True  15.7ms 7.4ms
    mem=0.2GB, ensurestart=False  4.3ms 8.1ms
    mem=0.0GB, ensurestart=True   2.3ms 0.7ms
    mem=0.0GB, ensurestart=False  0.8ms 0.8ms

    import time
    for memsize in [1_000_000_000, 500_000_000, 250_000_000, 0]:
        mem = 'a' * memsize
        for ensurestart in [True, False]:
            now = time.time()
            n = 100
            for i in range(n):
                procutil.runbgcommand([b'true'], {}, ensurestart=ensurestart)
            after = time.time()
            ms = (after - now) / float(n) * 1000
            print(f'mem={memsize / 1e9:.1f}GB, ensurestart={ensurestart} -> {ms:.1f}ms')

Differential Revision: https://phab.mercurial-scm.org/D9019

#include <Python.h>
#include <assert.h>
#include <stdlib.h>
#include <unistd.h>

#include "pyutil.h"

#include <iostream>
#include <string>

extern "C" {

static PYCODETYPE *code;

extern "C" int LLVMFuzzerInitialize(int *argc, char ***argv)
{
	contrib::initpy(*argv[0]);
	code = (PYCODETYPE *)Py_CompileString(R"py(
try:
    for fn in (
        parsers.isasciistr,
        parsers.asciilower,
        parsers.asciiupper,
        parsers.encodedir,
        parsers.pathencode,
        parsers.lowerencode,
    ):
        try:
            fn(data)
        except UnicodeDecodeError:
            pass  # some functions emit this exception
        except AttributeError:
            # pathencode needs hashlib, which fails to import because the time
            # module fails to import. We should try and fix that some day, but
            # for now we at least get coverage on non-hashencoded codepaths.
            if fn != pathencode:
                raise
        # uncomment this for debugging exceptions
        # except Exception as e:
        #     raise Exception('%r: %r' % (fn, e))
except Exception as e:
    pass
    # uncomment this print if you're editing this Python code
    # to debug failures.
    # print(e)
)py",
	                                      "fuzzer", Py_file_input);
	if (!code) {
		std::cerr << "failed to compile Python code!" << std::endl;
	}
	return 0;
}

int LLVMFuzzerTestOneInput(const uint8_t *Data, size_t Size)
{
	PyObject *mtext =
	    PyBytes_FromStringAndSize((const char *)Data, (Py_ssize_t)Size);
	PyObject *locals = PyDict_New();
	PyDict_SetItemString(locals, "data", mtext);
	PyObject *res = PyEval_EvalCode(code, contrib::pyglobals(), locals);
	if (!res) {
		PyErr_Print();
	}
	Py_XDECREF(res);
	Py_DECREF(locals);
	Py_DECREF(mtext);
	return 0; // Non-zero return values are reserved for future use.
}
}