contrib/packagelib.sh

streamclone: use backgroundfilecloser (issue4889)

Closing files that have been appended to is slow on Windows/NTFS.
CloseHandle() calls on this platform often take 1-10ms - and that's
on my i7-6700K Skylake processor with a modern and fast SSD. Contrast
with other I/O operations, such as writing data, which take <100us.

This means that creating/appending thousands of files can add
significant overhead. For example, cloning mozilla-central creates
~232,000 revlog files. Assuming 1ms per CloseHandle(), that yields
232s (3:52) of wall time waiting for file closes!

The impact of this overhead can be measured most directly when applying
stream clone bundles. Applying these files is effectively uncompressing
a tar archive (read: it's very fast).

Using a RAM disk (read: no I/O wait), the difference in wall time for a
`hg debugapplystreamclonebundle` for a ~1731 MB mozilla-central bundle
between Windows and Linux from the same machine is drastic:

Linux:    ~12.8s (128MB/s)
Windows: ~352.0s (4.7MB/s)

Windows is ~27.5x slower. Yikes!

After this patch:

Linux:    ~12.8s (128MB/s)
Windows: ~102.1s (16.1MB/s)

Windows is now ~3.4x faster. Unfortunately, it is still ~8x slower than
Linux. Profiling reveals a few hot code paths that could likely be
improved. But those are for other patches.

This patch introduces test-clone-uncompressed.t because existing tests
of `clone --uncompressed` are scattered about and adding a variation for
background thread closing to e.g. test-http.t doesn't feel correct.

# Extract version number into 4 parts, some of which may be empty:
#
# version: the numeric part of the most recent tag. Will always look like 1.3.
#
# type: if an rc build, "rc", otherwise empty
#
# distance: the distance from the nearest tag, or empty if built from a tag
#
# node: the node|short hg was built from, or empty if built from a tag
gethgversion() {
    make clean
    make local || make local PURE=--pure
    HG="$PWD/hg"

    $HG version > /dev/null || { echo 'abort: hg version failed!'; exit 1 ; }

    hgversion=`$HG version | sed -ne 's/.*(version \(.*\))$/\1/p'`

    if echo $hgversion | grep + > /dev/null 2>&1 ; then
        tmp=`echo $hgversion | cut -d+ -f 2`
        hgversion=`echo $hgversion | cut -d+ -f 1`
        distance=`echo $tmp | cut -d- -f 1`
        node=`echo $tmp | cut -d- -f 2`
    else
        distance=''
        node=''
    fi
    if echo $hgversion | grep -- '-' > /dev/null 2>&1; then
        version=`echo $hgversion | cut -d- -f1`
        type=`echo $hgversion | cut -d- -f2`
    else
        version=$hgversion
        type=''
    fi
}