tests/test-symlink-os-yes-fs-no.py

streamclone: use backgroundfilecloser (issue4889)

Closing files that have been appended to is slow on Windows/NTFS.
CloseHandle() calls on this platform often take 1-10ms - and that's
on my i7-6700K Skylake processor with a modern and fast SSD. Contrast
with other I/O operations, such as writing data, which take <100us.

This means that creating/appending thousands of files can add
significant overhead. For example, cloning mozilla-central creates
~232,000 revlog files. Assuming 1ms per CloseHandle(), that yields
232s (3:52) of wall time waiting for file closes!

The impact of this overhead can be measured most directly when applying
stream clone bundles. Applying these files is effectively uncompressing
a tar archive (read: it's very fast).

Using a RAM disk (read: no I/O wait), the difference in wall time for a
`hg debugapplystreamclonebundle` for a ~1731 MB mozilla-central bundle
between Windows and Linux from the same machine is drastic:

Linux:    ~12.8s (128MB/s)
Windows: ~352.0s (4.7MB/s)

Windows is ~27.5x slower. Yikes!

After this patch:

Linux:    ~12.8s (128MB/s)
Windows: ~102.1s (16.1MB/s)

Windows is now ~3.4x faster. Unfortunately, it is still ~8x slower than
Linux. Profiling reveals a few hot code paths that could likely be
improved. But those are for other patches.

This patch introduces test-clone-uncompressed.t because existing tests
of `clone --uncompressed` are scattered about and adding a variation for
background thread closing to e.g. test-http.t doesn't feel correct.

import os, sys, time
from mercurial import hg, ui, commands, util

TESTDIR = os.environ["TESTDIR"]
BUNDLEPATH = os.path.join(TESTDIR, 'bundles', 'test-no-symlinks.hg')

# only makes sense to test on os which supports symlinks
if not getattr(os, "symlink", False):
    sys.exit(80) # SKIPPED_STATUS defined in run-tests.py

u = ui.ui()
# hide outer repo
hg.peer(u, {}, '.', create=True)

# clone with symlink support
hg.clone(u, {}, BUNDLEPATH, 'test0')

repo = hg.repository(u, 'test0')

# wait a bit, or the status call wont update the dirstate
time.sleep(1)
commands.status(u, repo)

# now disable symlink support -- this is what os.symlink would do on a
# non-symlink file system
def symlink_failure(src, dst):
    raise OSError(1, "Operation not permitted")
os.symlink = symlink_failure

# dereference links as if a Samba server has exported this to a
# Windows client
for f in 'test0/a.lnk', 'test0/d/b.lnk':
    os.unlink(f)
    fp = open(f, 'wb')
    fp.write(util.readfile(f[:-4]))
    fp.close()

# reload repository
u = ui.ui()
repo = hg.repository(u, 'test0')
commands.status(u, repo)

# try cloning a repo which contains symlinks
u = ui.ui()
hg.clone(u, {}, BUNDLEPATH, 'test1')