streamclone: use backgroundfilecloser (issue4889)
Closing files that have been appended to is slow on Windows/NTFS.
CloseHandle() calls on this platform often take 1-10ms - and that's
on my i7-6700K Skylake processor with a modern and fast SSD. Contrast
with other I/O operations, such as writing data, which take <100us.
This means that creating/appending thousands of files can add
significant overhead. For example, cloning mozilla-central creates
~232,000 revlog files. Assuming 1ms per CloseHandle(), that yields
232s (3:52) of wall time waiting for file closes!
The impact of this overhead can be measured most directly when applying
stream clone bundles. Applying these files is effectively uncompressing
a tar archive (read: it's very fast).
Using a RAM disk (read: no I/O wait), the difference in wall time for a
`hg debugapplystreamclonebundle` for a ~1731 MB mozilla-central bundle
between Windows and Linux from the same machine is drastic:
Linux: ~12.8s (128MB/s)
Windows: ~352.0s (4.7MB/s)
Windows is ~27.5x slower. Yikes!
After this patch:
Linux: ~12.8s (128MB/s)
Windows: ~102.1s (16.1MB/s)
Windows is now ~3.4x faster. Unfortunately, it is still ~8x slower than
Linux. Profiling reveals a few hot code paths that could likely be
improved. But those are for other patches.
This patch introduces test-clone-uncompressed.t because existing tests
of `clone --uncompressed` are scattered about and adding a variation for
background thread closing to e.g. test-http.t doesn't feel correct.
$ unset HGUSER
$ EMAIL="My Name <myname@example.com>"
$ export EMAIL
$ hg init test
$ cd test
$ touch asdf
$ hg add asdf
$ hg commit -m commit-1
$ hg tip
changeset: 0:53f268a58230
tag: tip
user: My Name <myname@example.com>
date: Thu Jan 01 00:00:00 1970 +0000
summary: commit-1
$ unset EMAIL
$ echo 1234 > asdf
$ hg commit -u "foo@bar.com" -m commit-1
$ hg tip
changeset: 1:3871b2a9e9bf
tag: tip
user: foo@bar.com
date: Thu Jan 01 00:00:00 1970 +0000
summary: commit-1
$ echo "[ui]" >> .hg/hgrc
$ echo "username = foobar <foo@bar.com>" >> .hg/hgrc
$ echo 12 > asdf
$ hg commit -m commit-1
$ hg tip
changeset: 2:8eeac6695c1c
tag: tip
user: foobar <foo@bar.com>
date: Thu Jan 01 00:00:00 1970 +0000
summary: commit-1
$ echo 1 > asdf
$ hg commit -u "foo@bar.com" -m commit-1
$ hg tip
changeset: 3:957606a725e4
tag: tip
user: foo@bar.com
date: Thu Jan 01 00:00:00 1970 +0000
summary: commit-1
$ echo 123 > asdf
$ echo "[ui]" > .hg/hgrc
$ echo "username = " >> .hg/hgrc
$ hg commit -m commit-1
abort: no username supplied
(use "hg config --edit" to set your username)
[255]
# test alternate config var
$ echo 1234 > asdf
$ echo "[ui]" > .hg/hgrc
$ echo "user = Foo Bar II <foo2@bar.com>" >> .hg/hgrc
$ hg commit -m commit-1
$ hg tip
changeset: 4:6f24bfb4c617
tag: tip
user: Foo Bar II <foo2@bar.com>
date: Thu Jan 01 00:00:00 1970 +0000
summary: commit-1
# test no .hg/hgrc (uses generated non-interactive username)
$ echo space > asdf
$ rm .hg/hgrc
$ hg commit -m commit-1 2>&1
no username found, using '[^']*' instead (re)
$ echo space2 > asdf
$ hg commit -u ' ' -m commit-1
transaction abort!
rollback completed
abort: empty username!
[255]
# don't add tests here, previous test is unstable
$ cd ..