streamclone: use backgroundfilecloser (issue4889)
Closing files that have been appended to is slow on Windows/NTFS.
CloseHandle() calls on this platform often take 1-10ms - and that's
on my i7-6700K Skylake processor with a modern and fast SSD. Contrast
with other I/O operations, such as writing data, which take <100us.
This means that creating/appending thousands of files can add
significant overhead. For example, cloning mozilla-central creates
~232,000 revlog files. Assuming 1ms per CloseHandle(), that yields
232s (3:52) of wall time waiting for file closes!
The impact of this overhead can be measured most directly when applying
stream clone bundles. Applying these files is effectively uncompressing
a tar archive (read: it's very fast).
Using a RAM disk (read: no I/O wait), the difference in wall time for a
`hg debugapplystreamclonebundle` for a ~1731 MB mozilla-central bundle
between Windows and Linux from the same machine is drastic:
Linux: ~12.8s (128MB/s)
Windows: ~352.0s (4.7MB/s)
Windows is ~27.5x slower. Yikes!
After this patch:
Linux: ~12.8s (128MB/s)
Windows: ~102.1s (16.1MB/s)
Windows is now ~3.4x faster. Unfortunately, it is still ~8x slower than
Linux. Profiling reveals a few hot code paths that could likely be
improved. But those are for other patches.
This patch introduces test-clone-uncompressed.t because existing tests
of `clone --uncompressed` are scattered about and adding a variation for
background thread closing to e.g. test-http.t doesn't feel correct.
#require execbit
$ umask 027
$ hg init test1
$ cd test1
$ touch a b
$ hg add a b
$ hg ci -m "added a b"
$ cd ..
$ hg clone test1 test3
updating to branch default
2 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ hg init test2
$ cd test2
$ hg pull ../test1
pulling from ../test1
requesting all changes
adding changesets
adding manifests
adding file changes
added 1 changesets with 2 changes to 2 files
(run 'hg update' to get a working copy)
$ hg co
2 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ chmod +x a
$ hg ci -m "chmod +x a"
the changelog should mention file a:
$ hg tip --template '{files}\n'
a
$ cd ../test1
$ echo 123 >>a
$ hg ci -m "a updated"
$ hg pull ../test2
pulling from ../test2
searching for changes
adding changesets
adding manifests
adding file changes
added 1 changesets with 0 changes to 0 files (+1 heads)
(run 'hg heads' to see heads, 'hg merge' to merge)
$ hg heads
changeset: 2:7f4313b42a34
tag: tip
parent: 0:22a449e20da5
user: test
date: Thu Jan 01 00:00:00 1970 +0000
summary: chmod +x a
changeset: 1:c6ecefc45368
user: test
date: Thu Jan 01 00:00:00 1970 +0000
summary: a updated
$ hg history
changeset: 2:7f4313b42a34
tag: tip
parent: 0:22a449e20da5
user: test
date: Thu Jan 01 00:00:00 1970 +0000
summary: chmod +x a
changeset: 1:c6ecefc45368
user: test
date: Thu Jan 01 00:00:00 1970 +0000
summary: a updated
changeset: 0:22a449e20da5
user: test
date: Thu Jan 01 00:00:00 1970 +0000
summary: added a b
$ hg -v merge
resolving manifests
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
(branch merge, don't forget to commit)
$ cat a
123
$ [ -x a ]
$ cd ../test3
$ echo 123 >>b
$ hg ci -m "b updated"
$ hg pull ../test2
pulling from ../test2
searching for changes
adding changesets
adding manifests
adding file changes
added 1 changesets with 0 changes to 0 files (+1 heads)
(run 'hg heads' to see heads, 'hg merge' to merge)
$ hg heads
changeset: 2:7f4313b42a34
tag: tip
parent: 0:22a449e20da5
user: test
date: Thu Jan 01 00:00:00 1970 +0000
summary: chmod +x a
changeset: 1:dc57ead75f79
user: test
date: Thu Jan 01 00:00:00 1970 +0000
summary: b updated
$ hg history
changeset: 2:7f4313b42a34
tag: tip
parent: 0:22a449e20da5
user: test
date: Thu Jan 01 00:00:00 1970 +0000
summary: chmod +x a
changeset: 1:dc57ead75f79
user: test
date: Thu Jan 01 00:00:00 1970 +0000
summary: b updated
changeset: 0:22a449e20da5
user: test
date: Thu Jan 01 00:00:00 1970 +0000
summary: added a b
$ hg -v merge
resolving manifests
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
(branch merge, don't forget to commit)
$ ls -l ../test[123]/a > foo
$ cut -b 1-10 < foo
-rwxr-x---
-rwxr-x---
-rwxr-x---
$ hg debugindex a
rev offset length ..... linkrev nodeid p1 p2 (re)
0 0 0 ..... 0 b80de5d13875 000000000000 000000000000 (re)
$ hg debugindex -R ../test2 a
rev offset length ..... linkrev nodeid p1 p2 (re)
0 0 0 ..... 0 b80de5d13875 000000000000 000000000000 (re)
$ hg debugindex -R ../test1 a
rev offset length ..... linkrev nodeid p1 p2 (re)
0 0 0 ..... 0 b80de5d13875 000000000000 000000000000 (re)
1 0 5 ..... 1 7fe919cc0336 b80de5d13875 000000000000 (re)
$ cd ..