streamclone: use backgroundfilecloser (issue4889)
Closing files that have been appended to is slow on Windows/NTFS.
CloseHandle() calls on this platform often take 1-10ms - and that's
on my i7-6700K Skylake processor with a modern and fast SSD. Contrast
with other I/O operations, such as writing data, which take <100us.
This means that creating/appending thousands of files can add
significant overhead. For example, cloning mozilla-central creates
~232,000 revlog files. Assuming 1ms per CloseHandle(), that yields
232s (3:52) of wall time waiting for file closes!
The impact of this overhead can be measured most directly when applying
stream clone bundles. Applying these files is effectively uncompressing
a tar archive (read: it's very fast).
Using a RAM disk (read: no I/O wait), the difference in wall time for a
`hg debugapplystreamclonebundle` for a ~1731 MB mozilla-central bundle
between Windows and Linux from the same machine is drastic:
Linux: ~12.8s (128MB/s)
Windows: ~352.0s (4.7MB/s)
Windows is ~27.5x slower. Yikes!
After this patch:
Linux: ~12.8s (128MB/s)
Windows: ~102.1s (16.1MB/s)
Windows is now ~3.4x faster. Unfortunately, it is still ~8x slower than
Linux. Profiling reveals a few hot code paths that could likely be
improved. But those are for other patches.
This patch introduces test-clone-uncompressed.t because existing tests
of `clone --uncompressed` are scattered about and adding a variation for
background thread closing to e.g. test-http.t doesn't feel correct.
$ hg init
$ echo foo > bar
$ hg commit -Am default
adding bar
$ hg up -r null
0 files updated, 0 files merged, 1 files removed, 0 files unresolved
$ hg branch mine
marked working directory as branch mine
(branches are permanent and global, did you want a bookmark?)
$ echo hello > world
$ hg commit -Am hello
adding world
$ hg up -r null
0 files updated, 0 files merged, 1 files removed, 0 files unresolved
$ hg branch other
marked working directory as branch other
$ echo good > bye
$ hg commit -Am other
adding bye
$ hg up -r mine
1 files updated, 0 files merged, 1 files removed, 0 files unresolved
$ hg clone -U -u . .#other ../b -r 0 -r 1 -r 2 -b other
abort: cannot specify both --noupdate and --updaterev
[255]
$ hg clone -U .#other ../b -r 0 -r 1 -r 2 -b other
adding changesets
adding manifests
adding file changes
added 3 changesets with 3 changes to 3 files (+2 heads)
$ rm -rf ../b
$ hg clone -u . .#other ../b -r 0 -r 1 -r 2 -b other
adding changesets
adding manifests
adding file changes
added 3 changesets with 3 changes to 3 files (+2 heads)
updating to branch mine
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ rm -rf ../b
$ hg clone -u 0 .#other ../b -r 0 -r 1 -r 2 -b other
adding changesets
adding manifests
adding file changes
added 3 changesets with 3 changes to 3 files (+2 heads)
updating to branch default
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ rm -rf ../b
$ hg clone -u 1 .#other ../b -r 0 -r 1 -r 2 -b other
adding changesets
adding manifests
adding file changes
added 3 changesets with 3 changes to 3 files (+2 heads)
updating to branch mine
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ rm -rf ../b
$ hg clone -u 2 .#other ../b -r 0 -r 1 -r 2 -b other
adding changesets
adding manifests
adding file changes
added 3 changesets with 3 changes to 3 files (+2 heads)
updating to branch other
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ rm -rf ../b
Test -r mine ... mine is ignored:
$ hg clone -u 2 .#other ../b -r mine -r 0 -r 1 -r 2 -b other
adding changesets
adding manifests
adding file changes
added 3 changesets with 3 changes to 3 files (+2 heads)
updating to branch other
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ rm -rf ../b
$ hg clone .#other ../b -b default -b mine
adding changesets
adding manifests
adding file changes
added 3 changesets with 3 changes to 3 files (+2 heads)
updating to branch default
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ rm -rf ../b
$ hg clone .#other ../b
adding changesets
adding manifests
adding file changes
added 1 changesets with 1 changes to 1 files
updating to branch other
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ rm -rf ../b
$ hg clone -U . ../c -r 1 -r 2 > /dev/null
$ hg clone ../c ../b
updating to branch other
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ rm -rf ../b ../c