streamclone: use backgroundfilecloser (issue4889)
Closing files that have been appended to is slow on Windows/NTFS.
CloseHandle() calls on this platform often take 1-10ms - and that's
on my i7-6700K Skylake processor with a modern and fast SSD. Contrast
with other I/O operations, such as writing data, which take <100us.
This means that creating/appending thousands of files can add
significant overhead. For example, cloning mozilla-central creates
~232,000 revlog files. Assuming 1ms per CloseHandle(), that yields
232s (3:52) of wall time waiting for file closes!
The impact of this overhead can be measured most directly when applying
stream clone bundles. Applying these files is effectively uncompressing
a tar archive (read: it's very fast).
Using a RAM disk (read: no I/O wait), the difference in wall time for a
`hg debugapplystreamclonebundle` for a ~1731 MB mozilla-central bundle
between Windows and Linux from the same machine is drastic:
Linux: ~12.8s (128MB/s)
Windows: ~352.0s (4.7MB/s)
Windows is ~27.5x slower. Yikes!
After this patch:
Linux: ~12.8s (128MB/s)
Windows: ~102.1s (16.1MB/s)
Windows is now ~3.4x faster. Unfortunately, it is still ~8x slower than
Linux. Profiling reveals a few hot code paths that could likely be
improved. But those are for other patches.
This patch introduces test-clone-uncompressed.t because existing tests
of `clone --uncompressed` are scattered about and adding a variation for
background thread closing to e.g. test-http.t doesn't feel correct.
https://bz.mercurial-scm.org/522
In the merge below, the file "foo" has the same contents in both
parents, but if we look at the file-level history, we'll notice that
the version in p1 is an ancestor of the version in p2. This test makes
sure that we'll use the version from p2 in the manifest of the merge
revision.
$ hg init
$ echo foo > foo
$ hg ci -qAm 'add foo'
$ echo bar >> foo
$ hg ci -m 'change foo'
$ hg backout -r tip -m 'backout changed foo'
reverting foo
changeset 2:4d9e78aaceee backs out changeset 1:b515023e500e
$ hg up -C 0
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ touch bar
$ hg ci -qAm 'add bar'
$ hg merge --debug
searching for copies back to rev 1
unmatched files in local:
bar
resolving manifests
branchmerge: True, force: False, partial: False
ancestor: bbd179dfa0a7, local: 71766447bdbb+, remote: 4d9e78aaceee
foo: remote is newer -> g
getting foo
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
(branch merge, don't forget to commit)
$ hg debugstate | grep foo
m 0 -2 unset foo
$ hg st -A foo
M foo
$ hg ci -m 'merge'
$ hg manifest --debug | grep foo
c6fc755d7e68f49f880599da29f15add41f42f5a 644 foo
$ hg debugindex foo
rev offset length ..... linkrev nodeid p1 p2 (re)
0 0 5 ..... 0 2ed2a3912a0b 000000000000 000000000000 (re)
1 5 9 ..... 1 6f4310b00b9a 2ed2a3912a0b 000000000000 (re)
2 14 5 ..... 2 c6fc755d7e68 6f4310b00b9a 000000000000 (re)