streamclone: use backgroundfilecloser (issue4889)
Closing files that have been appended to is slow on Windows/NTFS.
CloseHandle() calls on this platform often take 1-10ms - and that's
on my i7-6700K Skylake processor with a modern and fast SSD. Contrast
with other I/O operations, such as writing data, which take <100us.
This means that creating/appending thousands of files can add
significant overhead. For example, cloning mozilla-central creates
~232,000 revlog files. Assuming 1ms per CloseHandle(), that yields
232s (3:52) of wall time waiting for file closes!
The impact of this overhead can be measured most directly when applying
stream clone bundles. Applying these files is effectively uncompressing
a tar archive (read: it's very fast).
Using a RAM disk (read: no I/O wait), the difference in wall time for a
`hg debugapplystreamclonebundle` for a ~1731 MB mozilla-central bundle
between Windows and Linux from the same machine is drastic:
Linux: ~12.8s (128MB/s)
Windows: ~352.0s (4.7MB/s)
Windows is ~27.5x slower. Yikes!
After this patch:
Linux: ~12.8s (128MB/s)
Windows: ~102.1s (16.1MB/s)
Windows is now ~3.4x faster. Unfortunately, it is still ~8x slower than
Linux. Profiling reveals a few hot code paths that could likely be
improved. But those are for other patches.
This patch introduces test-clone-uncompressed.t because existing tests
of `clone --uncompressed` are scattered about and adding a variation for
background thread closing to e.g. test-http.t doesn't feel correct.
$ cat >> $HGRCPATH <<EOF
> [extensions]
> rebase=
>
> [alias]
> tglog = log -G --template "{rev}: '{desc}' {branches}\n"
> EOF
$ hg init a
$ cd a
$ echo C1 > C1
$ hg ci -Am C1
adding C1
$ echo C2 > C2
$ hg ci -Am C2
adding C2
$ cd ..
$ hg clone a b
updating to branch default
2 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ hg clone a c
updating to branch default
2 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ cd b
$ echo L1 > L1
$ hg ci -Am L1
adding L1
$ cd ../a
$ echo R1 > R1
$ hg ci -Am R1
adding R1
$ cd ../b
Now b has one revision to be pulled from a:
$ hg pull --rebase
pulling from $TESTTMP/a (glob)
searching for changes
adding changesets
adding manifests
adding file changes
added 1 changesets with 1 changes to 1 files (+1 heads)
rebasing 2:ff8d69a621f9 "L1"
saved backup bundle to $TESTTMP/b/.hg/strip-backup/ff8d69a621f9-160fa373-backup.hg (glob)
$ hg tglog
@ 3: 'L1'
|
o 2: 'R1'
|
o 1: 'C2'
|
o 0: 'C1'
Re-run:
$ hg pull --rebase
pulling from $TESTTMP/a (glob)
searching for changes
no changes found
Invoke pull --rebase and nothing to rebase:
$ cd ../c
$ hg book norebase
$ hg pull --rebase
pulling from $TESTTMP/a (glob)
searching for changes
adding changesets
adding manifests
adding file changes
added 1 changesets with 1 changes to 1 files
nothing to rebase - working directory parent is already an ancestor of destination 77ae9631bcca
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
updating bookmark norebase
$ hg tglog -l 1
@ 2: 'R1'
|
pull --rebase --update should ignore --update:
$ hg pull --rebase --update
pulling from $TESTTMP/a (glob)
searching for changes
no changes found
pull --rebase doesn't update if nothing has been pulled:
$ hg up -q 1
$ hg pull --rebase
pulling from $TESTTMP/a (glob)
searching for changes
no changes found
$ hg tglog -l 1
o 2: 'R1'
|
$ cd ..
pull --rebase works when a specific revision is pulled (issue3619)
$ cd a
$ hg tglog
@ 2: 'R1'
|
o 1: 'C2'
|
o 0: 'C1'
$ echo R2 > R2
$ hg ci -Am R2
adding R2
$ echo R3 > R3
$ hg ci -Am R3
adding R3
$ cd ../c
$ hg tglog
o 2: 'R1'
|
@ 1: 'C2'
|
o 0: 'C1'
$ echo L1 > L1
$ hg ci -Am L1
adding L1
created new head
$ hg pull --rev tip --rebase
pulling from $TESTTMP/a (glob)
searching for changes
adding changesets
adding manifests
adding file changes
added 2 changesets with 2 changes to 2 files
rebasing 3:ff8d69a621f9 "L1"
saved backup bundle to $TESTTMP/c/.hg/strip-backup/ff8d69a621f9-160fa373-backup.hg (glob)
$ hg tglog
@ 5: 'L1'
|
o 4: 'R3'
|
o 3: 'R2'
|
o 2: 'R1'
|
o 1: 'C2'
|
o 0: 'C1'
pull --rebase works with bundle2 turned on
$ cd ../a
$ echo R4 > R4
$ hg ci -Am R4
adding R4
$ hg tglog
@ 5: 'R4'
|
o 4: 'R3'
|
o 3: 'R2'
|
o 2: 'R1'
|
o 1: 'C2'
|
o 0: 'C1'
$ cd ../c
$ hg pull --rebase
pulling from $TESTTMP/a (glob)
searching for changes
adding changesets
adding manifests
adding file changes
added 1 changesets with 1 changes to 1 files (+1 heads)
rebasing 5:518d153c0ba3 "L1"
saved backup bundle to $TESTTMP/c/.hg/strip-backup/518d153c0ba3-73407f14-backup.hg (glob)
$ hg tglog
@ 6: 'L1'
|
o 5: 'R4'
|
o 4: 'R3'
|
o 3: 'R2'
|
o 2: 'R1'
|
o 1: 'C2'
|
o 0: 'C1'