Mercurial > hg
view tests/test-eol-hook.t @ 38732:be4984261611
merge: mark file gets as not thread safe (issue5933)
In default installs, this has the effect of disabling the thread-based
worker on Windows when manifesting files in the working directory. My
measurements have shown that with revlog-based repositories, Mercurial
spends a lot of CPU time in revlog code resolving file data. This ends
up incurring a lot of context switching across threads and slows down
`hg update` operations when going from an empty working directory to
the tip of the repo.
On mozilla-unified (246,351 files) on an i7-6700K (4+4 CPUs):
before: 487s wall
after: 360s wall (equivalent to worker.enabled=false)
cpus=2: 379s wall
Even with only 2 threads, the thread pool is still slower.
The introduction of the thread-based worker (02b36e860e0b) states that
it resulted in a "~50%" speedup for `hg sparse --enable-profile` and
`hg sparse --disable-profile`. This disagrees with my measurement
above. I theorize a few reasons for this:
1) Removal of files from the working directory is I/O - not CPU - bound
and should benefit from a thread pool (unless I/O is insanely fast
and the GIL release is near instantaneous). So tests like `hg sparse
--enable-profile` may exercise deletion throughput and aren't good
benchmarks for worker tasks that are CPU heavy.
2) The patch was authored by someone at Facebook. The results were
likely measured against a repository using remotefilelog. And I
believe that revision retrieval during working directory updates with
remotefilelog will often use a remote store, thus being I/O and not
CPU bound. This probably resulted in an overstated performance gain.
Since there appears to be a need to enable the thread-based worker with
some stores, I've made the flagging of file gets as thread safe
configurable. I've made it experimental because I don't want to formalize
a boolean flag for this option and because this attribute is best
captured against the store implementation. But we don't have a proper
store API for this yet. I'd rather cross this bridge later.
It is possible there are revlog-based repositories that do benefit from
a thread-based worker. I didn't do very comprehensive testing. If there
are, we may want to devise a more proper algorithm for whether to use
the thread-based worker, including possibly config options to limit the
number of threads to use. But until I see evidence that justifies
complexity, simplicity wins.
Differential Revision: https://phab.mercurial-scm.org/D3963
author | Gregory Szorc <gregory.szorc@gmail.com> |
---|---|
date | Wed, 18 Jul 2018 09:49:34 -0700 |
parents | f5b6b4e574c1 |
children | d7304434390f |
line wrap: on
line source
Test the EOL hook $ hg init main $ cat > main/.hg/hgrc <<EOF > [hooks] > pretxnchangegroup = python:hgext.eol.hook > EOF $ hg clone main fork updating to branch default 0 files updated, 0 files merged, 0 files removed, 0 files unresolved $ cd fork Create repo $ cat > .hgeol <<EOF > [patterns] > mixed.txt = BIN > crlf.txt = CRLF > **.txt = native > EOF $ hg add .hgeol $ hg commit -m 'Commit .hgeol' $ printf "first\nsecond\nthird\n" > a.txt $ hg add a.txt $ hg commit -m 'LF a.txt' $ hg push ../main pushing to ../main searching for changes adding changesets adding manifests adding file changes added 2 changesets with 2 changes to 2 files $ printf "first\r\nsecond\r\nthird\n" > a.txt $ hg commit -m 'CRLF a.txt' $ hg push ../main pushing to ../main searching for changes adding changesets adding manifests adding file changes added 1 changesets with 1 changes to 1 files error: pretxnchangegroup hook failed: end-of-line check failed: a.txt in a8ee6548cd86 should not have CRLF line endings transaction abort! rollback completed abort: end-of-line check failed: a.txt in a8ee6548cd86 should not have CRLF line endings [255] $ printf "first\nsecond\nthird\n" > a.txt $ hg commit -m 'LF a.txt (fixed)' $ hg push ../main pushing to ../main searching for changes adding changesets adding manifests adding file changes added 2 changesets with 2 changes to 1 files $ printf "first\nsecond\nthird\n" > crlf.txt $ hg add crlf.txt $ hg commit -m 'LF crlf.txt' $ hg push ../main pushing to ../main searching for changes adding changesets adding manifests adding file changes added 1 changesets with 1 changes to 1 files error: pretxnchangegroup hook failed: end-of-line check failed: crlf.txt in 004ba2132725 should not have LF line endings transaction abort! rollback completed abort: end-of-line check failed: crlf.txt in 004ba2132725 should not have LF line endings [255] $ printf "first\r\nsecond\r\nthird\r\n" > crlf.txt $ hg commit -m 'CRLF crlf.txt (fixed)' $ hg push ../main pushing to ../main searching for changes adding changesets adding manifests adding file changes added 2 changesets with 2 changes to 1 files $ printf "first\r\nsecond" > b.txt $ hg add b.txt $ hg commit -m 'CRLF b.txt' $ hg push ../main pushing to ../main searching for changes adding changesets adding manifests adding file changes added 1 changesets with 1 changes to 1 files error: pretxnchangegroup hook failed: end-of-line check failed: b.txt in fbcf9b1025f5 should not have CRLF line endings transaction abort! rollback completed abort: end-of-line check failed: b.txt in fbcf9b1025f5 should not have CRLF line endings [255] $ hg up -r -2 0 files updated, 0 files merged, 1 files removed, 0 files unresolved $ printf "some\nother\nfile" > c.txt $ hg add c.txt $ hg commit -m "LF c.txt, b.txt doesn't exist here" created new head $ hg push -f ../main pushing to ../main searching for changes adding changesets adding manifests adding file changes added 2 changesets with 2 changes to 2 files (+1 heads) error: pretxnchangegroup hook failed: end-of-line check failed: b.txt in fbcf9b1025f5 should not have CRLF line endings transaction abort! rollback completed abort: end-of-line check failed: b.txt in fbcf9b1025f5 should not have CRLF line endings [255] Test checkheadshook alias $ cat > ../main/.hg/hgrc <<EOF > [hooks] > pretxnchangegroup = python:hgext.eol.checkheadshook > EOF $ hg push -f ../main pushing to ../main searching for changes adding changesets adding manifests adding file changes added 2 changesets with 2 changes to 2 files (+1 heads) error: pretxnchangegroup hook failed: end-of-line check failed: b.txt in fbcf9b1025f5 should not have CRLF line endings transaction abort! rollback completed abort: end-of-line check failed: b.txt in fbcf9b1025f5 should not have CRLF line endings [255] We can fix the head and push again $ hg up 6 1 files updated, 0 files merged, 1 files removed, 0 files unresolved $ printf "first\nsecond" > b.txt $ hg ci -m "remove CRLF from b.txt" $ hg push -f ../main pushing to ../main searching for changes adding changesets adding manifests adding file changes added 3 changesets with 3 changes to 2 files (+1 heads) $ hg -R ../main rollback repository tip rolled back to revision 5 (undo push) Test it still fails with checkallhook $ cat > ../main/.hg/hgrc <<EOF > [hooks] > pretxnchangegroup = python:hgext.eol.checkallhook > EOF $ hg push -f ../main pushing to ../main searching for changes adding changesets adding manifests adding file changes added 3 changesets with 3 changes to 2 files (+1 heads) error: pretxnchangegroup hook failed: end-of-line check failed: b.txt in fbcf9b1025f5 should not have CRLF line endings transaction abort! rollback completed abort: end-of-line check failed: b.txt in fbcf9b1025f5 should not have CRLF line endings [255] But we can push the clean head $ hg push -r7 -f ../main pushing to ../main searching for changes adding changesets adding manifests adding file changes added 1 changesets with 1 changes to 1 files Test multiple files/revisions output $ printf "another\r\nbad\r\none" > d.txt $ hg add d.txt $ hg ci -m "add d.txt" $ hg push -f ../main pushing to ../main searching for changes adding changesets adding manifests adding file changes added 3 changesets with 3 changes to 2 files (+1 heads) error: pretxnchangegroup hook failed: end-of-line check failed: b.txt in fbcf9b1025f5 should not have CRLF line endings d.txt in a7040e68714f should not have CRLF line endings transaction abort! rollback completed abort: end-of-line check failed: b.txt in fbcf9b1025f5 should not have CRLF line endings d.txt in a7040e68714f should not have CRLF line endings [255] $ cd ..