Mercurial > hg
view tests/test-repair-strip.t @ 38732:be4984261611
merge: mark file gets as not thread safe (issue5933)
In default installs, this has the effect of disabling the thread-based
worker on Windows when manifesting files in the working directory. My
measurements have shown that with revlog-based repositories, Mercurial
spends a lot of CPU time in revlog code resolving file data. This ends
up incurring a lot of context switching across threads and slows down
`hg update` operations when going from an empty working directory to
the tip of the repo.
On mozilla-unified (246,351 files) on an i7-6700K (4+4 CPUs):
before: 487s wall
after: 360s wall (equivalent to worker.enabled=false)
cpus=2: 379s wall
Even with only 2 threads, the thread pool is still slower.
The introduction of the thread-based worker (02b36e860e0b) states that
it resulted in a "~50%" speedup for `hg sparse --enable-profile` and
`hg sparse --disable-profile`. This disagrees with my measurement
above. I theorize a few reasons for this:
1) Removal of files from the working directory is I/O - not CPU - bound
and should benefit from a thread pool (unless I/O is insanely fast
and the GIL release is near instantaneous). So tests like `hg sparse
--enable-profile` may exercise deletion throughput and aren't good
benchmarks for worker tasks that are CPU heavy.
2) The patch was authored by someone at Facebook. The results were
likely measured against a repository using remotefilelog. And I
believe that revision retrieval during working directory updates with
remotefilelog will often use a remote store, thus being I/O and not
CPU bound. This probably resulted in an overstated performance gain.
Since there appears to be a need to enable the thread-based worker with
some stores, I've made the flagging of file gets as thread safe
configurable. I've made it experimental because I don't want to formalize
a boolean flag for this option and because this attribute is best
captured against the store implementation. But we don't have a proper
store API for this yet. I'd rather cross this bridge later.
It is possible there are revlog-based repositories that do benefit from
a thread-based worker. I didn't do very comprehensive testing. If there
are, we may want to devise a more proper algorithm for whether to use
the thread-based worker, including possibly config options to limit the
number of threads to use. But until I see evidence that justifies
complexity, simplicity wins.
Differential Revision: https://phab.mercurial-scm.org/D3963
author | Gregory Szorc <gregory.szorc@gmail.com> |
---|---|
date | Wed, 18 Jul 2018 09:49:34 -0700 |
parents | e1d4be95cd68 |
children | f1186c292d03 |
line wrap: on
line source
#require unix-permissions no-root reporevlogstore $ cat > $TESTTMP/dumpjournal.py <<EOF > import sys > for entry in sys.stdin.read().split('\n'): > if entry: > print(entry.split('\x00')[0]) > EOF $ echo "[extensions]" >> $HGRCPATH $ echo "mq=">> $HGRCPATH $ teststrip() { > hg -q up -C $1 > echo % before update $1, strip $2 > hg parents > chmod -$3 $4 > hg strip $2 2>&1 | sed 's/\(bundle\).*/\1/' | sed 's/Permission denied.*\.hg\/store\/\(.*\)/Permission denied \.hg\/store\/\1/' > echo % after update $1, strip $2 > chmod +$3 $4 > hg verify > echo % journal contents > if [ -f .hg/store/journal ]; then > cat .hg/store/journal | $PYTHON $TESTTMP/dumpjournal.py > else > echo "(no journal)" > fi > ls .hg/store/journal >/dev/null 2>&1 && hg recover > ls .hg/strip-backup/* >/dev/null 2>&1 && hg unbundle -q .hg/strip-backup/* > rm -rf .hg/strip-backup > } $ hg init test $ cd test $ echo a > a $ hg -q ci -m "a" -A $ echo b > b $ hg -q ci -m "b" -A $ echo b2 >> b $ hg -q ci -m "b2" -A $ echo c > c $ hg -q ci -m "c" -A $ teststrip 0 2 w .hg/store/data/b.i % before update 0, strip 2 changeset: 0:cb9a9f314b8b user: test date: Thu Jan 01 00:00:00 1970 +0000 summary: a saved backup bundle transaction abort! failed to truncate data/b.i rollback failed - please run hg recover strip failed, backup bundle abort: Permission denied .hg/store/data/b.i % after update 0, strip 2 abandoned transaction found - run hg recover checking changesets checking manifests crosschecking files in changesets and manifests checking files b@?: rev 1 points to nonexistent changeset 2 (expected 1) b@?: 736c29771fba not in manifests warning: orphan data file 'data/c.i' 2 files, 2 changesets, 3 total revisions 2 warnings encountered! 2 integrity errors encountered! % journal contents 00changelog.i 00manifest.i data/b.i data/c.i rolling back interrupted transaction checking changesets checking manifests crosschecking files in changesets and manifests checking files 2 files, 2 changesets, 2 total revisions $ teststrip 0 2 r .hg/store/data/b.i % before update 0, strip 2 changeset: 0:cb9a9f314b8b user: test date: Thu Jan 01 00:00:00 1970 +0000 summary: a abort: Permission denied .hg/store/data/b.i % after update 0, strip 2 checking changesets checking manifests crosschecking files in changesets and manifests checking files 3 files, 4 changesets, 4 total revisions % journal contents (no journal) $ teststrip 0 2 w .hg/store/00manifest.i % before update 0, strip 2 changeset: 0:cb9a9f314b8b user: test date: Thu Jan 01 00:00:00 1970 +0000 summary: a saved backup bundle transaction abort! failed to truncate 00manifest.i rollback failed - please run hg recover strip failed, backup bundle abort: Permission denied .hg/store/00manifest.i % after update 0, strip 2 abandoned transaction found - run hg recover checking changesets checking manifests manifest@?: rev 2 points to nonexistent changeset 2 manifest@?: 3362547cdf64 not in changesets manifest@?: rev 3 points to nonexistent changeset 3 manifest@?: 265a85892ecb not in changesets crosschecking files in changesets and manifests c@3: in manifest but not in changeset checking files b@?: rev 1 points to nonexistent changeset 2 (expected 1) c@?: rev 0 points to nonexistent changeset 3 3 files, 2 changesets, 4 total revisions 1 warnings encountered! 7 integrity errors encountered! (first damaged changeset appears to be 3) % journal contents 00changelog.i 00manifest.i data/b.i data/c.i rolling back interrupted transaction checking changesets checking manifests crosschecking files in changesets and manifests checking files 2 files, 2 changesets, 2 total revisions $ cd ..