Mercurial > hg
view tests/test-wireproto-command-heads.t @ 38732:be4984261611
merge: mark file gets as not thread safe (issue5933)
In default installs, this has the effect of disabling the thread-based
worker on Windows when manifesting files in the working directory. My
measurements have shown that with revlog-based repositories, Mercurial
spends a lot of CPU time in revlog code resolving file data. This ends
up incurring a lot of context switching across threads and slows down
`hg update` operations when going from an empty working directory to
the tip of the repo.
On mozilla-unified (246,351 files) on an i7-6700K (4+4 CPUs):
before: 487s wall
after: 360s wall (equivalent to worker.enabled=false)
cpus=2: 379s wall
Even with only 2 threads, the thread pool is still slower.
The introduction of the thread-based worker (02b36e860e0b) states that
it resulted in a "~50%" speedup for `hg sparse --enable-profile` and
`hg sparse --disable-profile`. This disagrees with my measurement
above. I theorize a few reasons for this:
1) Removal of files from the working directory is I/O - not CPU - bound
and should benefit from a thread pool (unless I/O is insanely fast
and the GIL release is near instantaneous). So tests like `hg sparse
--enable-profile` may exercise deletion throughput and aren't good
benchmarks for worker tasks that are CPU heavy.
2) The patch was authored by someone at Facebook. The results were
likely measured against a repository using remotefilelog. And I
believe that revision retrieval during working directory updates with
remotefilelog will often use a remote store, thus being I/O and not
CPU bound. This probably resulted in an overstated performance gain.
Since there appears to be a need to enable the thread-based worker with
some stores, I've made the flagging of file gets as thread safe
configurable. I've made it experimental because I don't want to formalize
a boolean flag for this option and because this attribute is best
captured against the store implementation. But we don't have a proper
store API for this yet. I'd rather cross this bridge later.
It is possible there are revlog-based repositories that do benefit from
a thread-based worker. I didn't do very comprehensive testing. If there
are, we may want to devise a more proper algorithm for whether to use
the thread-based worker, including possibly config options to limit the
number of threads to use. But until I see evidence that justifies
complexity, simplicity wins.
Differential Revision: https://phab.mercurial-scm.org/D3963
author | Gregory Szorc <gregory.szorc@gmail.com> |
---|---|
date | Wed, 18 Jul 2018 09:49:34 -0700 |
parents | 3ea8323d6f95 |
children | 0f549da54379 |
line wrap: on
line source
$ . $TESTDIR/wireprotohelpers.sh $ hg init server $ enablehttpv2 server $ cd server $ hg debugdrawdag << EOF > H I J > | | | > E F G > | |/ > C D > |/ > B > | > A > EOF $ hg phase --force --secret J $ hg phase --public E $ hg log -r 'E + H + I + G + J' -T '{rev}:{node} {desc} {phase}\n' 4:78d2dca436b2f5b188ac267e29b81e07266d38fc E public 7:ae492e36b0c8339ffaf328d00b85b4525de1165e H draft 8:1d6f6b91d44aaba6d5e580bc30a9948530dbe00b I draft 6:29446d2dc5419c5f97447a8bc062e4cc328bf241 G draft 9:dec04b246d7cbb670c6689806c05ad17c835284e J secret $ hg serve -p $HGPORT -d --pid-file hg.pid -E error.log $ cat hg.pid > $DAEMON_PIDS All non-secret heads returned by default $ sendhttpv2peer << EOF > command heads > EOF creating http peer for wire protocol version 2 sending heads command s> POST /api/exp-http-v2-0001/ro/heads HTTP/1.1\r\n s> Accept-Encoding: identity\r\n s> accept: application/mercurial-exp-framing-0005\r\n s> content-type: application/mercurial-exp-framing-0005\r\n s> content-length: 20\r\n s> host: $LOCALIP:$HGPORT\r\n (glob) s> user-agent: Mercurial debugwireproto\r\n s> \r\n s> \x0c\x00\x00\x01\x00\x01\x01\x11\xa1DnameEheads s> makefile('rb', None) s> HTTP/1.1 200 OK\r\n s> Server: testing stub value\r\n s> Date: $HTTP_DATE$\r\n s> Content-Type: application/mercurial-exp-framing-0005\r\n s> Transfer-Encoding: chunked\r\n s> \r\n s> 53\r\n s> K\x00\x00\x01\x00\x02\x012 s> \xa1FstatusBok\x83T\x1dok\x91\xd4J\xab\xa6\xd5\xe5\x80\xbc0\xa9\x94\x850\xdb\xe0\x0bT\xaeI.6\xb0\xc83\x9f\xfa\xf3(\xd0\x0b\x85\xb4R]\xe1\x16^T)Dm-\xc5A\x9c_\x97Dz\x8b\xc0b\xe4\xcc2\x8b\xf2A s> \r\n received frame(size=75; request=1; stream=2; streamflags=stream-begin; type=command-response; flags=eos) s> 0\r\n s> \r\n response: [b'\x1dok\x91\xd4J\xab\xa6\xd5\xe5\x80\xbc0\xa9\x94\x850\xdb\xe0\x0b', b'\xaeI.6\xb0\xc83\x9f\xfa\xf3(\xd0\x0b\x85\xb4R]\xe1\x16^', b')Dm-\xc5A\x9c_\x97Dz\x8b\xc0b\xe4\xcc2\x8b\xf2A'] Requesting just the public heads works $ sendhttpv2peer << EOF > command heads > publiconly 1 > EOF creating http peer for wire protocol version 2 sending heads command s> POST /api/exp-http-v2-0001/ro/heads HTTP/1.1\r\n s> Accept-Encoding: identity\r\n s> accept: application/mercurial-exp-framing-0005\r\n s> content-type: application/mercurial-exp-framing-0005\r\n s> content-length: 39\r\n s> host: $LOCALIP:$HGPORT\r\n (glob) s> user-agent: Mercurial debugwireproto\r\n s> \r\n s> \x1f\x00\x00\x01\x00\x01\x01\x11\xa2Dargs\xa1JpubliconlyA1DnameEheads s> makefile('rb', None) s> HTTP/1.1 200 OK\r\n s> Server: testing stub value\r\n s> Date: $HTTP_DATE$\r\n s> Content-Type: application/mercurial-exp-framing-0005\r\n s> Transfer-Encoding: chunked\r\n s> \r\n s> 29\r\n s> !\x00\x00\x01\x00\x02\x012 s> \xa1FstatusBok\x81Tx\xd2\xdc\xa46\xb2\xf5\xb1\x88\xac&~)\xb8\x1e\x07&m8\xfc s> \r\n received frame(size=33; request=1; stream=2; streamflags=stream-begin; type=command-response; flags=eos) s> 0\r\n s> \r\n response: [b'x\xd2\xdc\xa46\xb2\xf5\xb1\x88\xac&~)\xb8\x1e\x07&m8\xfc'] $ cat error.log