Mercurial > hg
view tests/test-serve.t @ 38732:be4984261611
merge: mark file gets as not thread safe (issue5933)
In default installs, this has the effect of disabling the thread-based
worker on Windows when manifesting files in the working directory. My
measurements have shown that with revlog-based repositories, Mercurial
spends a lot of CPU time in revlog code resolving file data. This ends
up incurring a lot of context switching across threads and slows down
`hg update` operations when going from an empty working directory to
the tip of the repo.
On mozilla-unified (246,351 files) on an i7-6700K (4+4 CPUs):
before: 487s wall
after: 360s wall (equivalent to worker.enabled=false)
cpus=2: 379s wall
Even with only 2 threads, the thread pool is still slower.
The introduction of the thread-based worker (02b36e860e0b) states that
it resulted in a "~50%" speedup for `hg sparse --enable-profile` and
`hg sparse --disable-profile`. This disagrees with my measurement
above. I theorize a few reasons for this:
1) Removal of files from the working directory is I/O - not CPU - bound
and should benefit from a thread pool (unless I/O is insanely fast
and the GIL release is near instantaneous). So tests like `hg sparse
--enable-profile` may exercise deletion throughput and aren't good
benchmarks for worker tasks that are CPU heavy.
2) The patch was authored by someone at Facebook. The results were
likely measured against a repository using remotefilelog. And I
believe that revision retrieval during working directory updates with
remotefilelog will often use a remote store, thus being I/O and not
CPU bound. This probably resulted in an overstated performance gain.
Since there appears to be a need to enable the thread-based worker with
some stores, I've made the flagging of file gets as thread safe
configurable. I've made it experimental because I don't want to formalize
a boolean flag for this option and because this attribute is best
captured against the store implementation. But we don't have a proper
store API for this yet. I'd rather cross this bridge later.
It is possible there are revlog-based repositories that do benefit from
a thread-based worker. I didn't do very comprehensive testing. If there
are, we may want to devise a more proper algorithm for whether to use
the thread-based worker, including possibly config options to limit the
number of threads to use. But until I see evidence that justifies
complexity, simplicity wins.
Differential Revision: https://phab.mercurial-scm.org/D3963
author | Gregory Szorc <gregory.szorc@gmail.com> |
---|---|
date | Wed, 18 Jul 2018 09:49:34 -0700 |
parents | 89793289c891 |
children | 5abc47d4ca6b |
line wrap: on
line source
#require serve $ hgserve() > { > hg serve -a localhost -d --pid-file=hg.pid -E errors.log -v $@ \ > | sed -e "s/:$HGPORT1\\([^0-9]\\)/:HGPORT1\1/g" \ > -e "s/:$HGPORT2\\([^0-9]\\)/:HGPORT2\1/g" \ > -e 's/http:\/\/[^/]*\//http:\/\/localhost\//' > if [ -f hg.pid ]; then > killdaemons.py hg.pid > fi > echo % errors > cat errors.log > } $ hg init test $ cd test $ echo '[web]' > .hg/hgrc $ echo 'accesslog = access.log' >> .hg/hgrc $ echo "port = $HGPORT1" >> .hg/hgrc Without -v $ hg serve -a localhost -p $HGPORT -d --pid-file=hg.pid -E errors.log $ cat hg.pid >> "$DAEMON_PIDS" $ if [ -f access.log ]; then > echo 'access log created - .hg/hgrc respected' > fi access log created - .hg/hgrc respected errors $ cat errors.log With -v $ hgserve listening at http://localhost/ (bound to *$LOCALIP*:HGPORT1) (glob) (?) % errors With -v and -p HGPORT2 $ hgserve -p "$HGPORT2" listening at http://localhost/ (bound to *$LOCALIP*:HGPORT2) (glob) (?) % errors With -v and -p daytime (should fail because low port) #if no-root no-windows $ KILLQUIETLY=Y $ hgserve -p daytime abort: cannot start server at 'localhost:13': Permission denied abort: child process failed to start % errors $ KILLQUIETLY=N #endif With --prefix foo $ hgserve --prefix foo listening at http://localhost/foo/ (bound to *$LOCALIP*:HGPORT1) (glob) (?) % errors With --prefix /foo $ hgserve --prefix /foo listening at http://localhost/foo/ (bound to *$LOCALIP*:HGPORT1) (glob) (?) % errors With --prefix foo/ $ hgserve --prefix foo/ listening at http://localhost/foo/ (bound to *$LOCALIP*:HGPORT1) (glob) (?) % errors With --prefix /foo/ $ hgserve --prefix /foo/ listening at http://localhost/foo/ (bound to *$LOCALIP*:HGPORT1) (glob) (?) % errors $ $PYTHON $RUNTESTDIR/killdaemons.py $DAEMON_PIDS With out of bounds accesses $ rm access.log $ hg serve -a localhost -p $HGPORT -d --prefix some/dir \ > --pid-file=hg.pid -E errors.log $ cat hg.pid >> "$DAEMON_PIDS" $ hg id http://localhost:$HGPORT/some/dir7 abort: HTTP Error 404: Not Found [255] $ hg id http://localhost:$HGPORT/some abort: HTTP Error 404: Not Found [255] $ cat access.log errors.log $LOCALIP - - [$LOGDATE$] "GET /some/dir7?cmd=capabilities HTTP/1.1" 404 - (glob) $LOCALIP - - [$LOGDATE$] "GET /some?cmd=capabilities HTTP/1.1" 404 - (glob) $ cd ..