view tests/test-pull-http.t @ 38732:be4984261611

merge: mark file gets as not thread safe (issue5933)

In default installs, this has the effect of disabling the thread-based
worker on Windows when manifesting files in the working directory. My
measurements have shown that with revlog-based repositories, Mercurial
spends a lot of CPU time in revlog code resolving file data. This ends
up incurring a lot of context switching across threads and slows down
`hg update` operations when going from an empty working directory to
the tip of the repo.

On mozilla-unified (246,351 files) on an i7-6700K (4+4 CPUs):

before: 487s wall
after:  360s wall (equivalent to worker.enabled=false)
cpus=2: 379s wall

Even with only 2 threads, the thread pool is still slower.

The introduction of the thread-based worker (02b36e860e0b) states that
it resulted in a "~50%" speedup for `hg sparse --enable-profile` and
`hg sparse --disable-profile`. This disagrees with my measurement
above. I theorize a few reasons for this:

1) Removal of files from the working directory is I/O - not CPU - bound
   and should benefit from a thread pool (unless I/O is insanely fast
   and the GIL release is near instantaneous). So tests like `hg sparse
   --enable-profile` may exercise deletion throughput and aren't good
   benchmarks for worker tasks that are CPU heavy.
2) The patch was authored by someone at Facebook. The results were
   likely measured against a repository using remotefilelog. And I
   believe that revision retrieval during working directory updates with
   remotefilelog will often use a remote store, thus being I/O and not
   CPU bound. This probably resulted in an overstated performance gain.

Since there appears to be a need to enable the thread-based worker with
some stores, I've made the flagging of file gets as thread safe
configurable. I've made it experimental because I don't want to formalize
a boolean flag for this option and because this attribute is best
captured against the store implementation. But we don't have a proper
store API for this yet. I'd rather cross this bridge later.

It is possible there are revlog-based repositories that do benefit from
a thread-based worker. I didn't do very comprehensive testing. If there
are, we may want to devise a more proper algorithm for whether to use
the thread-based worker, including possibly config options to limit the
number of threads to use. But until I see evidence that justifies
complexity, simplicity wins.

Differential Revision: https://phab.mercurial-scm.org/D3963

#if no-windows
For debugging: this is a pretty simple test that is a good candidate
for tracking down network-related bugs. Sometimes a command in this
hangs, so having showstack pre-loaded is sometimes helpful. This also
gives us a test that at least proves showstack can be loaded.
  $ cat >> $HGRCPATH <<EOF
  > [extensions]
  > showstack = $TESTDIR/../contrib/showstack.py
  > EOF
#endif

  $ hg init test
  $ cd test
  $ echo a > a
  $ hg ci -Ama
  adding a
  $ cd ..
  $ hg clone test test2
  updating to branch default
  1 files updated, 0 files merged, 0 files removed, 0 files unresolved
  $ cd test2
  $ echo a >> a
  $ hg ci -mb

Cloning with a password in the URL should not save the password in .hg/hgrc:

  $ hg serve -p $HGPORT -d --pid-file=hg.pid -E errors.log
  $ cat hg.pid >> $DAEMON_PIDS
  $ hg clone http://foo:xyzzy@localhost:$HGPORT/ test3
  requesting all changes
  adding changesets
  adding manifests
  adding file changes
  added 2 changesets with 2 changes to 1 files
  new changesets cb9a9f314b8b:ba677d0156c1
  updating to branch default
  1 files updated, 0 files merged, 0 files removed, 0 files unresolved
  $ cat test3/.hg/hgrc
  # example repository config (see 'hg help config' for more info)
  [paths]
  default = http://foo@localhost:$HGPORT/
  
  # path aliases to other clones of this repo in URLs or filesystem paths
  # (see 'hg help config.paths' for more info)
  #
  # default:pushurl = ssh://[e-mail]/hg/jdoes-fork
  # my-fork         = ssh://[e-mail]/hg/jdoes-fork
  # my-clone        = /home/jdoe/jdoes-clone
  
  [ui]
  # name and email (local to this repository, optional), e.g.
  # username = Jane Doe <[e-mail]>
  $ killdaemons.py

expect error, cloning not allowed

  $ echo '[web]' > .hg/hgrc
  $ echo 'allowpull = false' >> .hg/hgrc
  $ hg serve -p $HGPORT -d --pid-file=hg.pid -E errors.log
  $ cat hg.pid >> $DAEMON_PIDS
  $ hg clone http://localhost:$HGPORT/ test4 # bundle2+
  abort: authorization failed
  [255]
  $ hg clone http://localhost:$HGPORT/ test4 --config devel.legacy.exchange=bundle1
  abort: authorization failed
  [255]
  $ killdaemons.py

serve errors

  $ cat errors.log
  $ req() {
  >     hg serve -p $HGPORT -d --pid-file=hg.pid -E errors.log
  >     cat hg.pid >> $DAEMON_PIDS
  >     hg --cwd ../test pull http://localhost:$HGPORT/
  >     killdaemons.py hg.pid
  >     echo % serve errors
  >     cat errors.log
  > }

expect error, pulling not allowed

  $ req
  pulling from http://localhost:$HGPORT/
  abort: authorization failed
  % serve errors

  $ cd ..