view tests/test-http-proxy.t @ 51977:42a116f1cdc1

branchmap-v3: introduce a "stop_rev" argument to `headsrevs`

The `headsrevs` method of the revlog already have a `revs` argument to compute
the headrevs of a limited set of heads. However, it disable the use of the
native compiled code to compute the heads, which slows down the branchmap v3
code a lot.

The branchmap v3 usage is actually quite constrained as we will always only
ignores a part at the top of the graph. So we could be significantly faster.

We start by making small change to the python side to improve the situation and
introduce the new API. More collaboration with the native code are coming later.

This massively speedup operation and close most of the remaining gaps between
branchmap-v3 and branchmap-v2. especially on repository with many revs like
mozilla-try. A small overhead remains mostly because the `headrevs` logic
currently has some inefficiently. We will look into them from there.

### benchmark.name                                 = hg.command.unbundle
  # bin-env-vars.hg.py-re2-module                  = default
  # benchmark.variants.issue6528                   = disabled
  # benchmark.variants.resource-usage              = default
  # benchmark.variants.reuse-external-delta-parent = yes
  # benchmark.variants.revs                        = any-1-extra-rev
  # benchmark.variants.source                      = unbundle
  # benchmark.variants.validate                    = default
  # benchmark.variants.verbosity                   = quiet

 ## data-env-vars.name                             = netbeans-2018-08-01-zstd-sparse-revlog
  # bin-env-vars.hg.flavor                         = default
branch-v2:        0.233711  ~~~~~
branch-v3 before: 0.368769  (+57.79%, +0.14)
branch-v3 after:  0.239857   (+2.63%, +0.01)
  # bin-env-vars.hg.flavor                         = rust
branch-v2:        0.235230  ~~~~~
branch-v3 before: 0.372460  (+58.34%, +0.14)
branch-v3 after:  0.240972   (+2.44%, +0.01)
 ## data-env-vars.name                             = netbeans-2018-08-01-ds2-pnm
  # bin-env-vars.hg.flavor                         = rust
branch-v2:        0.255586  ~~~~~
branch-v3 before: 0.318907  (+24.78%, +0.06)
branch-v3 after:  0.268560   (+5.08%, +0.01)

 ## data-env-vars.name                             = mozilla-central-2024-03-22-zstd-sparse-revlog
  # bin-env-vars.hg.flavor                         = default
branch-v2:        0.339010  ~~~~~
branch-v3 before: 0.349752   (+3.17%, +0.01)
branch-v3 after:  0.349389   (+3.06%, +0.01)
  # bin-env-vars.hg.flavor                         = rust
branch-v2:        0.346525  ~~~~~
branch-v3 before: 0.354300   (+2.24%, +0.01)
branch-v3 after:  0.355661   (+2.64%, +0.01)
 ## data-env-vars.name                             = mozilla-central-2024-03-22-ds2-pnm
  # bin-env-vars.hg.flavor                         = rust
branch-v2:        0.380202  ~~~~~
branch-v3 before: 0.396293   (+4.23%, +0.02)
branch-v3 after:  0.408851   (+7.54%, +0.03)

 ## data-env-vars.name                             = mozilla-unified-2024-03-22-zstd-sparse-revlog
  # bin-env-vars.hg.flavor                         = default
branch-v2:        0.412165  ~~~~~
branch-v3 before: 0.424769   (+3.06%, +0.01)
branch-v3 after:  0.427782   (+3.79%, +0.02)
  # bin-env-vars.hg.flavor                         = rust
branch-v2:        0.412397  ~~~~~
branch-v3 before: 0.421796   (+2.28%, +0.01)
branch-v3 after:  0.422354   (+2.41%, +0.01)
 ## data-env-vars.name                             = mozilla-unified-2024-03-22-ds2-pnm
  # bin-env-vars.hg.flavor                         = rust
branch-v2:        0.429501  ~~~~~
branch-v3 before: 0.443849   (+3.34%, +0.01)
branch-v3 after:  0.443197   (+3.19%, +0.01)

 ## data-env-vars.name                             = mozilla-try-2024-03-26-zstd-sparse-revlog
  # bin-env-vars.hg.flavor                         = default
branch-v2:        3.403171  ~~~~~
branch-v3 before: 6.234055  (+83.18%, +2.83)
branch-v3 after:  3.819477  (+12.23%, +0.42)
  # bin-env-vars.hg.flavor                         = rust
branch-v2:        3.454876  ~~~~~
branch-v3 before: 6.307813  (+82.58%, +2.85)
branch-v3 after:  3.590284   (+3.92%, +0.14)
 ## data-env-vars.name                             = mozilla-try-2024-03-26-ds2-pnm
  # bin-env-vars.hg.flavor                         = rust
branch-v2:        3.465435  ~~~~~
branch-v3 before: 5.176076  (+49.36%, +1.71)
branch-v3 after:  3.633278   (+4.84%, +0.17)

#require serve

  $ hg init a
  $ cd a
  $ echo a > a
  $ hg ci -Ama -d '1123456789 0'
  adding a
  $ hg serve --config server.uncompressed=True -p $HGPORT -d --pid-file=hg.pid
  $ cat hg.pid >> $DAEMON_PIDS
  $ cd ..
  $ tinyproxy.py $HGPORT1 localhost 2>proxy.log >/dev/null </dev/null &
  $ while [ ! -f proxy.pid ]; do sleep 0; done
  $ cat proxy.pid >> $DAEMON_PIDS

url for proxy, stream

  $ http_proxy=http://localhost:$HGPORT1/ hg --config http_proxy.always=True clone --stream http://localhost:$HGPORT/ b
  streaming all changes
  7 files to transfer, 412 bytes of data (reporevlogstore no-rust !)
  9 files to transfer, 538 bytes of data (reporevlogstore rust !)
  4 files to transfer, 330 bytes of data (reposimplestore !)
  transferred * bytes in * seconds (*/sec) (glob)
  updating to branch default
  1 files updated, 0 files merged, 0 files removed, 0 files unresolved
  $ cd b
  $ hg verify -q
  $ cd ..

url for proxy, pull

  $ http_proxy=http://localhost:$HGPORT1/ hg --config http_proxy.always=True clone http://localhost:$HGPORT/ b-pull
  requesting all changes
  adding changesets
  adding manifests
  adding file changes
  added 1 changesets with 1 changes to 1 files
  new changesets 83180e7845de
  updating to branch default
  1 files updated, 0 files merged, 0 files removed, 0 files unresolved
  $ cd b-pull
  $ hg verify -q
  $ cd ..

host:port for proxy

  $ http_proxy=localhost:$HGPORT1 hg clone --config http_proxy.always=True http://localhost:$HGPORT/ c
  requesting all changes
  adding changesets
  adding manifests
  adding file changes
  added 1 changesets with 1 changes to 1 files
  new changesets 83180e7845de
  updating to branch default
  1 files updated, 0 files merged, 0 files removed, 0 files unresolved

proxy url with user name and password

  $ http_proxy=http://user:passwd@localhost:$HGPORT1 hg clone --config http_proxy.always=True http://localhost:$HGPORT/ d
  requesting all changes
  adding changesets
  adding manifests
  adding file changes
  added 1 changesets with 1 changes to 1 files
  new changesets 83180e7845de
  updating to branch default
  1 files updated, 0 files merged, 0 files removed, 0 files unresolved

url with user name and password

  $ http_proxy=http://user:passwd@localhost:$HGPORT1 hg clone --config http_proxy.always=True http://user:passwd@localhost:$HGPORT/ e
  requesting all changes
  adding changesets
  adding manifests
  adding file changes
  added 1 changesets with 1 changes to 1 files
  new changesets 83180e7845de
  updating to branch default
  1 files updated, 0 files merged, 0 files removed, 0 files unresolved

bad host:port for proxy ("Protocol not supported" can happen on
misconfigured hosts)

  $ http_proxy=localhost:$HGPORT2 hg clone --config http_proxy.always=True http://localhost:$HGPORT/ f
  abort: error: (\$ECONNREFUSED\$|Protocol not supported|.* actively refused it|\$EADDRNOTAVAIL\$|No route to host) (re)
  [100]

do not use the proxy if it is in the no list

  $ http_proxy=localhost:$HGPORT1 hg clone --config http_proxy.no=localhost http://localhost:$HGPORT/ g
  requesting all changes
  adding changesets
  adding manifests
  adding file changes
  added 1 changesets with 1 changes to 1 files
  new changesets 83180e7845de
  updating to branch default
  1 files updated, 0 files merged, 0 files removed, 0 files unresolved

proxy can't connect to server

  $ http_proxy=http://localhost:$HGPORT1/ hg --config http_proxy.always=True clone http://localhost:$HGPORT2/ h
  abort: HTTP Error 404: $ECONNREFUSED$
  [100]

  $ cat proxy.log
  * - - [*] "GET http://localhost:$HGPORT/?cmd=capabilities HTTP/1.1" - - (glob)
  $LOCALIP - - [$LOGDATE$] "GET http://localhost:$HGPORT/?cmd=batch HTTP/1.1" - - x-hgarg-1:cmds=heads+%3Bknown+nodes%3D x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$ partial-pull (glob)
  $LOCALIP - - [$LOGDATE$] "GET http://localhost:$HGPORT/?cmd=getbundle HTTP/1.1" - - x-hgarg-1:bookmarks=1&$USUAL_BUNDLE_CAPS$&cg=0&common=0000000000000000000000000000000000000000&heads=83180e7845de420a1bb46896fd5fe05294f8d629&listkeys=bookmarks&stream=1 x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$ partial-pull (glob)
  * - - [*] "GET http://localhost:$HGPORT/?cmd=capabilities HTTP/1.1" - - (glob)
  $LOCALIP - - [$LOGDATE$] "GET http://localhost:$HGPORT/?cmd=batch HTTP/1.1" - - x-hgarg-1:cmds=heads+%3Bknown+nodes%3D x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$ partial-pull (glob)
  $LOCALIP - - [$LOGDATE$] "GET http://localhost:$HGPORT/?cmd=getbundle HTTP/1.1" - - x-hgarg-1:bookmarks=1&$USUAL_BUNDLE_CAPS$&cg=1&common=0000000000000000000000000000000000000000&heads=83180e7845de420a1bb46896fd5fe05294f8d629&listkeys=bookmarks&phases=1 x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$ partial-pull (glob)
  * - - [*] "GET http://localhost:$HGPORT/?cmd=capabilities HTTP/1.1" - - (glob)
  $LOCALIP - - [$LOGDATE$] "GET http://localhost:$HGPORT/?cmd=batch HTTP/1.1" - - x-hgarg-1:cmds=heads+%3Bknown+nodes%3D x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$ partial-pull (glob)
  $LOCALIP - - [$LOGDATE$] "GET http://localhost:$HGPORT/?cmd=getbundle HTTP/1.1" - - x-hgarg-1:bookmarks=1&$USUAL_BUNDLE_CAPS$&cg=1&common=0000000000000000000000000000000000000000&heads=83180e7845de420a1bb46896fd5fe05294f8d629&listkeys=bookmarks&phases=1 x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$ partial-pull (glob)
  * - - [*] "GET http://localhost:$HGPORT/?cmd=capabilities HTTP/1.1" - - (glob)
  $LOCALIP - - [$LOGDATE$] "GET http://localhost:$HGPORT/?cmd=batch HTTP/1.1" - - x-hgarg-1:cmds=heads+%3Bknown+nodes%3D x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$ partial-pull (glob)
  $LOCALIP - - [$LOGDATE$] "GET http://localhost:$HGPORT/?cmd=getbundle HTTP/1.1" - - x-hgarg-1:bookmarks=1&$USUAL_BUNDLE_CAPS$&cg=1&common=0000000000000000000000000000000000000000&heads=83180e7845de420a1bb46896fd5fe05294f8d629&listkeys=bookmarks&phases=1 x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$ partial-pull (glob)
  * - - [*] "GET http://localhost:$HGPORT/?cmd=capabilities HTTP/1.1" - - (glob)
  $LOCALIP - - [$LOGDATE$] "GET http://localhost:$HGPORT/?cmd=batch HTTP/1.1" - - x-hgarg-1:cmds=heads+%3Bknown+nodes%3D x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$ partial-pull (glob)
  $LOCALIP - - [$LOGDATE$] "GET http://localhost:$HGPORT/?cmd=getbundle HTTP/1.1" - - x-hgarg-1:bookmarks=1&$USUAL_BUNDLE_CAPS$&cg=1&common=0000000000000000000000000000000000000000&heads=83180e7845de420a1bb46896fd5fe05294f8d629&listkeys=bookmarks&phases=1 x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$ partial-pull (glob)
  * - - [*] code 404, message $ECONNREFUSED$ (glob)
  $LOCALIP - - [$LOGDATE$] "GET http://localhost:$HGPORT2/?cmd=capabilities HTTP/1.1" 404 - (glob)