view tests/test-pull-bundle.t @ 51977:42a116f1cdc1

branchmap-v3: introduce a "stop_rev" argument to `headsrevs`

The `headsrevs` method of the revlog already have a `revs` argument to compute
the headrevs of a limited set of heads. However, it disable the use of the
native compiled code to compute the heads, which slows down the branchmap v3
code a lot.

The branchmap v3 usage is actually quite constrained as we will always only
ignores a part at the top of the graph. So we could be significantly faster.

We start by making small change to the python side to improve the situation and
introduce the new API. More collaboration with the native code are coming later.

This massively speedup operation and close most of the remaining gaps between
branchmap-v3 and branchmap-v2. especially on repository with many revs like
mozilla-try. A small overhead remains mostly because the `headrevs` logic
currently has some inefficiently. We will look into them from there.

### benchmark.name                                 = hg.command.unbundle
  # bin-env-vars.hg.py-re2-module                  = default
  # benchmark.variants.issue6528                   = disabled
  # benchmark.variants.resource-usage              = default
  # benchmark.variants.reuse-external-delta-parent = yes
  # benchmark.variants.revs                        = any-1-extra-rev
  # benchmark.variants.source                      = unbundle
  # benchmark.variants.validate                    = default
  # benchmark.variants.verbosity                   = quiet

 ## data-env-vars.name                             = netbeans-2018-08-01-zstd-sparse-revlog
  # bin-env-vars.hg.flavor                         = default
branch-v2:        0.233711  ~~~~~
branch-v3 before: 0.368769  (+57.79%, +0.14)
branch-v3 after:  0.239857   (+2.63%, +0.01)
  # bin-env-vars.hg.flavor                         = rust
branch-v2:        0.235230  ~~~~~
branch-v3 before: 0.372460  (+58.34%, +0.14)
branch-v3 after:  0.240972   (+2.44%, +0.01)
 ## data-env-vars.name                             = netbeans-2018-08-01-ds2-pnm
  # bin-env-vars.hg.flavor                         = rust
branch-v2:        0.255586  ~~~~~
branch-v3 before: 0.318907  (+24.78%, +0.06)
branch-v3 after:  0.268560   (+5.08%, +0.01)

 ## data-env-vars.name                             = mozilla-central-2024-03-22-zstd-sparse-revlog
  # bin-env-vars.hg.flavor                         = default
branch-v2:        0.339010  ~~~~~
branch-v3 before: 0.349752   (+3.17%, +0.01)
branch-v3 after:  0.349389   (+3.06%, +0.01)
  # bin-env-vars.hg.flavor                         = rust
branch-v2:        0.346525  ~~~~~
branch-v3 before: 0.354300   (+2.24%, +0.01)
branch-v3 after:  0.355661   (+2.64%, +0.01)
 ## data-env-vars.name                             = mozilla-central-2024-03-22-ds2-pnm
  # bin-env-vars.hg.flavor                         = rust
branch-v2:        0.380202  ~~~~~
branch-v3 before: 0.396293   (+4.23%, +0.02)
branch-v3 after:  0.408851   (+7.54%, +0.03)

 ## data-env-vars.name                             = mozilla-unified-2024-03-22-zstd-sparse-revlog
  # bin-env-vars.hg.flavor                         = default
branch-v2:        0.412165  ~~~~~
branch-v3 before: 0.424769   (+3.06%, +0.01)
branch-v3 after:  0.427782   (+3.79%, +0.02)
  # bin-env-vars.hg.flavor                         = rust
branch-v2:        0.412397  ~~~~~
branch-v3 before: 0.421796   (+2.28%, +0.01)
branch-v3 after:  0.422354   (+2.41%, +0.01)
 ## data-env-vars.name                             = mozilla-unified-2024-03-22-ds2-pnm
  # bin-env-vars.hg.flavor                         = rust
branch-v2:        0.429501  ~~~~~
branch-v3 before: 0.443849   (+3.34%, +0.01)
branch-v3 after:  0.443197   (+3.19%, +0.01)

 ## data-env-vars.name                             = mozilla-try-2024-03-26-zstd-sparse-revlog
  # bin-env-vars.hg.flavor                         = default
branch-v2:        3.403171  ~~~~~
branch-v3 before: 6.234055  (+83.18%, +2.83)
branch-v3 after:  3.819477  (+12.23%, +0.42)
  # bin-env-vars.hg.flavor                         = rust
branch-v2:        3.454876  ~~~~~
branch-v3 before: 6.307813  (+82.58%, +2.85)
branch-v3 after:  3.590284   (+3.92%, +0.14)
 ## data-env-vars.name                             = mozilla-try-2024-03-26-ds2-pnm
  # bin-env-vars.hg.flavor                         = rust
branch-v2:        3.465435  ~~~~~
branch-v3 before: 5.176076  (+49.36%, +1.71)
branch-v3 after:  3.633278   (+4.84%, +0.17)

#require no-chg

  $ hg init repo
  $ cd repo
  $ hg debugbuilddag '+3<3+1'

  $ hg log
  changeset:   3:6100d3090acf
  tag:         tip
  parent:      0:1ea73414a91b
  user:        debugbuilddag
  date:        Thu Jan 01 00:00:03 1970 +0000
  summary:     r3
  
  changeset:   2:01241442b3c2
  user:        debugbuilddag
  date:        Thu Jan 01 00:00:02 1970 +0000
  summary:     r2
  
  changeset:   1:66f7d451a68b
  user:        debugbuilddag
  date:        Thu Jan 01 00:00:01 1970 +0000
  summary:     r1
  
  changeset:   0:1ea73414a91b
  user:        debugbuilddag
  date:        Thu Jan 01 00:00:00 1970 +0000
  summary:     r0
  
  $ cd ..

Test pullbundle functionality

  $ cd repo
  $ cat <<EOF > .hg/hgrc
  > [experimental]
  > evolution = True
  > [extensions]
  > blackbox =
  > EOF
  $ hg bundle --base null -r 0 .hg/0.hg
  1 changesets found
  $ hg bundle --base 0 -r 1 .hg/1.hg
  1 changesets found
  $ hg bundle --base 1 -r 2 .hg/2.hg
  1 changesets found
  $ hg bundle --base 1 -r 3 .hg/3.hg
  1 changesets found
  $ cat <<EOF > .hg/pullbundles.manifest
  > 3.hg BUNDLESPEC=none-v2 heads=6100d3090acf50ed11ec23196cec20f5bd7323aa bases=1ea73414a91b0920940797d8fc6a11e447f8ea1e
  > 2.hg BUNDLESPEC=none-v2 heads=01241442b3c2bf3211e593b549c655ea65b295e3 bases=66f7d451a68b85ed82ff5fcc254daf50c74144bd
  > 1.hg BUNDLESPEC=bzip2-v2 heads=66f7d451a68b85ed82ff5fcc254daf50c74144bd bases=1ea73414a91b0920940797d8fc6a11e447f8ea1e
  > 0.hg BUNDLESPEC=gzip-v2 heads=1ea73414a91b0920940797d8fc6a11e447f8ea1e
  > EOF
  $ hg --config blackbox.track=debug --debug serve -p $HGPORT2 -d --pid-file=../repo.pid -E ../error.txt
  listening at http://*:$HGPORT2/ (bound to $LOCALIP:$HGPORT2) (glob) (?)
  $ cat ../repo.pid >> $DAEMON_PIDS
  $ cd ..
  $ hg clone -r 0 http://localhost:$HGPORT2/ repo.pullbundle
  adding changesets
  adding manifests
  adding file changes
  added 1 changesets with 0 changes to 0 files
  new changesets 1ea73414a91b (1 drafts)
  updating to branch default
  0 files updated, 0 files merged, 0 files removed, 0 files unresolved
  $ cat error.txt
  $ cd repo.pullbundle
  $ hg pull -r 1
  pulling from http://localhost:$HGPORT2/
  searching for changes
  adding changesets
  adding manifests
  adding file changes
  added 1 changesets with 0 changes to 0 files
  new changesets 66f7d451a68b (1 drafts)
  (run 'hg update' to get a working copy)
  $ hg pull -r 3
  pulling from http://localhost:$HGPORT2/
  searching for changes
  adding changesets
  adding manifests
  adding file changes
  added 1 changesets with 0 changes to 0 files (+1 heads)
  new changesets 6100d3090acf (1 drafts)
  (run 'hg heads' to see heads, 'hg merge' to merge)
  $ cd ..
  $ killdaemons.py
  $ grep 'sending pullbundle ' repo/.hg/blackbox.log
  * sending pullbundle "0.hg" (glob)
  * sending pullbundle "1.hg" (glob)
  * sending pullbundle "3.hg" (glob)
  $ rm repo/.hg/blackbox.log

Test pullbundle functionality for incremental pulls

  $ cd repo
  $ hg --config blackbox.track=debug --debug serve -p $HGPORT2 -d --pid-file=../repo.pid
  listening at http://*:$HGPORT2/ (bound to $LOCALIP:$HGPORT2) (glob) (?)
  $ cat ../repo.pid >> $DAEMON_PIDS
  $ cd ..
  $ hg clone http://localhost:$HGPORT2/ repo.pullbundle2
  requesting all changes
  adding changesets
  adding manifests
  adding file changes
  adding changesets
  adding manifests
  adding file changes
  adding changesets
  adding manifests
  adding file changes
  adding changesets
  adding manifests
  adding file changes
  added 4 changesets with 0 changes to 0 files (+1 heads)
  new changesets 1ea73414a91b:01241442b3c2 (4 drafts)
  updating to branch default
  0 files updated, 0 files merged, 0 files removed, 0 files unresolved
  $ killdaemons.py
  $ grep 'sending pullbundle ' repo/.hg/blackbox.log
  * sending pullbundle "0.hg" (glob)
  * sending pullbundle "3.hg" (glob)
  * sending pullbundle "1.hg" (glob)
  * sending pullbundle "2.hg" (glob)
  $ rm repo/.hg/blackbox.log

Test pullbundle functionality for incoming

  $ cd repo
  $ hg --config blackbox.track=debug --debug serve -p $HGPORT2 -d --pid-file=../repo.pid
  listening at http://*:$HGPORT2/ (bound to $LOCALIP:$HGPORT2) (glob) (?)
  $ cat ../repo.pid >> $DAEMON_PIDS
  $ cd ..
  $ hg clone http://localhost:$HGPORT2/ repo.pullbundle2a -r 0
  adding changesets
  adding manifests
  adding file changes
  added 1 changesets with 0 changes to 0 files
  new changesets 1ea73414a91b (1 drafts)
  updating to branch default
  0 files updated, 0 files merged, 0 files removed, 0 files unresolved
  $ cd repo.pullbundle2a
  $ hg incoming -r 66f7d451a68b
  comparing with http://localhost:$HGPORT2/
  searching for changes
  changeset:   1:66f7d451a68b
  tag:         tip
  user:        debugbuilddag
  date:        Thu Jan 01 00:00:01 1970 +0000
  summary:     r1
  
  $ cd ..
  $ killdaemons.py
  $ grep 'sending pullbundle ' repo/.hg/blackbox.log
  * sending pullbundle "0.hg" (glob)
  * sending pullbundle "1.hg" (glob)
  $ rm repo/.hg/blackbox.log

Test recovery from misconfigured server sending no new data

  $ cd repo
  $ cat <<EOF > .hg/pullbundles.manifest
  > 0.hg heads=66f7d451a68b85ed82ff5fcc254daf50c74144bd bases=1ea73414a91b0920940797d8fc6a11e447f8ea1e
  > 0.hg heads=1ea73414a91b0920940797d8fc6a11e447f8ea1e
  > EOF
  $ hg --config blackbox.track=debug --debug serve -p $HGPORT2 -d --pid-file=../repo.pid
  listening at http://*:$HGPORT2/ (bound to $LOCALIP:$HGPORT2) (glob) (?)
  $ cat ../repo.pid >> $DAEMON_PIDS
  $ cd ..
  $ hg clone -r 0 http://localhost:$HGPORT2/ repo.pullbundle3
  adding changesets
  adding manifests
  adding file changes
  added 1 changesets with 0 changes to 0 files
  new changesets 1ea73414a91b (1 drafts)
  updating to branch default
  0 files updated, 0 files merged, 0 files removed, 0 files unresolved
  $ cd repo.pullbundle3
  $ hg pull -r 1
  pulling from http://localhost:$HGPORT2/
  searching for changes
  adding changesets
  adding manifests
  adding file changes
  abort: 00changelog@66f7d451a68b85ed82ff5fcc254daf50c74144bd: no node
  [50]
  $ cd ..
  $ killdaemons.py
  $ grep 'sending pullbundle ' repo/.hg/blackbox.log
  * sending pullbundle "0.hg" (glob)
  * sending pullbundle "0.hg" (glob)
  $ rm repo/.hg/blackbox.log

Test processing when nodes used in the pullbundles.manifest end up being hidden

  $ hg --repo repo debugobsolete ed1b79f46b9a29f5a6efa59cf12fcfca43bead5a
  1 new obsolescence markers
  $ hg serve --repo repo --config server.view=visible -p $HGPORT -d --pid-file=hg.pid -E errors.log
  $ cat hg.pid >> $DAEMON_PIDS
  $ hg clone http://localhost:$HGPORT repo-obs
  requesting all changes
  adding changesets
  adding manifests
  adding file changes
  adding changesets
  adding manifests
  adding file changes
  added 1 changesets with 0 changes to 0 files
  new changesets 1ea73414a91b (1 drafts)
  updating to branch default
  0 files updated, 0 files merged, 0 files removed, 0 files unresolved
  $ killdaemons.py