view tests/test-unrelated-pull.t @ 49658:523cacdfd324

delta-find: set the default candidate chunk size to 10

I ran performance and storage tests on repositories of various sizes and shapes
for the following values of the config : 5, 10, 20, 50, 100, no-chunking

The performance tests do not show any statistical impact on computation
times for large pushes and pulls.

For searching for an individual delta, this can provide a significant
performance improvement with a minor degradation of space-quality on the
result. (see data at the end of the commit).

For overall store size, the change :
- does not have any impact on many small repositories,
- has an observable, but very negligible impact on most larger repositories.
- One private repository we use for testing sees a small increase in size
  (1%) in the narrower version.
  We will try to get more numbers on a larger version of that repository to
  make sure nothing pathological happens.

We pick "10" as the limit as "5" seems a bit more risky.

There are room to improve the current code, by using more aggressive filtering
and better (i.e any) sorting of the candidates. However this is already a large
improvement for pathological cases, with little impact in the common
situations.

The initial motivation for this change is to fix performance of delta
computation for a file where the previous code ended up testing 20 000 possible
candidate-bases in one go, which is… slow. This affected about ½ of the file
revisions leading to atrocious performance, especially during some push/pull
operations.


Details about individual delta finding timing:
----------------------------------------------

The vast majority of benchmark cases are unchanged but the three below. The first
two do not see any impact on the final delta. The last one sees a change in
delta-size that is negligible compared to the full text size.


### data-env-vars.name     = mozilla-try-2019-02-18-zstd-sparse-revlog
  # benchmark.name         = perf-delta-find
  # benchmark.variants.rev = manifest-snapshot-many-tries-a (revision 756096)
∞:   5.844783
5:   4.473523  (-23.46%)
10:  4.970053  (-14.97%)
20:  5.770386   (-1.27%)
50   5.821358
100: 5.834887

MANIFESTLOG: rev = 756096: (no-limit)
    delta-base = 301840
    search-rounds = 6
    try-count = 60
    delta-type = snapshot
    snap-depth = 7
    delta-size = 179

MANIFESTLOG: rev=756096: (limit = 10)
    delta-base=301840
    search-rounds=9
    try-count=51
    delta-type=snapshot
    snap-depth=7
    delta-size=179

### data-env-vars.name     = mozilla-try-2019-02-18-zstd-sparse-revlog
  # benchmark.name         = perf-delta-find
  # benchmark.variants.rev = manifest-snapshot-many-tries-d (revision 754060)
∞:   5.017663
5:   3.655931  (-27.14%)
10:  4.095436  (-18.38%)
20:  4.828949   (-3.76%)
50   4.987574
100: 4.994889

MANIFESTLOG: rev=754060: (no limit)
    delta-base=301840
    search-rounds=5
    try-count=53
    delta-type=snapshot
    snap-depth=7
    delta-size = 179

MANIFESTLOG: rev=754060: (limite = 10)
    delta-base=301840
    search-rounds=8
    try-count=45
    delta-type=snapshot
    snap-depth=7
    delta-size = 179

### data-env-vars.name     = mozilla-try-2019-02-18-zstd-sparse-revlog
  # benchmark.name         = perf-delta-find
  # bin-env-vars.hg.flavor = rust
  # benchmark.variants.rev = manifest-snapshot-many-tries-e (revision 693368)
∞:   4.869282
5:   2.039732  (-58.11%)
10:  2.413537  (-50.43%)
20:  4.449639   (-8.62%)
50   4.865863
100: 4.882649

MANIFESTLOG: rev=693368:
    delta-base=693336
    search-rounds=6
    try-count=53
    delta-type=snapshot
    snap-depth=6
    full-test-size=131065
    delta-size=199

MANIFESTLOG: rev=693368:
    delta-base=278023
    search-rounds=5
    try-count=21
    delta-type=snapshot
    snap-depth=4
    full-test-size=131065
    delta-size=278


Raw data for store size (in bytes) for various chunk size value below:
----------------------------------------------------------------------


   440 134 384          5 pypy/.hg/store/
   440 134 384         10 pypy/.hg/store/
   440 134 384         20 pypy/.hg/store/
   440 134 384         50 pypy/.hg/store/
   440 134 384        100 pypy/.hg/store/
   440 134 384        ... pypy/.hg/store/

   666 987 471          5 netbsd-xsrc-2022-11-15/.hg/store/
   666 987 471         10 netbsd-xsrc-2022-11-15/.hg/store/
   666 987 471         20 netbsd-xsrc-2022-11-15/.hg/store/
   666 987 471         50 netbsd-xsrc-2022-11-15/.hg/store/
   666 987 471        100 netbsd-xsrc-2022-11-15/.hg/store/
   666 987 471        ... netbsd-xsrc-2022-11-15/.hg/store/

   852 844 884          5 netbsd-pkgsrc-2022-11-15/.hg/store/
   852 844 884         10 netbsd-pkgsrc-2022-11-15/.hg/store/
   852 844 884         20 netbsd-pkgsrc-2022-11-15/.hg/store/
   852 844 884         50 netbsd-pkgsrc-2022-11-15/.hg/store/
   852 844 884        100 netbsd-pkgsrc-2022-11-15/.hg/store/
   852 844 884        ... netbsd-pkgsrc-2022-11-15/.hg/store/

 1 504 227 981           5 netbeans-2018-08-01-sparse-zstd/.hg/store/
 1 504 227 871         10 netbeans-2018-08-01-sparse-zstd/.hg/store/
 1 504 227 813         20 netbeans-2018-08-01-sparse-zstd/.hg/store/
 1 504 227 813         50 netbeans-2018-08-01-sparse-zstd/.hg/store/
 1 504 227 813        100 netbeans-2018-08-01-sparse-zstd/.hg/store/
 1 504 227 813        ... netbeans-2018-08-01-sparse-zstd/.hg/store/

 3 875 801 068          5 netbsd-src-2022-11-15/.hg/store/
 3 875 696 767         10 netbsd-src-2022-11-15/.hg/store/
 3 875 696 757         20 netbsd-src-2022-11-15/.hg/store/
 3 875 696 653         50 netbsd-src-2022-11-15/.hg/store/
 3 875 696 653        100 netbsd-src-2022-11-15/.hg/store/
 3 875 696 653        ... netbsd-src-2022-11-15/.hg/store/

 4 531 441 314          5 mozilla-central/.hg/store/
 4 531 435 157         10 mozilla-central/.hg/store/
 4 531 432 045         20 mozilla-central/.hg/store/
 4 531 429 119         50 mozilla-central/.hg/store/
 4 531 429 119        100 mozilla-central/.hg/store/
 4 531 429 119        ... mozilla-central/.hg/store/

 4 875 861 390          5 mozilla-unified/.hg/store/
 4 875 855 155         10 mozilla-unified/.hg/store/
 4 875 852 027         20 mozilla-unified/.hg/store/
 4 875 848 851         50 mozilla-unified/.hg/store/
 4 875 848 851        100 mozilla-unified/.hg/store/
 4 875 848 851        ... mozilla-unified/.hg/store/

11 498 764 601          5 mozilla-try/.hg/store/
11 497 968 858         10 mozilla-try/.hg/store/
11 497 958 730         20 mozilla-try/.hg/store/
11 497 927 156         50 mozilla-try/.hg/store/
11 497 925 963        100 mozilla-try/.hg/store/
11 497 923 428        ... mozilla-try/.hg/store/

10 047 914 031          5 private-repo
 9 969 132 101         10 private-repo
 9 944 745 015         20 private-repo
 9 939 756 703         50 private-repo
 9 939 833 016        100 private-repo
 9 939 822 035        ... private-repo

  $ hg init a
  $ cd a
  $ echo 123 > a
  $ hg add a
  $ hg commit -m "a" -u a

  $ cd ..
  $ hg init b
  $ cd b
  $ echo 321 > b
  $ hg add b
  $ hg commit -m "b" -u b

  $ hg pull ../a
  pulling from ../a
  searching for changes
  abort: repository is unrelated
  [255]

  $ hg pull -f ../a
  pulling from ../a
  searching for changes
  warning: repository is unrelated
  requesting all changes
  adding changesets
  adding manifests
  adding file changes
  added 1 changesets with 1 changes to 1 files (+1 heads)
  new changesets 9a79c33a9db3
  (run 'hg heads' to see heads, 'hg merge' to merge)

  $ hg heads
  changeset:   1:9a79c33a9db3
  tag:         tip
  parent:      -1:000000000000
  user:        a
  date:        Thu Jan 01 00:00:00 1970 +0000
  summary:     a
  
  changeset:   0:01f8062b2de5
  user:        b
  date:        Thu Jan 01 00:00:00 1970 +0000
  summary:     b
  

  $ cd ..