view tests/test-status-rev.t @ 49658:523cacdfd324

delta-find: set the default candidate chunk size to 10

I ran performance and storage tests on repositories of various sizes and shapes
for the following values of the config : 5, 10, 20, 50, 100, no-chunking

The performance tests do not show any statistical impact on computation
times for large pushes and pulls.

For searching for an individual delta, this can provide a significant
performance improvement with a minor degradation of space-quality on the
result. (see data at the end of the commit).

For overall store size, the change :
- does not have any impact on many small repositories,
- has an observable, but very negligible impact on most larger repositories.
- One private repository we use for testing sees a small increase in size
  (1%) in the narrower version.
  We will try to get more numbers on a larger version of that repository to
  make sure nothing pathological happens.

We pick "10" as the limit as "5" seems a bit more risky.

There are room to improve the current code, by using more aggressive filtering
and better (i.e any) sorting of the candidates. However this is already a large
improvement for pathological cases, with little impact in the common
situations.

The initial motivation for this change is to fix performance of delta
computation for a file where the previous code ended up testing 20 000 possible
candidate-bases in one go, which is… slow. This affected about ½ of the file
revisions leading to atrocious performance, especially during some push/pull
operations.


Details about individual delta finding timing:
----------------------------------------------

The vast majority of benchmark cases are unchanged but the three below. The first
two do not see any impact on the final delta. The last one sees a change in
delta-size that is negligible compared to the full text size.


### data-env-vars.name     = mozilla-try-2019-02-18-zstd-sparse-revlog
  # benchmark.name         = perf-delta-find
  # benchmark.variants.rev = manifest-snapshot-many-tries-a (revision 756096)
∞:   5.844783
5:   4.473523  (-23.46%)
10:  4.970053  (-14.97%)
20:  5.770386   (-1.27%)
50   5.821358
100: 5.834887

MANIFESTLOG: rev = 756096: (no-limit)
    delta-base = 301840
    search-rounds = 6
    try-count = 60
    delta-type = snapshot
    snap-depth = 7
    delta-size = 179

MANIFESTLOG: rev=756096: (limit = 10)
    delta-base=301840
    search-rounds=9
    try-count=51
    delta-type=snapshot
    snap-depth=7
    delta-size=179

### data-env-vars.name     = mozilla-try-2019-02-18-zstd-sparse-revlog
  # benchmark.name         = perf-delta-find
  # benchmark.variants.rev = manifest-snapshot-many-tries-d (revision 754060)
∞:   5.017663
5:   3.655931  (-27.14%)
10:  4.095436  (-18.38%)
20:  4.828949   (-3.76%)
50   4.987574
100: 4.994889

MANIFESTLOG: rev=754060: (no limit)
    delta-base=301840
    search-rounds=5
    try-count=53
    delta-type=snapshot
    snap-depth=7
    delta-size = 179

MANIFESTLOG: rev=754060: (limite = 10)
    delta-base=301840
    search-rounds=8
    try-count=45
    delta-type=snapshot
    snap-depth=7
    delta-size = 179

### data-env-vars.name     = mozilla-try-2019-02-18-zstd-sparse-revlog
  # benchmark.name         = perf-delta-find
  # bin-env-vars.hg.flavor = rust
  # benchmark.variants.rev = manifest-snapshot-many-tries-e (revision 693368)
∞:   4.869282
5:   2.039732  (-58.11%)
10:  2.413537  (-50.43%)
20:  4.449639   (-8.62%)
50   4.865863
100: 4.882649

MANIFESTLOG: rev=693368:
    delta-base=693336
    search-rounds=6
    try-count=53
    delta-type=snapshot
    snap-depth=6
    full-test-size=131065
    delta-size=199

MANIFESTLOG: rev=693368:
    delta-base=278023
    search-rounds=5
    try-count=21
    delta-type=snapshot
    snap-depth=4
    full-test-size=131065
    delta-size=278


Raw data for store size (in bytes) for various chunk size value below:
----------------------------------------------------------------------


   440 134 384          5 pypy/.hg/store/
   440 134 384         10 pypy/.hg/store/
   440 134 384         20 pypy/.hg/store/
   440 134 384         50 pypy/.hg/store/
   440 134 384        100 pypy/.hg/store/
   440 134 384        ... pypy/.hg/store/

   666 987 471          5 netbsd-xsrc-2022-11-15/.hg/store/
   666 987 471         10 netbsd-xsrc-2022-11-15/.hg/store/
   666 987 471         20 netbsd-xsrc-2022-11-15/.hg/store/
   666 987 471         50 netbsd-xsrc-2022-11-15/.hg/store/
   666 987 471        100 netbsd-xsrc-2022-11-15/.hg/store/
   666 987 471        ... netbsd-xsrc-2022-11-15/.hg/store/

   852 844 884          5 netbsd-pkgsrc-2022-11-15/.hg/store/
   852 844 884         10 netbsd-pkgsrc-2022-11-15/.hg/store/
   852 844 884         20 netbsd-pkgsrc-2022-11-15/.hg/store/
   852 844 884         50 netbsd-pkgsrc-2022-11-15/.hg/store/
   852 844 884        100 netbsd-pkgsrc-2022-11-15/.hg/store/
   852 844 884        ... netbsd-pkgsrc-2022-11-15/.hg/store/

 1 504 227 981           5 netbeans-2018-08-01-sparse-zstd/.hg/store/
 1 504 227 871         10 netbeans-2018-08-01-sparse-zstd/.hg/store/
 1 504 227 813         20 netbeans-2018-08-01-sparse-zstd/.hg/store/
 1 504 227 813         50 netbeans-2018-08-01-sparse-zstd/.hg/store/
 1 504 227 813        100 netbeans-2018-08-01-sparse-zstd/.hg/store/
 1 504 227 813        ... netbeans-2018-08-01-sparse-zstd/.hg/store/

 3 875 801 068          5 netbsd-src-2022-11-15/.hg/store/
 3 875 696 767         10 netbsd-src-2022-11-15/.hg/store/
 3 875 696 757         20 netbsd-src-2022-11-15/.hg/store/
 3 875 696 653         50 netbsd-src-2022-11-15/.hg/store/
 3 875 696 653        100 netbsd-src-2022-11-15/.hg/store/
 3 875 696 653        ... netbsd-src-2022-11-15/.hg/store/

 4 531 441 314          5 mozilla-central/.hg/store/
 4 531 435 157         10 mozilla-central/.hg/store/
 4 531 432 045         20 mozilla-central/.hg/store/
 4 531 429 119         50 mozilla-central/.hg/store/
 4 531 429 119        100 mozilla-central/.hg/store/
 4 531 429 119        ... mozilla-central/.hg/store/

 4 875 861 390          5 mozilla-unified/.hg/store/
 4 875 855 155         10 mozilla-unified/.hg/store/
 4 875 852 027         20 mozilla-unified/.hg/store/
 4 875 848 851         50 mozilla-unified/.hg/store/
 4 875 848 851        100 mozilla-unified/.hg/store/
 4 875 848 851        ... mozilla-unified/.hg/store/

11 498 764 601          5 mozilla-try/.hg/store/
11 497 968 858         10 mozilla-try/.hg/store/
11 497 958 730         20 mozilla-try/.hg/store/
11 497 927 156         50 mozilla-try/.hg/store/
11 497 925 963        100 mozilla-try/.hg/store/
11 497 923 428        ... mozilla-try/.hg/store/

10 047 914 031          5 private-repo
 9 969 132 101         10 private-repo
 9 944 745 015         20 private-repo
 9 939 756 703         50 private-repo
 9 939 833 016        100 private-repo
 9 939 822 035        ... private-repo

Tests of 'hg status --rev <rev>' to make sure status between <rev> and '.' get
combined correctly with the dirstate status.

  $ hg init repo
  $ cd repo

First commit

  $ "$PYTHON" $TESTDIR/generate-working-copy-states.py state 2 1
  $ hg addremove --similarity 0
  adding content1_content1_content1-tracked
  adding content1_content1_content1-untracked
  adding content1_content1_content3-tracked
  adding content1_content1_content3-untracked
  adding content1_content1_missing-tracked
  adding content1_content1_missing-untracked
  adding content1_content2_content1-tracked
  adding content1_content2_content1-untracked
  adding content1_content2_content2-tracked
  adding content1_content2_content2-untracked
  adding content1_content2_content3-tracked
  adding content1_content2_content3-untracked
  adding content1_content2_missing-tracked
  adding content1_content2_missing-untracked
  adding content1_missing_content1-tracked
  adding content1_missing_content1-untracked
  adding content1_missing_content3-tracked
  adding content1_missing_content3-untracked
  adding content1_missing_missing-tracked
  adding content1_missing_missing-untracked
  $ hg commit -m first

Second commit

  $ "$PYTHON" $TESTDIR/generate-working-copy-states.py state 2 2
  $ hg addremove --similarity 0
  removing content1_missing_content1-tracked
  removing content1_missing_content1-untracked
  removing content1_missing_content3-tracked
  removing content1_missing_content3-untracked
  removing content1_missing_missing-tracked
  removing content1_missing_missing-untracked
  adding missing_content2_content2-tracked
  adding missing_content2_content2-untracked
  adding missing_content2_content3-tracked
  adding missing_content2_content3-untracked
  adding missing_content2_missing-tracked
  adding missing_content2_missing-untracked
  $ hg commit -m second

Working copy

  $ "$PYTHON" $TESTDIR/generate-working-copy-states.py state 2 wc
  $ hg addremove --similarity 0
  adding content1_missing_content1-tracked
  adding content1_missing_content1-untracked
  adding content1_missing_content3-tracked
  adding content1_missing_content3-untracked
  adding content1_missing_missing-tracked
  adding content1_missing_missing-untracked
  adding missing_missing_content3-tracked
  adding missing_missing_content3-untracked
  adding missing_missing_missing-tracked
  adding missing_missing_missing-untracked
  $ hg forget *_*_*-untracked
  $ rm *_*_missing-*

Status compared to parent of the working copy, i.e. the dirstate status

  $ hg status -A --rev 1 'glob:missing_content2_content3-tracked'
  M missing_content2_content3-tracked
  $ hg status -A --rev 1 'glob:missing_content2_content2-tracked'
  C missing_content2_content2-tracked
  $ hg status -A --rev 1 'glob:missing_missing_content3-tracked'
  A missing_missing_content3-tracked
  $ hg status -A --rev 1 'glob:missing_missing_content3-untracked'
  ? missing_missing_content3-untracked
  $ hg status -A --rev 1 'glob:missing_content2_*-untracked'
  R missing_content2_content2-untracked
  R missing_content2_content3-untracked
  R missing_content2_missing-untracked
  $ hg status -A --rev 1 'glob:missing_*_missing-tracked'
  ! missing_content2_missing-tracked
  ! missing_missing_missing-tracked

  $ hg status -A --rev 1 'glob:missing_missing_missing-untracked'
  missing_missing_missing-untracked: $ENOENT$

Status between first and second commit. Should ignore dirstate status.

  $ hg status -A --rev 0:1 'glob:content1_content2_*'
  M content1_content2_content1-tracked
  M content1_content2_content1-untracked
  M content1_content2_content2-tracked
  M content1_content2_content2-untracked
  M content1_content2_content3-tracked
  M content1_content2_content3-untracked
  M content1_content2_missing-tracked
  M content1_content2_missing-untracked
  $ hg status -A --rev 0:1 'glob:content1_content1_*'
  C content1_content1_content1-tracked
  C content1_content1_content1-untracked
  C content1_content1_content3-tracked
  C content1_content1_content3-untracked
  C content1_content1_missing-tracked
  C content1_content1_missing-untracked
  $ hg status -A --rev 0:1 'glob:missing_content2_*'
  A missing_content2_content2-tracked
  A missing_content2_content2-untracked
  A missing_content2_content3-tracked
  A missing_content2_content3-untracked
  A missing_content2_missing-tracked
  A missing_content2_missing-untracked
  $ hg status -A --rev 0:1 'glob:content1_missing_*'
  R content1_missing_content1-tracked
  R content1_missing_content1-untracked
  R content1_missing_content3-tracked
  R content1_missing_content3-untracked
  R content1_missing_missing-tracked
  R content1_missing_missing-untracked
  $ hg status -A --rev 0:1 'glob:missing_missing_*'

Status compared to one revision back, checking that the dirstate status
is correctly combined with the inter-revision status

  $ hg status -A --rev 0 'glob:content1_*_content[23]-tracked'
  M content1_content1_content3-tracked
  M content1_content2_content2-tracked
  M content1_content2_content3-tracked
  M content1_missing_content3-tracked
  $ hg status -A --rev 0 'glob:content1_*_content1-tracked'
  C content1_content1_content1-tracked
  C content1_content2_content1-tracked
  C content1_missing_content1-tracked
  $ hg status -A --rev 0 'glob:missing_*_content?-tracked'
  A missing_content2_content2-tracked
  A missing_content2_content3-tracked
  A missing_missing_content3-tracked
BROKEN: missing_content2_content[23]-untracked exist, so should be listed
  $ hg status -A --rev 0 'glob:missing_*_content?-untracked'
  ? missing_missing_content3-untracked
  $ hg status -A --rev 0 'glob:content1_*_*-untracked'
  R content1_content1_content1-untracked
  R content1_content1_content3-untracked
  R content1_content1_missing-untracked
  R content1_content2_content1-untracked
  R content1_content2_content2-untracked
  R content1_content2_content3-untracked
  R content1_content2_missing-untracked
  R content1_missing_content1-untracked
  R content1_missing_content3-untracked
  R content1_missing_missing-untracked
  $ hg status -A --rev 0 'glob:*_*_missing-tracked'
  ! content1_content1_missing-tracked
  ! content1_content2_missing-tracked
  ! content1_missing_missing-tracked
  ! missing_content2_missing-tracked
  ! missing_missing_missing-tracked
  $ hg status -A --rev 0 'glob:missing_*_missing-untracked'