view tests/test-log-exthook.t @ 49658:523cacdfd324

delta-find: set the default candidate chunk size to 10

I ran performance and storage tests on repositories of various sizes and shapes
for the following values of the config : 5, 10, 20, 50, 100, no-chunking

The performance tests do not show any statistical impact on computation
times for large pushes and pulls.

For searching for an individual delta, this can provide a significant
performance improvement with a minor degradation of space-quality on the
result. (see data at the end of the commit).

For overall store size, the change :
- does not have any impact on many small repositories,
- has an observable, but very negligible impact on most larger repositories.
- One private repository we use for testing sees a small increase in size
  (1%) in the narrower version.
  We will try to get more numbers on a larger version of that repository to
  make sure nothing pathological happens.

We pick "10" as the limit as "5" seems a bit more risky.

There are room to improve the current code, by using more aggressive filtering
and better (i.e any) sorting of the candidates. However this is already a large
improvement for pathological cases, with little impact in the common
situations.

The initial motivation for this change is to fix performance of delta
computation for a file where the previous code ended up testing 20 000 possible
candidate-bases in one go, which is… slow. This affected about ½ of the file
revisions leading to atrocious performance, especially during some push/pull
operations.


Details about individual delta finding timing:
----------------------------------------------

The vast majority of benchmark cases are unchanged but the three below. The first
two do not see any impact on the final delta. The last one sees a change in
delta-size that is negligible compared to the full text size.


### data-env-vars.name     = mozilla-try-2019-02-18-zstd-sparse-revlog
  # benchmark.name         = perf-delta-find
  # benchmark.variants.rev = manifest-snapshot-many-tries-a (revision 756096)
∞:   5.844783
5:   4.473523  (-23.46%)
10:  4.970053  (-14.97%)
20:  5.770386   (-1.27%)
50   5.821358
100: 5.834887

MANIFESTLOG: rev = 756096: (no-limit)
    delta-base = 301840
    search-rounds = 6
    try-count = 60
    delta-type = snapshot
    snap-depth = 7
    delta-size = 179

MANIFESTLOG: rev=756096: (limit = 10)
    delta-base=301840
    search-rounds=9
    try-count=51
    delta-type=snapshot
    snap-depth=7
    delta-size=179

### data-env-vars.name     = mozilla-try-2019-02-18-zstd-sparse-revlog
  # benchmark.name         = perf-delta-find
  # benchmark.variants.rev = manifest-snapshot-many-tries-d (revision 754060)
∞:   5.017663
5:   3.655931  (-27.14%)
10:  4.095436  (-18.38%)
20:  4.828949   (-3.76%)
50   4.987574
100: 4.994889

MANIFESTLOG: rev=754060: (no limit)
    delta-base=301840
    search-rounds=5
    try-count=53
    delta-type=snapshot
    snap-depth=7
    delta-size = 179

MANIFESTLOG: rev=754060: (limite = 10)
    delta-base=301840
    search-rounds=8
    try-count=45
    delta-type=snapshot
    snap-depth=7
    delta-size = 179

### data-env-vars.name     = mozilla-try-2019-02-18-zstd-sparse-revlog
  # benchmark.name         = perf-delta-find
  # bin-env-vars.hg.flavor = rust
  # benchmark.variants.rev = manifest-snapshot-many-tries-e (revision 693368)
∞:   4.869282
5:   2.039732  (-58.11%)
10:  2.413537  (-50.43%)
20:  4.449639   (-8.62%)
50   4.865863
100: 4.882649

MANIFESTLOG: rev=693368:
    delta-base=693336
    search-rounds=6
    try-count=53
    delta-type=snapshot
    snap-depth=6
    full-test-size=131065
    delta-size=199

MANIFESTLOG: rev=693368:
    delta-base=278023
    search-rounds=5
    try-count=21
    delta-type=snapshot
    snap-depth=4
    full-test-size=131065
    delta-size=278


Raw data for store size (in bytes) for various chunk size value below:
----------------------------------------------------------------------


   440 134 384          5 pypy/.hg/store/
   440 134 384         10 pypy/.hg/store/
   440 134 384         20 pypy/.hg/store/
   440 134 384         50 pypy/.hg/store/
   440 134 384        100 pypy/.hg/store/
   440 134 384        ... pypy/.hg/store/

   666 987 471          5 netbsd-xsrc-2022-11-15/.hg/store/
   666 987 471         10 netbsd-xsrc-2022-11-15/.hg/store/
   666 987 471         20 netbsd-xsrc-2022-11-15/.hg/store/
   666 987 471         50 netbsd-xsrc-2022-11-15/.hg/store/
   666 987 471        100 netbsd-xsrc-2022-11-15/.hg/store/
   666 987 471        ... netbsd-xsrc-2022-11-15/.hg/store/

   852 844 884          5 netbsd-pkgsrc-2022-11-15/.hg/store/
   852 844 884         10 netbsd-pkgsrc-2022-11-15/.hg/store/
   852 844 884         20 netbsd-pkgsrc-2022-11-15/.hg/store/
   852 844 884         50 netbsd-pkgsrc-2022-11-15/.hg/store/
   852 844 884        100 netbsd-pkgsrc-2022-11-15/.hg/store/
   852 844 884        ... netbsd-pkgsrc-2022-11-15/.hg/store/

 1 504 227 981           5 netbeans-2018-08-01-sparse-zstd/.hg/store/
 1 504 227 871         10 netbeans-2018-08-01-sparse-zstd/.hg/store/
 1 504 227 813         20 netbeans-2018-08-01-sparse-zstd/.hg/store/
 1 504 227 813         50 netbeans-2018-08-01-sparse-zstd/.hg/store/
 1 504 227 813        100 netbeans-2018-08-01-sparse-zstd/.hg/store/
 1 504 227 813        ... netbeans-2018-08-01-sparse-zstd/.hg/store/

 3 875 801 068          5 netbsd-src-2022-11-15/.hg/store/
 3 875 696 767         10 netbsd-src-2022-11-15/.hg/store/
 3 875 696 757         20 netbsd-src-2022-11-15/.hg/store/
 3 875 696 653         50 netbsd-src-2022-11-15/.hg/store/
 3 875 696 653        100 netbsd-src-2022-11-15/.hg/store/
 3 875 696 653        ... netbsd-src-2022-11-15/.hg/store/

 4 531 441 314          5 mozilla-central/.hg/store/
 4 531 435 157         10 mozilla-central/.hg/store/
 4 531 432 045         20 mozilla-central/.hg/store/
 4 531 429 119         50 mozilla-central/.hg/store/
 4 531 429 119        100 mozilla-central/.hg/store/
 4 531 429 119        ... mozilla-central/.hg/store/

 4 875 861 390          5 mozilla-unified/.hg/store/
 4 875 855 155         10 mozilla-unified/.hg/store/
 4 875 852 027         20 mozilla-unified/.hg/store/
 4 875 848 851         50 mozilla-unified/.hg/store/
 4 875 848 851        100 mozilla-unified/.hg/store/
 4 875 848 851        ... mozilla-unified/.hg/store/

11 498 764 601          5 mozilla-try/.hg/store/
11 497 968 858         10 mozilla-try/.hg/store/
11 497 958 730         20 mozilla-try/.hg/store/
11 497 927 156         50 mozilla-try/.hg/store/
11 497 925 963        100 mozilla-try/.hg/store/
11 497 923 428        ... mozilla-try/.hg/store/

10 047 914 031          5 private-repo
 9 969 132 101         10 private-repo
 9 944 745 015         20 private-repo
 9 939 756 703         50 private-repo
 9 939 833 016        100 private-repo
 9 939 822 035        ... private-repo

Test hg log changeset printer external hook
-------------------------------------------

  $ cat > $TESTTMP/logexthook.py <<EOF
  > import codecs
  > from mercurial import (
  >   commands,
  >   logcmdutil,
  >   repair,
  > )
  > def brot13(b):
  >     return codecs.encode(b.decode('utf8'), 'rot-13').encode('utf8')
  > def rot13description(self, ctx):
  >     description = ctx.description().strip().splitlines()[0]
  >     self.ui.write(b"%s:     %s\n" % (brot13(b"summary"),
  >                                      brot13(description)))
  > def reposetup(ui, repo):
  >     logcmdutil.changesetprinter._exthook = rot13description
  > EOF

Prepare the repository

  $ hg init empty
  $ cd empty
  $ touch ROOT
  $ hg commit -A -m "Root" ROOT

  $ touch a b c
  $ hg commit -A -m "Add A, B, C" a b c

Check the log

  $ hg log --config extensions.t=$TESTTMP/logexthook.py
  changeset:   1:70fc82b23320
  tag:         tip
  user:        test
  date:        Thu Jan 01 00:00:00 1970 +0000
  fhzznel:     Nqq N, O, P
  summary:     Add A, B, C
  
  changeset:   0:b00443a54871
  user:        test
  date:        Thu Jan 01 00:00:00 1970 +0000
  fhzznel:     Ebbg
  summary:     Root
  
Check that exthook is working with graph log too

  $ hg log -G --config extensions.t=$TESTTMP/logexthook.py
  @  changeset:   1:70fc82b23320
  |  tag:         tip
  |  user:        test
  |  date:        Thu Jan 01 00:00:00 1970 +0000
  |  fhzznel:     Nqq N, O, P
  |  summary:     Add A, B, C
  |
  o  changeset:   0:b00443a54871
     user:        test
     date:        Thu Jan 01 00:00:00 1970 +0000
     fhzznel:     Ebbg
     summary:     Root