view contrib/check-pytype.sh @ 49773:523cacdfd324

delta-find: set the default candidate chunk size to 10

I ran performance and storage tests on repositories of various sizes and shapes
for the following values of the config : 5, 10, 20, 50, 100, no-chunking

The performance tests do not show any statistical impact on computation
times for large pushes and pulls.

For searching for an individual delta, this can provide a significant
performance improvement with a minor degradation of space-quality on the
result. (see data at the end of the commit).

For overall store size, the change :
- does not have any impact on many small repositories,
- has an observable, but very negligible impact on most larger repositories.
- One private repository we use for testing sees a small increase in size
  (1%) in the narrower version.
  We will try to get more numbers on a larger version of that repository to
  make sure nothing pathological happens.

We pick "10" as the limit as "5" seems a bit more risky.

There are room to improve the current code, by using more aggressive filtering
and better (i.e any) sorting of the candidates. However this is already a large
improvement for pathological cases, with little impact in the common
situations.

The initial motivation for this change is to fix performance of delta
computation for a file where the previous code ended up testing 20 000 possible
candidate-bases in one go, which is… slow. This affected about ½ of the file
revisions leading to atrocious performance, especially during some push/pull
operations.


Details about individual delta finding timing:
----------------------------------------------

The vast majority of benchmark cases are unchanged but the three below. The first
two do not see any impact on the final delta. The last one sees a change in
delta-size that is negligible compared to the full text size.


### data-env-vars.name     = mozilla-try-2019-02-18-zstd-sparse-revlog
  # benchmark.name         = perf-delta-find
  # benchmark.variants.rev = manifest-snapshot-many-tries-a (revision 756096)
∞:   5.844783
5:   4.473523  (-23.46%)
10:  4.970053  (-14.97%)
20:  5.770386   (-1.27%)
50   5.821358
100: 5.834887

MANIFESTLOG: rev = 756096: (no-limit)
    delta-base = 301840
    search-rounds = 6
    try-count = 60
    delta-type = snapshot
    snap-depth = 7
    delta-size = 179

MANIFESTLOG: rev=756096: (limit = 10)
    delta-base=301840
    search-rounds=9
    try-count=51
    delta-type=snapshot
    snap-depth=7
    delta-size=179

### data-env-vars.name     = mozilla-try-2019-02-18-zstd-sparse-revlog
  # benchmark.name         = perf-delta-find
  # benchmark.variants.rev = manifest-snapshot-many-tries-d (revision 754060)
∞:   5.017663
5:   3.655931  (-27.14%)
10:  4.095436  (-18.38%)
20:  4.828949   (-3.76%)
50   4.987574
100: 4.994889

MANIFESTLOG: rev=754060: (no limit)
    delta-base=301840
    search-rounds=5
    try-count=53
    delta-type=snapshot
    snap-depth=7
    delta-size = 179

MANIFESTLOG: rev=754060: (limite = 10)
    delta-base=301840
    search-rounds=8
    try-count=45
    delta-type=snapshot
    snap-depth=7
    delta-size = 179

### data-env-vars.name     = mozilla-try-2019-02-18-zstd-sparse-revlog
  # benchmark.name         = perf-delta-find
  # bin-env-vars.hg.flavor = rust
  # benchmark.variants.rev = manifest-snapshot-many-tries-e (revision 693368)
∞:   4.869282
5:   2.039732  (-58.11%)
10:  2.413537  (-50.43%)
20:  4.449639   (-8.62%)
50   4.865863
100: 4.882649

MANIFESTLOG: rev=693368:
    delta-base=693336
    search-rounds=6
    try-count=53
    delta-type=snapshot
    snap-depth=6
    full-test-size=131065
    delta-size=199

MANIFESTLOG: rev=693368:
    delta-base=278023
    search-rounds=5
    try-count=21
    delta-type=snapshot
    snap-depth=4
    full-test-size=131065
    delta-size=278


Raw data for store size (in bytes) for various chunk size value below:
----------------------------------------------------------------------


   440 134 384          5 pypy/.hg/store/
   440 134 384         10 pypy/.hg/store/
   440 134 384         20 pypy/.hg/store/
   440 134 384         50 pypy/.hg/store/
   440 134 384        100 pypy/.hg/store/
   440 134 384        ... pypy/.hg/store/

   666 987 471          5 netbsd-xsrc-2022-11-15/.hg/store/
   666 987 471         10 netbsd-xsrc-2022-11-15/.hg/store/
   666 987 471         20 netbsd-xsrc-2022-11-15/.hg/store/
   666 987 471         50 netbsd-xsrc-2022-11-15/.hg/store/
   666 987 471        100 netbsd-xsrc-2022-11-15/.hg/store/
   666 987 471        ... netbsd-xsrc-2022-11-15/.hg/store/

   852 844 884          5 netbsd-pkgsrc-2022-11-15/.hg/store/
   852 844 884         10 netbsd-pkgsrc-2022-11-15/.hg/store/
   852 844 884         20 netbsd-pkgsrc-2022-11-15/.hg/store/
   852 844 884         50 netbsd-pkgsrc-2022-11-15/.hg/store/
   852 844 884        100 netbsd-pkgsrc-2022-11-15/.hg/store/
   852 844 884        ... netbsd-pkgsrc-2022-11-15/.hg/store/

 1 504 227 981           5 netbeans-2018-08-01-sparse-zstd/.hg/store/
 1 504 227 871         10 netbeans-2018-08-01-sparse-zstd/.hg/store/
 1 504 227 813         20 netbeans-2018-08-01-sparse-zstd/.hg/store/
 1 504 227 813         50 netbeans-2018-08-01-sparse-zstd/.hg/store/
 1 504 227 813        100 netbeans-2018-08-01-sparse-zstd/.hg/store/
 1 504 227 813        ... netbeans-2018-08-01-sparse-zstd/.hg/store/

 3 875 801 068          5 netbsd-src-2022-11-15/.hg/store/
 3 875 696 767         10 netbsd-src-2022-11-15/.hg/store/
 3 875 696 757         20 netbsd-src-2022-11-15/.hg/store/
 3 875 696 653         50 netbsd-src-2022-11-15/.hg/store/
 3 875 696 653        100 netbsd-src-2022-11-15/.hg/store/
 3 875 696 653        ... netbsd-src-2022-11-15/.hg/store/

 4 531 441 314          5 mozilla-central/.hg/store/
 4 531 435 157         10 mozilla-central/.hg/store/
 4 531 432 045         20 mozilla-central/.hg/store/
 4 531 429 119         50 mozilla-central/.hg/store/
 4 531 429 119        100 mozilla-central/.hg/store/
 4 531 429 119        ... mozilla-central/.hg/store/

 4 875 861 390          5 mozilla-unified/.hg/store/
 4 875 855 155         10 mozilla-unified/.hg/store/
 4 875 852 027         20 mozilla-unified/.hg/store/
 4 875 848 851         50 mozilla-unified/.hg/store/
 4 875 848 851        100 mozilla-unified/.hg/store/
 4 875 848 851        ... mozilla-unified/.hg/store/

11 498 764 601          5 mozilla-try/.hg/store/
11 497 968 858         10 mozilla-try/.hg/store/
11 497 958 730         20 mozilla-try/.hg/store/
11 497 927 156         50 mozilla-try/.hg/store/
11 497 925 963        100 mozilla-try/.hg/store/
11 497 923 428        ... mozilla-try/.hg/store/

10 047 914 031          5 private-repo
 9 969 132 101         10 private-repo
 9 944 745 015         20 private-repo
 9 939 756 703         50 private-repo
 9 939 833 016        100 private-repo
 9 939 822 035        ... private-repo

#!/bin/sh

set -e
set -u

cd `hg root`

# Many of the individual files that are excluded here confuse pytype
# because they do a mix of Python 2 and Python 3 things
# conditionally. There's no good way to help it out with that as far as
# I can tell, so let's just hide those files from it for now. We should
# endeavor to empty this list out over time, as some of these are
# probably hiding real problems.
#
# mercurial/bundlerepo.py       # no vfs and ui attrs on bundlerepo
# mercurial/context.py          # many [attribute-error]
# mercurial/crecord.py          # tons of [attribute-error], [module-attr]
# mercurial/debugcommands.py    # [wrong-arg-types]
# mercurial/dispatch.py         # initstdio: No attribute ... on TextIO [attribute-error]
# mercurial/exchange.py         # [attribute-error]
# mercurial/hgweb/hgweb_mod.py  # [attribute-error], [name-error], [wrong-arg-types]
# mercurial/hgweb/server.py     # [attribute-error], [name-error], [module-attr]
# mercurial/hgweb/wsgicgi.py    # confused values in os.environ
# mercurial/httppeer.py         # [attribute-error], [wrong-arg-types]
# mercurial/interfaces          # No attribute 'capabilities' on peer [attribute-error]
# mercurial/keepalive.py        # [attribute-error]
# mercurial/localrepo.py        # [attribute-error]
# mercurial/manifest.py         # [unsupported-operands], [wrong-arg-types]
# mercurial/minirst.py          # [unsupported-operands], [attribute-error]
# mercurial/pure/osutil.py      # [invalid-typevar], [not-callable]
# mercurial/pure/parsers.py     # [attribute-error]
# mercurial/repoview.py         # [attribute-error]
# mercurial/testing/storage.py  # tons of [attribute-error]
# mercurial/ui.py               # [attribute-error], [wrong-arg-types]
# mercurial/unionrepo.py        # ui, svfs, unfiltered [attribute-error]
# mercurial/win32.py            # [not-callable]
# mercurial/wireprotoframing.py # [unsupported-operands], [attribute-error], [import-error]
# mercurial/wireprotov1peer.py  # [attribute-error]
# mercurial/wireprotov1server.py  # BUG?: BundleValueError handler accesses subclass's attrs

# TODO: use --no-cache on test server?  Caching the files locally helps during
#       development, but may be a hinderance for CI testing.

# TODO: include hgext and hgext3rd

pytype -V 3.7 --keep-going --jobs auto mercurial \
    -x mercurial/bundlerepo.py \
    -x mercurial/context.py \
    -x mercurial/crecord.py \
    -x mercurial/debugcommands.py \
    -x mercurial/dispatch.py \
    -x mercurial/exchange.py \
    -x mercurial/hgweb/hgweb_mod.py \
    -x mercurial/hgweb/server.py \
    -x mercurial/hgweb/wsgicgi.py \
    -x mercurial/httppeer.py \
    -x mercurial/interfaces \
    -x mercurial/keepalive.py \
    -x mercurial/localrepo.py \
    -x mercurial/manifest.py \
    -x mercurial/minirst.py \
    -x mercurial/pure/osutil.py \
    -x mercurial/pure/parsers.py \
    -x mercurial/repoview.py \
    -x mercurial/testing/storage.py \
    -x mercurial/thirdparty \
    -x mercurial/ui.py \
    -x mercurial/unionrepo.py \
    -x mercurial/win32.py \
    -x mercurial/wireprotoframing.py \
    -x mercurial/wireprotov1peer.py \
    -x mercurial/wireprotov1server.py

echo 'pytype crashed while generating the following type stubs:'
find .pytype/pyi -name '*.pyi' | xargs grep -l '# Caught error' | sort