annotate contrib/memory.py @ 49773:523cacdfd324

delta-find: set the default candidate chunk size to 10

I ran performance and storage tests on repositories of various sizes and shapes
for the following values of the config : 5, 10, 20, 50, 100, no-chunking

The performance tests do not show any statistical impact on computation
times for large pushes and pulls.

For searching for an individual delta, this can provide a significant
performance improvement with a minor degradation of space-quality on the
result. (see data at the end of the commit).

For overall store size, the change :
- does not have any impact on many small repositories,
- has an observable, but very negligible impact on most larger repositories.
- One private repository we use for testing sees a small increase in size
  (1%) in the narrower version.
  We will try to get more numbers on a larger version of that repository to
  make sure nothing pathological happens.

We pick "10" as the limit as "5" seems a bit more risky.

There are room to improve the current code, by using more aggressive filtering
and better (i.e any) sorting of the candidates. However this is already a large
improvement for pathological cases, with little impact in the common
situations.

The initial motivation for this change is to fix performance of delta
computation for a file where the previous code ended up testing 20 000 possible
candidate-bases in one go, which is… slow. This affected about ½ of the file
revisions leading to atrocious performance, especially during some push/pull
operations.


Details about individual delta finding timing:
----------------------------------------------

The vast majority of benchmark cases are unchanged but the three below. The first
two do not see any impact on the final delta. The last one sees a change in
delta-size that is negligible compared to the full text size.


### data-env-vars.name     = mozilla-try-2019-02-18-zstd-sparse-revlog
  # benchmark.name         = perf-delta-find
  # benchmark.variants.rev = manifest-snapshot-many-tries-a (revision 756096)
∞:   5.844783
5:   4.473523  (-23.46%)
10:  4.970053  (-14.97%)
20:  5.770386   (-1.27%)
50   5.821358
100: 5.834887

MANIFESTLOG: rev = 756096: (no-limit)
    delta-base = 301840
    search-rounds = 6
    try-count = 60
    delta-type = snapshot
    snap-depth = 7
    delta-size = 179

MANIFESTLOG: rev=756096: (limit = 10)
    delta-base=301840
    search-rounds=9
    try-count=51
    delta-type=snapshot
    snap-depth=7
    delta-size=179

### data-env-vars.name     = mozilla-try-2019-02-18-zstd-sparse-revlog
  # benchmark.name         = perf-delta-find
  # benchmark.variants.rev = manifest-snapshot-many-tries-d (revision 754060)
∞:   5.017663
5:   3.655931  (-27.14%)
10:  4.095436  (-18.38%)
20:  4.828949   (-3.76%)
50   4.987574
100: 4.994889

MANIFESTLOG: rev=754060: (no limit)
    delta-base=301840
    search-rounds=5
    try-count=53
    delta-type=snapshot
    snap-depth=7
    delta-size = 179

MANIFESTLOG: rev=754060: (limite = 10)
    delta-base=301840
    search-rounds=8
    try-count=45
    delta-type=snapshot
    snap-depth=7
    delta-size = 179

### data-env-vars.name     = mozilla-try-2019-02-18-zstd-sparse-revlog
  # benchmark.name         = perf-delta-find
  # bin-env-vars.hg.flavor = rust
  # benchmark.variants.rev = manifest-snapshot-many-tries-e (revision 693368)
∞:   4.869282
5:   2.039732  (-58.11%)
10:  2.413537  (-50.43%)
20:  4.449639   (-8.62%)
50   4.865863
100: 4.882649

MANIFESTLOG: rev=693368:
    delta-base=693336
    search-rounds=6
    try-count=53
    delta-type=snapshot
    snap-depth=6
    full-test-size=131065
    delta-size=199

MANIFESTLOG: rev=693368:
    delta-base=278023
    search-rounds=5
    try-count=21
    delta-type=snapshot
    snap-depth=4
    full-test-size=131065
    delta-size=278


Raw data for store size (in bytes) for various chunk size value below:
----------------------------------------------------------------------


   440 134 384          5 pypy/.hg/store/
   440 134 384         10 pypy/.hg/store/
   440 134 384         20 pypy/.hg/store/
   440 134 384         50 pypy/.hg/store/
   440 134 384        100 pypy/.hg/store/
   440 134 384        ... pypy/.hg/store/

   666 987 471          5 netbsd-xsrc-2022-11-15/.hg/store/
   666 987 471         10 netbsd-xsrc-2022-11-15/.hg/store/
   666 987 471         20 netbsd-xsrc-2022-11-15/.hg/store/
   666 987 471         50 netbsd-xsrc-2022-11-15/.hg/store/
   666 987 471        100 netbsd-xsrc-2022-11-15/.hg/store/
   666 987 471        ... netbsd-xsrc-2022-11-15/.hg/store/

   852 844 884          5 netbsd-pkgsrc-2022-11-15/.hg/store/
   852 844 884         10 netbsd-pkgsrc-2022-11-15/.hg/store/
   852 844 884         20 netbsd-pkgsrc-2022-11-15/.hg/store/
   852 844 884         50 netbsd-pkgsrc-2022-11-15/.hg/store/
   852 844 884        100 netbsd-pkgsrc-2022-11-15/.hg/store/
   852 844 884        ... netbsd-pkgsrc-2022-11-15/.hg/store/

 1 504 227 981           5 netbeans-2018-08-01-sparse-zstd/.hg/store/
 1 504 227 871         10 netbeans-2018-08-01-sparse-zstd/.hg/store/
 1 504 227 813         20 netbeans-2018-08-01-sparse-zstd/.hg/store/
 1 504 227 813         50 netbeans-2018-08-01-sparse-zstd/.hg/store/
 1 504 227 813        100 netbeans-2018-08-01-sparse-zstd/.hg/store/
 1 504 227 813        ... netbeans-2018-08-01-sparse-zstd/.hg/store/

 3 875 801 068          5 netbsd-src-2022-11-15/.hg/store/
 3 875 696 767         10 netbsd-src-2022-11-15/.hg/store/
 3 875 696 757         20 netbsd-src-2022-11-15/.hg/store/
 3 875 696 653         50 netbsd-src-2022-11-15/.hg/store/
 3 875 696 653        100 netbsd-src-2022-11-15/.hg/store/
 3 875 696 653        ... netbsd-src-2022-11-15/.hg/store/

 4 531 441 314          5 mozilla-central/.hg/store/
 4 531 435 157         10 mozilla-central/.hg/store/
 4 531 432 045         20 mozilla-central/.hg/store/
 4 531 429 119         50 mozilla-central/.hg/store/
 4 531 429 119        100 mozilla-central/.hg/store/
 4 531 429 119        ... mozilla-central/.hg/store/

 4 875 861 390          5 mozilla-unified/.hg/store/
 4 875 855 155         10 mozilla-unified/.hg/store/
 4 875 852 027         20 mozilla-unified/.hg/store/
 4 875 848 851         50 mozilla-unified/.hg/store/
 4 875 848 851        100 mozilla-unified/.hg/store/
 4 875 848 851        ... mozilla-unified/.hg/store/

11 498 764 601          5 mozilla-try/.hg/store/
11 497 968 858         10 mozilla-try/.hg/store/
11 497 958 730         20 mozilla-try/.hg/store/
11 497 927 156         50 mozilla-try/.hg/store/
11 497 925 963        100 mozilla-try/.hg/store/
11 497 923 428        ... mozilla-try/.hg/store/

10 047 914 031          5 private-repo
 9 969 132 101         10 private-repo
 9 944 745 015         20 private-repo
 9 939 756 703         50 private-repo
 9 939 833 016        100 private-repo
 9 939 822 035        ... private-repo

# memory.py - track memory usage
#
# Copyright 2009 Olivia Mackall <[e-mail]> and others
#
# This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version.

'''helper extension to measure memory usage

Reads current and peak memory usage from ``/proc/self/status`` and
prints it to ``stderr`` on exit.
'''


def memusage(ui):
    """Report memory usage of the current process."""
    result = {'peak': 0, 'rss': 0}
    with open('/proc/self/status', 'r') as status:
        # This will only work on systems with a /proc file system
        # (like Linux).
        for line in status:
            parts = line.split()
            key = parts[0][2:-1].lower()
            if key in result:
                result[key] = int(parts[1])
    ui.write_err(
        ", ".join(
            ["%s: %.1f MiB" % (k, v / 1024.0) for k, v in result.iteritems()]
        )
        + "\n"
    )


def extsetup(ui):
    ui.atexit(memusage, ui)