Mercurial > hg
view hgext/churn.py @ 49658:523cacdfd324
delta-find: set the default candidate chunk size to 10
I ran performance and storage tests on repositories of various sizes and shapes
for the following values of the config : 5, 10, 20, 50, 100, no-chunking
The performance tests do not show any statistical impact on computation
times for large pushes and pulls.
For searching for an individual delta, this can provide a significant
performance improvement with a minor degradation of space-quality on the
result. (see data at the end of the commit).
For overall store size, the change :
- does not have any impact on many small repositories,
- has an observable, but very negligible impact on most larger repositories.
- One private repository we use for testing sees a small increase in size
(1%) in the narrower version.
We will try to get more numbers on a larger version of that repository to
make sure nothing pathological happens.
We pick "10" as the limit as "5" seems a bit more risky.
There are room to improve the current code, by using more aggressive filtering
and better (i.e any) sorting of the candidates. However this is already a large
improvement for pathological cases, with little impact in the common
situations.
The initial motivation for this change is to fix performance of delta
computation for a file where the previous code ended up testing 20 000 possible
candidate-bases in one go, which is… slow. This affected about ½ of the file
revisions leading to atrocious performance, especially during some push/pull
operations.
Details about individual delta finding timing:
----------------------------------------------
The vast majority of benchmark cases are unchanged but the three below. The first
two do not see any impact on the final delta. The last one sees a change in
delta-size that is negligible compared to the full text size.
### data-env-vars.name = mozilla-try-2019-02-18-zstd-sparse-revlog
# benchmark.name = perf-delta-find
# benchmark.variants.rev = manifest-snapshot-many-tries-a (revision 756096)
∞: 5.844783
5: 4.473523 (-23.46%)
10: 4.970053 (-14.97%)
20: 5.770386 (-1.27%)
50 5.821358
100: 5.834887
MANIFESTLOG: rev = 756096: (no-limit)
delta-base = 301840
search-rounds = 6
try-count = 60
delta-type = snapshot
snap-depth = 7
delta-size = 179
MANIFESTLOG: rev=756096: (limit = 10)
delta-base=301840
search-rounds=9
try-count=51
delta-type=snapshot
snap-depth=7
delta-size=179
### data-env-vars.name = mozilla-try-2019-02-18-zstd-sparse-revlog
# benchmark.name = perf-delta-find
# benchmark.variants.rev = manifest-snapshot-many-tries-d (revision 754060)
∞: 5.017663
5: 3.655931 (-27.14%)
10: 4.095436 (-18.38%)
20: 4.828949 (-3.76%)
50 4.987574
100: 4.994889
MANIFESTLOG: rev=754060: (no limit)
delta-base=301840
search-rounds=5
try-count=53
delta-type=snapshot
snap-depth=7
delta-size = 179
MANIFESTLOG: rev=754060: (limite = 10)
delta-base=301840
search-rounds=8
try-count=45
delta-type=snapshot
snap-depth=7
delta-size = 179
### data-env-vars.name = mozilla-try-2019-02-18-zstd-sparse-revlog
# benchmark.name = perf-delta-find
# bin-env-vars.hg.flavor = rust
# benchmark.variants.rev = manifest-snapshot-many-tries-e (revision 693368)
∞: 4.869282
5: 2.039732 (-58.11%)
10: 2.413537 (-50.43%)
20: 4.449639 (-8.62%)
50 4.865863
100: 4.882649
MANIFESTLOG: rev=693368:
delta-base=693336
search-rounds=6
try-count=53
delta-type=snapshot
snap-depth=6
full-test-size=131065
delta-size=199
MANIFESTLOG: rev=693368:
delta-base=278023
search-rounds=5
try-count=21
delta-type=snapshot
snap-depth=4
full-test-size=131065
delta-size=278
Raw data for store size (in bytes) for various chunk size value below:
----------------------------------------------------------------------
440 134 384 5 pypy/.hg/store/
440 134 384 10 pypy/.hg/store/
440 134 384 20 pypy/.hg/store/
440 134 384 50 pypy/.hg/store/
440 134 384 100 pypy/.hg/store/
440 134 384 ... pypy/.hg/store/
666 987 471 5 netbsd-xsrc-2022-11-15/.hg/store/
666 987 471 10 netbsd-xsrc-2022-11-15/.hg/store/
666 987 471 20 netbsd-xsrc-2022-11-15/.hg/store/
666 987 471 50 netbsd-xsrc-2022-11-15/.hg/store/
666 987 471 100 netbsd-xsrc-2022-11-15/.hg/store/
666 987 471 ... netbsd-xsrc-2022-11-15/.hg/store/
852 844 884 5 netbsd-pkgsrc-2022-11-15/.hg/store/
852 844 884 10 netbsd-pkgsrc-2022-11-15/.hg/store/
852 844 884 20 netbsd-pkgsrc-2022-11-15/.hg/store/
852 844 884 50 netbsd-pkgsrc-2022-11-15/.hg/store/
852 844 884 100 netbsd-pkgsrc-2022-11-15/.hg/store/
852 844 884 ... netbsd-pkgsrc-2022-11-15/.hg/store/
1 504 227 981 5 netbeans-2018-08-01-sparse-zstd/.hg/store/
1 504 227 871 10 netbeans-2018-08-01-sparse-zstd/.hg/store/
1 504 227 813 20 netbeans-2018-08-01-sparse-zstd/.hg/store/
1 504 227 813 50 netbeans-2018-08-01-sparse-zstd/.hg/store/
1 504 227 813 100 netbeans-2018-08-01-sparse-zstd/.hg/store/
1 504 227 813 ... netbeans-2018-08-01-sparse-zstd/.hg/store/
3 875 801 068 5 netbsd-src-2022-11-15/.hg/store/
3 875 696 767 10 netbsd-src-2022-11-15/.hg/store/
3 875 696 757 20 netbsd-src-2022-11-15/.hg/store/
3 875 696 653 50 netbsd-src-2022-11-15/.hg/store/
3 875 696 653 100 netbsd-src-2022-11-15/.hg/store/
3 875 696 653 ... netbsd-src-2022-11-15/.hg/store/
4 531 441 314 5 mozilla-central/.hg/store/
4 531 435 157 10 mozilla-central/.hg/store/
4 531 432 045 20 mozilla-central/.hg/store/
4 531 429 119 50 mozilla-central/.hg/store/
4 531 429 119 100 mozilla-central/.hg/store/
4 531 429 119 ... mozilla-central/.hg/store/
4 875 861 390 5 mozilla-unified/.hg/store/
4 875 855 155 10 mozilla-unified/.hg/store/
4 875 852 027 20 mozilla-unified/.hg/store/
4 875 848 851 50 mozilla-unified/.hg/store/
4 875 848 851 100 mozilla-unified/.hg/store/
4 875 848 851 ... mozilla-unified/.hg/store/
11 498 764 601 5 mozilla-try/.hg/store/
11 497 968 858 10 mozilla-try/.hg/store/
11 497 958 730 20 mozilla-try/.hg/store/
11 497 927 156 50 mozilla-try/.hg/store/
11 497 925 963 100 mozilla-try/.hg/store/
11 497 923 428 ... mozilla-try/.hg/store/
10 047 914 031 5 private-repo
9 969 132 101 10 private-repo
9 944 745 015 20 private-repo
9 939 756 703 50 private-repo
9 939 833 016 100 private-repo
9 939 822 035 ... private-repo
author | Pierre-Yves David <pierre-yves.david@octobus.net> |
---|---|
date | Wed, 23 Nov 2022 19:08:27 +0100 |
parents | 6000f5b25c9b |
children | 5a22b2594d96 |
line wrap: on
line source
# churn.py - create a graph of revisions count grouped by template # # Copyright 2006 Josef "Jeff" Sipek <jeffpc@josefsipek.net> # Copyright 2008 Alexander Solovyov <piranha@piranha.org.ua> # # This software may be used and distributed according to the terms of the # GNU General Public License version 2 or any later version. '''command to display statistics about repository history''' import datetime import os import time from mercurial.i18n import _ from mercurial.pycompat import open from mercurial import ( cmdutil, encoding, logcmdutil, patch, pycompat, registrar, scmutil, ) cmdtable = {} command = registrar.command(cmdtable) # Note for extension authors: ONLY specify testedwith = 'ships-with-hg-core' for # extensions which SHIP WITH MERCURIAL. Non-mainline extensions should # be specifying the version(s) of Mercurial they are tested with, or # leave the attribute unspecified. testedwith = b'ships-with-hg-core' def changedlines(ui, repo, ctx1, ctx2, fmatch): added, removed = 0, 0 diff = b''.join(patch.diff(repo, ctx1.node(), ctx2.node(), fmatch)) inhunk = False for l in diff.split(b'\n'): if inhunk and l.startswith(b"+"): added += 1 elif inhunk and l.startswith(b"-"): removed += 1 elif l.startswith(b"@"): inhunk = True elif l.startswith(b"d"): inhunk = False return (added, removed) def countrate(ui, repo, amap, *pats, **opts): """Calculate stats""" opts = pycompat.byteskwargs(opts) if opts.get(b'dateformat'): def getkey(ctx): t, tz = ctx.date() date = datetime.datetime(*time.gmtime(float(t) - tz)[:6]) return encoding.strtolocal( date.strftime(encoding.strfromlocal(opts[b'dateformat'])) ) else: tmpl = opts.get(b'oldtemplate') or opts.get(b'template') tmpl = logcmdutil.maketemplater(ui, repo, tmpl) def getkey(ctx): ui.pushbuffer() tmpl.show(ctx) return ui.popbuffer() progress = ui.makeprogress( _(b'analyzing'), unit=_(b'revisions'), total=len(repo) ) rate = {} def prep(ctx, fmatch): rev = ctx.rev() key = getkey(ctx).strip() key = amap.get(key, key) # alias remap if opts.get(b'changesets'): rate[key] = (rate.get(key, (0,))[0] + 1, 0) else: parents = ctx.parents() if len(parents) > 1: ui.note(_(b'revision %d is a merge, ignoring...\n') % (rev,)) return ctx1 = parents[0] lines = changedlines(ui, repo, ctx1, ctx, fmatch) rate[key] = [r + l for r, l in zip(rate.get(key, (0, 0)), lines)] progress.increment() wopts = logcmdutil.walkopts( pats=pats, opts=opts, revspec=opts[b'rev'], date=opts[b'date'], include_pats=opts[b'include'], exclude_pats=opts[b'exclude'], ) revs, makefilematcher = logcmdutil.makewalker(repo, wopts) for ctx in scmutil.walkchangerevs(repo, revs, makefilematcher, prep): continue progress.complete() return rate @command( b'churn', [ ( b'r', b'rev', [], _(b'count rate for the specified revision or revset'), _(b'REV'), ), ( b'd', b'date', b'', _(b'count rate for revisions matching date spec'), _(b'DATE'), ), ( b't', b'oldtemplate', b'', _(b'template to group changesets (DEPRECATED)'), _(b'TEMPLATE'), ), ( b'T', b'template', b'{author|email}', _(b'template to group changesets'), _(b'TEMPLATE'), ), ( b'f', b'dateformat', b'', _(b'strftime-compatible format for grouping by date'), _(b'FORMAT'), ), (b'c', b'changesets', False, _(b'count rate by number of changesets')), (b's', b'sort', False, _(b'sort by key (default: sort by count)')), (b'', b'diffstat', False, _(b'display added/removed lines separately')), (b'', b'aliases', b'', _(b'file with email aliases'), _(b'FILE')), ] + cmdutil.walkopts, _(b"hg churn [-d DATE] [-r REV] [--aliases FILE] [FILE]"), helpcategory=command.CATEGORY_MAINTENANCE, inferrepo=True, ) def churn(ui, repo, *pats, **opts): """histogram of changes to the repository This command will display a histogram representing the number of changed lines or revisions, grouped according to the given template. The default template will group changes by author. The --dateformat option may be used to group the results by date instead. Statistics are based on the number of changed lines, or alternatively the number of matching revisions if the --changesets option is specified. Examples:: # display count of changed lines for every committer hg churn -T "{author|email}" # display daily activity graph hg churn -f "%H" -s -c # display activity of developers by month hg churn -f "%Y-%m" -s -c # display count of lines changed in every year hg churn -f "%Y" -s # display count of lines changed in a time range hg churn -d "2020-04 to 2020-09" It is possible to map alternate email addresses to a main address by providing a file using the following format:: <alias email> = <actual email> Such a file may be specified with the --aliases option, otherwise a .hgchurn file will be looked for in the working directory root. Aliases will be split from the rightmost "=". """ def pad(s, l): return s + b" " * (l - encoding.colwidth(s)) amap = {} aliases = opts.get('aliases') if not aliases and os.path.exists(repo.wjoin(b'.hgchurn')): aliases = repo.wjoin(b'.hgchurn') if aliases: for l in open(aliases, b"rb"): try: alias, actual = l.rsplit(b'=' in l and b'=' or None, 1) amap[alias.strip()] = actual.strip() except ValueError: l = l.strip() if l: ui.warn(_(b"skipping malformed alias: %s\n") % l) continue rate = list(countrate(ui, repo, amap, *pats, **opts).items()) if not rate: return if opts.get('sort'): rate.sort() else: rate.sort(key=lambda x: (-sum(x[1]), x)) # Be careful not to have a zero maxcount (issue833) maxcount = float(max(sum(v) for k, v in rate)) or 1.0 maxname = max(len(k) for k, v in rate) ttywidth = ui.termwidth() ui.debug(b"assuming %i character terminal\n" % ttywidth) width = ttywidth - maxname - 2 - 2 - 2 if opts.get('diffstat'): width -= 15 def format(name, diffstat): added, removed = diffstat return b"%s %15s %s%s\n" % ( pad(name, maxname), b'+%d/-%d' % (added, removed), ui.label(b'+' * charnum(added), b'diffstat.inserted'), ui.label(b'-' * charnum(removed), b'diffstat.deleted'), ) else: width -= 6 def format(name, count): return b"%s %6d %s\n" % ( pad(name, maxname), sum(count), b'*' * charnum(sum(count)), ) def charnum(count): return int(count * width // maxcount) for name, count in rate: ui.write(format(name, count))