Mercurial > hg
annotate hgext/automv.py @ 30817:2b279126b8f5
revlog: use compression engine APIs for decompression
Now that compression engines declare their header in revlog chunks
and can decompress revlog chunks, we refactor revlog.decompress()
to use them.
Making full use of the property that revlog compressor objects are
reusable, revlog instances now maintain a dict mapping an engine's
revlog header to a compressor object. This is not only a performance
optimization for engines where compressor object reuse can result in
better performance, but it also serves as a cache of header values
so we don't need to perform redundant lookups against the compression
engine manager. (Yes, I measured and the overhead of a function call
versus a dict lookup was observed.)
Replacing the previous inline lookup table with a dict lookup was
measured to make chunk reading ~2.5% slower on changelogs and ~4.5%
slower on manifests. So, the inline lookup table has been mostly
preserved so we don't lose performance. This is unfortunate. But
many decompression operations complete in microseconds, so Python
attribute lookup, dict lookup, and function calls do matter.
The impact of this change on mozilla-unified is as follows:
$ hg perfrevlogchunks -c
! chunk
! wall 1.953663 comb 1.950000 user 1.920000 sys 0.030000 (best of 6)
! wall 1.946000 comb 1.940000 user 1.910000 sys 0.030000 (best of 6)
! chunk batch
! wall 1.791075 comb 1.800000 user 1.760000 sys 0.040000 (best of 6)
! wall 1.785690 comb 1.770000 user 1.750000 sys 0.020000 (best of 6)
$ hg perfrevlogchunks -m
! chunk
! wall 2.587262 comb 2.580000 user 2.550000 sys 0.030000 (best of 4)
! wall 2.616330 comb 2.610000 user 2.560000 sys 0.050000 (best of 4)
! chunk batch
! wall 2.427092 comb 2.420000 user 2.400000 sys 0.020000 (best of 5)
! wall 2.462061 comb 2.460000 user 2.400000 sys 0.060000 (best of 4)
Changelog chunk reading is slightly faster but manifest reading is
slower. What gives?
On this repo, 99.85% of changelog entries are zlib compressed (the 'x'
header). On the manifest, 67.5% are zlib and 32.4% are '\0'. This patch
swapped the test order of 'x' and '\0' so now 'x' is tested first. This
makes changelogs faster since they almost always hit the first branch.
This makes a significant percentage of manifest '\0' chunks slower
because that code path now performs an extra test. Yes, I too can't
believe we're able to measure the impact of an if..elif with simple
string compares. I reckon this code would benefit from being written
in C...
author | Gregory Szorc <gregory.szorc@gmail.com> |
---|---|
date | Fri, 13 Jan 2017 19:58:00 -0800 |
parents | a0939666b836 |
children | e4aefdb58ebe |
rev | line source |
---|---|
28129
7c40b4b7f8f1
automv: new experimental extension
Martijn Pieters <mjpieters@fb.com>
parents:
diff
changeset
|
1 # automv.py |
7c40b4b7f8f1
automv: new experimental extension
Martijn Pieters <mjpieters@fb.com>
parents:
diff
changeset
|
2 # |
7c40b4b7f8f1
automv: new experimental extension
Martijn Pieters <mjpieters@fb.com>
parents:
diff
changeset
|
3 # Copyright 2013-2016 Facebook, Inc. |
7c40b4b7f8f1
automv: new experimental extension
Martijn Pieters <mjpieters@fb.com>
parents:
diff
changeset
|
4 # |
7c40b4b7f8f1
automv: new experimental extension
Martijn Pieters <mjpieters@fb.com>
parents:
diff
changeset
|
5 # This software may be used and distributed according to the terms of the |
7c40b4b7f8f1
automv: new experimental extension
Martijn Pieters <mjpieters@fb.com>
parents:
diff
changeset
|
6 # GNU General Public License version 2 or any later version. |
7c40b4b7f8f1
automv: new experimental extension
Martijn Pieters <mjpieters@fb.com>
parents:
diff
changeset
|
7 """Check for unrecorded moves at commit time (EXPERIMENTAL) |
7c40b4b7f8f1
automv: new experimental extension
Martijn Pieters <mjpieters@fb.com>
parents:
diff
changeset
|
8 |
7c40b4b7f8f1
automv: new experimental extension
Martijn Pieters <mjpieters@fb.com>
parents:
diff
changeset
|
9 This extension checks at commit/amend time if any of the committed files |
7c40b4b7f8f1
automv: new experimental extension
Martijn Pieters <mjpieters@fb.com>
parents:
diff
changeset
|
10 comes from an unrecorded mv. |
7c40b4b7f8f1
automv: new experimental extension
Martijn Pieters <mjpieters@fb.com>
parents:
diff
changeset
|
11 |
7c40b4b7f8f1
automv: new experimental extension
Martijn Pieters <mjpieters@fb.com>
parents:
diff
changeset
|
12 The threshold at which a file is considered a move can be set with the |
28152
5ec1ce8fdf0a
automv: switch to specifying the similarity as an integer (0-100)
Martijn Pieters <mjpieters@fb.com>
parents:
28151
diff
changeset
|
13 ``automv.similarity`` config option. This option takes a percentage between 0 |
28183
e07daee83029
automv: use 95 as the default similarity threshold
Martijn Pieters <mjpieters@fb.com>
parents:
28152
diff
changeset
|
14 (disabled) and 100 (files must be identical), the default is 95. |
28129
7c40b4b7f8f1
automv: new experimental extension
Martijn Pieters <mjpieters@fb.com>
parents:
diff
changeset
|
15 |
7c40b4b7f8f1
automv: new experimental extension
Martijn Pieters <mjpieters@fb.com>
parents:
diff
changeset
|
16 """ |
28183
e07daee83029
automv: use 95 as the default similarity threshold
Martijn Pieters <mjpieters@fb.com>
parents:
28152
diff
changeset
|
17 |
e07daee83029
automv: use 95 as the default similarity threshold
Martijn Pieters <mjpieters@fb.com>
parents:
28152
diff
changeset
|
18 # Using 95 as a default similarity is based on an analysis of the mercurial |
e07daee83029
automv: use 95 as the default similarity threshold
Martijn Pieters <mjpieters@fb.com>
parents:
28152
diff
changeset
|
19 # repositories of the cpython, mozilla-central & mercurial repositories, as |
e07daee83029
automv: use 95 as the default similarity threshold
Martijn Pieters <mjpieters@fb.com>
parents:
28152
diff
changeset
|
20 # well as 2 very large facebook repositories. At 95 50% of all potential |
e07daee83029
automv: use 95 as the default similarity threshold
Martijn Pieters <mjpieters@fb.com>
parents:
28152
diff
changeset
|
21 # missed moves would be caught, as well as correspond with 87% of all |
e07daee83029
automv: use 95 as the default similarity threshold
Martijn Pieters <mjpieters@fb.com>
parents:
28152
diff
changeset
|
22 # explicitly marked moves. Together, 80% of moved files are 95% similar or |
e07daee83029
automv: use 95 as the default similarity threshold
Martijn Pieters <mjpieters@fb.com>
parents:
28152
diff
changeset
|
23 # more. |
e07daee83029
automv: use 95 as the default similarity threshold
Martijn Pieters <mjpieters@fb.com>
parents:
28152
diff
changeset
|
24 # |
e07daee83029
automv: use 95 as the default similarity threshold
Martijn Pieters <mjpieters@fb.com>
parents:
28152
diff
changeset
|
25 # See http://markmail.org/thread/5pxnljesvufvom57 for context. |
e07daee83029
automv: use 95 as the default similarity threshold
Martijn Pieters <mjpieters@fb.com>
parents:
28152
diff
changeset
|
26 |
28129
7c40b4b7f8f1
automv: new experimental extension
Martijn Pieters <mjpieters@fb.com>
parents:
diff
changeset
|
27 from __future__ import absolute_import |
7c40b4b7f8f1
automv: new experimental extension
Martijn Pieters <mjpieters@fb.com>
parents:
diff
changeset
|
28 |
29205
a0939666b836
py3: move up symbol imports to enforce import-checker rules
Yuya Nishihara <yuya@tcha.org>
parents:
28183
diff
changeset
|
29 from mercurial.i18n import _ |
28129
7c40b4b7f8f1
automv: new experimental extension
Martijn Pieters <mjpieters@fb.com>
parents:
diff
changeset
|
30 from mercurial import ( |
7c40b4b7f8f1
automv: new experimental extension
Martijn Pieters <mjpieters@fb.com>
parents:
diff
changeset
|
31 commands, |
7c40b4b7f8f1
automv: new experimental extension
Martijn Pieters <mjpieters@fb.com>
parents:
diff
changeset
|
32 copies, |
28183
e07daee83029
automv: use 95 as the default similarity threshold
Martijn Pieters <mjpieters@fb.com>
parents:
28152
diff
changeset
|
33 error, |
28129
7c40b4b7f8f1
automv: new experimental extension
Martijn Pieters <mjpieters@fb.com>
parents:
diff
changeset
|
34 extensions, |
7c40b4b7f8f1
automv: new experimental extension
Martijn Pieters <mjpieters@fb.com>
parents:
diff
changeset
|
35 scmutil, |
7c40b4b7f8f1
automv: new experimental extension
Martijn Pieters <mjpieters@fb.com>
parents:
diff
changeset
|
36 similar |
7c40b4b7f8f1
automv: new experimental extension
Martijn Pieters <mjpieters@fb.com>
parents:
diff
changeset
|
37 ) |
7c40b4b7f8f1
automv: new experimental extension
Martijn Pieters <mjpieters@fb.com>
parents:
diff
changeset
|
38 |
7c40b4b7f8f1
automv: new experimental extension
Martijn Pieters <mjpieters@fb.com>
parents:
diff
changeset
|
39 def extsetup(ui): |
7c40b4b7f8f1
automv: new experimental extension
Martijn Pieters <mjpieters@fb.com>
parents:
diff
changeset
|
40 entry = extensions.wrapcommand( |
7c40b4b7f8f1
automv: new experimental extension
Martijn Pieters <mjpieters@fb.com>
parents:
diff
changeset
|
41 commands.table, 'commit', mvcheck) |
7c40b4b7f8f1
automv: new experimental extension
Martijn Pieters <mjpieters@fb.com>
parents:
diff
changeset
|
42 entry[1].append( |
7c40b4b7f8f1
automv: new experimental extension
Martijn Pieters <mjpieters@fb.com>
parents:
diff
changeset
|
43 ('', 'no-automv', None, |
7c40b4b7f8f1
automv: new experimental extension
Martijn Pieters <mjpieters@fb.com>
parents:
diff
changeset
|
44 _('disable automatic file move detection'))) |
7c40b4b7f8f1
automv: new experimental extension
Martijn Pieters <mjpieters@fb.com>
parents:
diff
changeset
|
45 |
7c40b4b7f8f1
automv: new experimental extension
Martijn Pieters <mjpieters@fb.com>
parents:
diff
changeset
|
46 def mvcheck(orig, ui, repo, *pats, **opts): |
28149
d356d5250ab2
automv: improve function docstrings
Martijn Pieters <mjpieters@fb.com>
parents:
28148
diff
changeset
|
47 """Hook to check for moves at commit time""" |
28151
74e3d634a30e
automv: do not release lock between marking files and the actual commit
Martijn Pieters <mjpieters@fb.com>
parents:
28150
diff
changeset
|
48 renames = None |
28129
7c40b4b7f8f1
automv: new experimental extension
Martijn Pieters <mjpieters@fb.com>
parents:
diff
changeset
|
49 disabled = opts.pop('no_automv', False) |
7c40b4b7f8f1
automv: new experimental extension
Martijn Pieters <mjpieters@fb.com>
parents:
diff
changeset
|
50 if not disabled: |
28183
e07daee83029
automv: use 95 as the default similarity threshold
Martijn Pieters <mjpieters@fb.com>
parents:
28152
diff
changeset
|
51 threshold = ui.configint('automv', 'similarity', 95) |
e07daee83029
automv: use 95 as the default similarity threshold
Martijn Pieters <mjpieters@fb.com>
parents:
28152
diff
changeset
|
52 if not 0 <= threshold <= 100: |
e07daee83029
automv: use 95 as the default similarity threshold
Martijn Pieters <mjpieters@fb.com>
parents:
28152
diff
changeset
|
53 raise error.Abort(_('automv.similarity must be between 0 and 100')) |
28129
7c40b4b7f8f1
automv: new experimental extension
Martijn Pieters <mjpieters@fb.com>
parents:
diff
changeset
|
54 if threshold > 0: |
7c40b4b7f8f1
automv: new experimental extension
Martijn Pieters <mjpieters@fb.com>
parents:
diff
changeset
|
55 match = scmutil.match(repo[None], pats, opts) |
7c40b4b7f8f1
automv: new experimental extension
Martijn Pieters <mjpieters@fb.com>
parents:
diff
changeset
|
56 added, removed = _interestingfiles(repo, match) |
28152
5ec1ce8fdf0a
automv: switch to specifying the similarity as an integer (0-100)
Martijn Pieters <mjpieters@fb.com>
parents:
28151
diff
changeset
|
57 renames = _findrenames(repo, match, added, removed, |
5ec1ce8fdf0a
automv: switch to specifying the similarity as an integer (0-100)
Martijn Pieters <mjpieters@fb.com>
parents:
28151
diff
changeset
|
58 threshold / 100.0) |
28151
74e3d634a30e
automv: do not release lock between marking files and the actual commit
Martijn Pieters <mjpieters@fb.com>
parents:
28150
diff
changeset
|
59 |
74e3d634a30e
automv: do not release lock between marking files and the actual commit
Martijn Pieters <mjpieters@fb.com>
parents:
28150
diff
changeset
|
60 with repo.wlock(): |
74e3d634a30e
automv: do not release lock between marking files and the actual commit
Martijn Pieters <mjpieters@fb.com>
parents:
28150
diff
changeset
|
61 if renames is not None: |
28150
7a984cece04a
automv: reuse existing scutil._markchanges() function
Martijn Pieters <mjpieters@fb.com>
parents:
28149
diff
changeset
|
62 scmutil._markchanges(repo, (), (), renames) |
28151
74e3d634a30e
automv: do not release lock between marking files and the actual commit
Martijn Pieters <mjpieters@fb.com>
parents:
28150
diff
changeset
|
63 return orig(ui, repo, *pats, **opts) |
28129
7c40b4b7f8f1
automv: new experimental extension
Martijn Pieters <mjpieters@fb.com>
parents:
diff
changeset
|
64 |
7c40b4b7f8f1
automv: new experimental extension
Martijn Pieters <mjpieters@fb.com>
parents:
diff
changeset
|
65 def _interestingfiles(repo, matcher): |
28149
d356d5250ab2
automv: improve function docstrings
Martijn Pieters <mjpieters@fb.com>
parents:
28148
diff
changeset
|
66 """Find what files were added or removed in this commit. |
d356d5250ab2
automv: improve function docstrings
Martijn Pieters <mjpieters@fb.com>
parents:
28148
diff
changeset
|
67 |
d356d5250ab2
automv: improve function docstrings
Martijn Pieters <mjpieters@fb.com>
parents:
28148
diff
changeset
|
68 Returns a tuple of two lists: (added, removed). Only files not *already* |
d356d5250ab2
automv: improve function docstrings
Martijn Pieters <mjpieters@fb.com>
parents:
28148
diff
changeset
|
69 marked as moved are included in the added list. |
d356d5250ab2
automv: improve function docstrings
Martijn Pieters <mjpieters@fb.com>
parents:
28148
diff
changeset
|
70 |
d356d5250ab2
automv: improve function docstrings
Martijn Pieters <mjpieters@fb.com>
parents:
28148
diff
changeset
|
71 """ |
28146
28024d0d42dc
automv: simplify retrieving the status
Martijn Pieters <mjpieters@fb.com>
parents:
28129
diff
changeset
|
72 stat = repo.status(match=matcher) |
28129
7c40b4b7f8f1
automv: new experimental extension
Martijn Pieters <mjpieters@fb.com>
parents:
diff
changeset
|
73 added = stat[1] |
7c40b4b7f8f1
automv: new experimental extension
Martijn Pieters <mjpieters@fb.com>
parents:
diff
changeset
|
74 removed = stat[2] |
7c40b4b7f8f1
automv: new experimental extension
Martijn Pieters <mjpieters@fb.com>
parents:
diff
changeset
|
75 |
7c40b4b7f8f1
automv: new experimental extension
Martijn Pieters <mjpieters@fb.com>
parents:
diff
changeset
|
76 copy = copies._forwardcopies(repo['.'], repo[None], matcher) |
7c40b4b7f8f1
automv: new experimental extension
Martijn Pieters <mjpieters@fb.com>
parents:
diff
changeset
|
77 # remove the copy files for which we already have copy info |
7c40b4b7f8f1
automv: new experimental extension
Martijn Pieters <mjpieters@fb.com>
parents:
diff
changeset
|
78 added = [f for f in added if f not in copy] |
7c40b4b7f8f1
automv: new experimental extension
Martijn Pieters <mjpieters@fb.com>
parents:
diff
changeset
|
79 |
7c40b4b7f8f1
automv: new experimental extension
Martijn Pieters <mjpieters@fb.com>
parents:
diff
changeset
|
80 return added, removed |
7c40b4b7f8f1
automv: new experimental extension
Martijn Pieters <mjpieters@fb.com>
parents:
diff
changeset
|
81 |
7c40b4b7f8f1
automv: new experimental extension
Martijn Pieters <mjpieters@fb.com>
parents:
diff
changeset
|
82 def _findrenames(repo, matcher, added, removed, similarity): |
28149
d356d5250ab2
automv: improve function docstrings
Martijn Pieters <mjpieters@fb.com>
parents:
28148
diff
changeset
|
83 """Find what files in added are really moved files. |
d356d5250ab2
automv: improve function docstrings
Martijn Pieters <mjpieters@fb.com>
parents:
28148
diff
changeset
|
84 |
d356d5250ab2
automv: improve function docstrings
Martijn Pieters <mjpieters@fb.com>
parents:
28148
diff
changeset
|
85 Any file named in removed that is at least similarity% similar to a file |
d356d5250ab2
automv: improve function docstrings
Martijn Pieters <mjpieters@fb.com>
parents:
28148
diff
changeset
|
86 in added is seen as a rename. |
d356d5250ab2
automv: improve function docstrings
Martijn Pieters <mjpieters@fb.com>
parents:
28148
diff
changeset
|
87 |
d356d5250ab2
automv: improve function docstrings
Martijn Pieters <mjpieters@fb.com>
parents:
28148
diff
changeset
|
88 """ |
28129
7c40b4b7f8f1
automv: new experimental extension
Martijn Pieters <mjpieters@fb.com>
parents:
diff
changeset
|
89 renames = {} |
7c40b4b7f8f1
automv: new experimental extension
Martijn Pieters <mjpieters@fb.com>
parents:
diff
changeset
|
90 if similarity > 0: |
7c40b4b7f8f1
automv: new experimental extension
Martijn Pieters <mjpieters@fb.com>
parents:
diff
changeset
|
91 for src, dst, score in similar.findrenames( |
7c40b4b7f8f1
automv: new experimental extension
Martijn Pieters <mjpieters@fb.com>
parents:
diff
changeset
|
92 repo, added, removed, similarity): |
7c40b4b7f8f1
automv: new experimental extension
Martijn Pieters <mjpieters@fb.com>
parents:
diff
changeset
|
93 if repo.ui.verbose: |
7c40b4b7f8f1
automv: new experimental extension
Martijn Pieters <mjpieters@fb.com>
parents:
diff
changeset
|
94 repo.ui.status( |
7c40b4b7f8f1
automv: new experimental extension
Martijn Pieters <mjpieters@fb.com>
parents:
diff
changeset
|
95 _('detected move of %s as %s (%d%% similar)\n') % ( |
7c40b4b7f8f1
automv: new experimental extension
Martijn Pieters <mjpieters@fb.com>
parents:
diff
changeset
|
96 matcher.rel(src), matcher.rel(dst), score * 100)) |
7c40b4b7f8f1
automv: new experimental extension
Martijn Pieters <mjpieters@fb.com>
parents:
diff
changeset
|
97 renames[dst] = src |
7c40b4b7f8f1
automv: new experimental extension
Martijn Pieters <mjpieters@fb.com>
parents:
diff
changeset
|
98 if renames: |
7c40b4b7f8f1
automv: new experimental extension
Martijn Pieters <mjpieters@fb.com>
parents:
diff
changeset
|
99 repo.ui.status(_('detected move of %d files\n') % len(renames)) |
7c40b4b7f8f1
automv: new experimental extension
Martijn Pieters <mjpieters@fb.com>
parents:
diff
changeset
|
100 return renames |