Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 07 Nov 2022 14:24:52 -0500] rev 49775
debug-revlog: move the code in revlogutils module
We have a module dedicated to debug code, let us use it.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 07 Nov 2022 14:13:59 -0500] rev 49774
debug-revlog: move the --dump code in `revlogutils` module
We have a module dedicated to debug code, let us use it.
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 23 Nov 2022 19:08:27 +0100] rev 49773
delta-find: set the default candidate chunk size to 10
I ran performance and storage tests on repositories of various sizes and shapes
for the following values of the config : 5, 10, 20, 50, 100, no-chunking
The performance tests do not show any statistical impact on computation
times for large pushes and pulls.
For searching for an individual delta, this can provide a significant
performance improvement with a minor degradation of space-quality on the
result. (see data at the end of the commit).
For overall store size, the change :
- does not have any impact on many small repositories,
- has an observable, but very negligible impact on most larger repositories.
- One private repository we use for testing sees a small increase in size
(1%) in the narrower version.
We will try to get more numbers on a larger version of that repository to
make sure nothing pathological happens.
We pick "10" as the limit as "5" seems a bit more risky.
There are room to improve the current code, by using more aggressive filtering
and better (i.e any) sorting of the candidates. However this is already a large
improvement for pathological cases, with little impact in the common
situations.
The initial motivation for this change is to fix performance of delta
computation for a file where the previous code ended up testing 20 000 possible
candidate-bases in one go, which is… slow. This affected about ½ of the file
revisions leading to atrocious performance, especially during some push/pull
operations.
Details about individual delta finding timing:
----------------------------------------------
The vast majority of benchmark cases are unchanged but the three below. The first
two do not see any impact on the final delta. The last one sees a change in
delta-size that is negligible compared to the full text size.
### data-env-vars.name = mozilla-try-2019-02-18-zstd-sparse-revlog
# benchmark.name = perf-delta-find
# benchmark.variants.rev = manifest-snapshot-many-tries-a (revision 756096)
∞: 5.844783
5: 4.473523 (-23.46%)
10: 4.970053 (-14.97%)
20: 5.770386 (-1.27%)
50 5.821358
100: 5.834887
MANIFESTLOG: rev = 756096: (no-limit)
delta-base = 301840
search-rounds = 6
try-count = 60
delta-type = snapshot
snap-depth = 7
delta-size = 179
MANIFESTLOG: rev=756096: (limit = 10)
delta-base=301840
search-rounds=9
try-count=51
delta-type=snapshot
snap-depth=7
delta-size=179
### data-env-vars.name = mozilla-try-2019-02-18-zstd-sparse-revlog
# benchmark.name = perf-delta-find
# benchmark.variants.rev = manifest-snapshot-many-tries-d (revision 754060)
∞: 5.017663
5: 3.655931 (-27.14%)
10: 4.095436 (-18.38%)
20: 4.828949 (-3.76%)
50 4.987574
100: 4.994889
MANIFESTLOG: rev=754060: (no limit)
delta-base=301840
search-rounds=5
try-count=53
delta-type=snapshot
snap-depth=7
delta-size = 179
MANIFESTLOG: rev=754060: (limite = 10)
delta-base=301840
search-rounds=8
try-count=45
delta-type=snapshot
snap-depth=7
delta-size = 179
### data-env-vars.name = mozilla-try-2019-02-18-zstd-sparse-revlog
# benchmark.name = perf-delta-find
# bin-env-vars.hg.flavor = rust
# benchmark.variants.rev = manifest-snapshot-many-tries-e (revision 693368)
∞: 4.869282
5: 2.039732 (-58.11%)
10: 2.413537 (-50.43%)
20: 4.449639 (-8.62%)
50 4.865863
100: 4.882649
MANIFESTLOG: rev=693368:
delta-base=693336
search-rounds=6
try-count=53
delta-type=snapshot
snap-depth=6
full-test-size=131065
delta-size=199
MANIFESTLOG: rev=693368:
delta-base=278023
search-rounds=5
try-count=21
delta-type=snapshot
snap-depth=4
full-test-size=131065
delta-size=278
Raw data for store size (in bytes) for various chunk size value below:
----------------------------------------------------------------------
440 134 384 5 pypy/.hg/store/
440 134 384 10 pypy/.hg/store/
440 134 384 20 pypy/.hg/store/
440 134 384 50 pypy/.hg/store/
440 134 384 100 pypy/.hg/store/
440 134 384 ... pypy/.hg/store/
666 987 471 5 netbsd-xsrc-2022-11-15/.hg/store/
666 987 471 10 netbsd-xsrc-2022-11-15/.hg/store/
666 987 471 20 netbsd-xsrc-2022-11-15/.hg/store/
666 987 471 50 netbsd-xsrc-2022-11-15/.hg/store/
666 987 471 100 netbsd-xsrc-2022-11-15/.hg/store/
666 987 471 ... netbsd-xsrc-2022-11-15/.hg/store/
852 844 884 5 netbsd-pkgsrc-2022-11-15/.hg/store/
852 844 884 10 netbsd-pkgsrc-2022-11-15/.hg/store/
852 844 884 20 netbsd-pkgsrc-2022-11-15/.hg/store/
852 844 884 50 netbsd-pkgsrc-2022-11-15/.hg/store/
852 844 884 100 netbsd-pkgsrc-2022-11-15/.hg/store/
852 844 884 ... netbsd-pkgsrc-2022-11-15/.hg/store/
1 504 227 981 5 netbeans-2018-08-01-sparse-zstd/.hg/store/
1 504 227 871 10 netbeans-2018-08-01-sparse-zstd/.hg/store/
1 504 227 813 20 netbeans-2018-08-01-sparse-zstd/.hg/store/
1 504 227 813 50 netbeans-2018-08-01-sparse-zstd/.hg/store/
1 504 227 813 100 netbeans-2018-08-01-sparse-zstd/.hg/store/
1 504 227 813 ... netbeans-2018-08-01-sparse-zstd/.hg/store/
3 875 801 068 5 netbsd-src-2022-11-15/.hg/store/
3 875 696 767 10 netbsd-src-2022-11-15/.hg/store/
3 875 696 757 20 netbsd-src-2022-11-15/.hg/store/
3 875 696 653 50 netbsd-src-2022-11-15/.hg/store/
3 875 696 653 100 netbsd-src-2022-11-15/.hg/store/
3 875 696 653 ... netbsd-src-2022-11-15/.hg/store/
4 531 441 314 5 mozilla-central/.hg/store/
4 531 435 157 10 mozilla-central/.hg/store/
4 531 432 045 20 mozilla-central/.hg/store/
4 531 429 119 50 mozilla-central/.hg/store/
4 531 429 119 100 mozilla-central/.hg/store/
4 531 429 119 ... mozilla-central/.hg/store/
4 875 861 390 5 mozilla-unified/.hg/store/
4 875 855 155 10 mozilla-unified/.hg/store/
4 875 852 027 20 mozilla-unified/.hg/store/
4 875 848 851 50 mozilla-unified/.hg/store/
4 875 848 851 100 mozilla-unified/.hg/store/
4 875 848 851 ... mozilla-unified/.hg/store/
11 498 764 601 5 mozilla-try/.hg/store/
11 497 968 858 10 mozilla-try/.hg/store/
11 497 958 730 20 mozilla-try/.hg/store/
11 497 927 156 50 mozilla-try/.hg/store/
11 497 925 963 100 mozilla-try/.hg/store/
11 497 923 428 ... mozilla-try/.hg/store/
10 047 914 031 5 private-repo
9 969 132 101 10 private-repo
9 944 745 015 20 private-repo
9 939 756 703 50 private-repo
9 939 833 016 100 private-repo
9 939 822 035 ... private-repo
Pierre-Yves David <pierre-yves.david@octobus.net> [Sun, 06 Nov 2022 14:47:17 -0500] rev 49772
delta-find: add a way to control the number of bases tested at the same time
See inline comment for details.
The feature is currently disabled, but should be enabled by default to mitigate
some existing pathological cases.
Also see the next changeset for details.
Matt Harbison <matt_harbison@yahoo.com> [Wed, 23 Nov 2022 21:11:46 -0500] rev 49771
setup: include vendored 3rd party type stubs
While pytype may not support PEP 561, PyCharm does, so having the stubs
available means it can determine `foo = attr.ib(type=int)` means `foo` is an
int. This only applies when using Mercurial as a library, like with TortoiseHg
development- PyCharm is already smart enough to use the *.pyi files in the
Mercurial source tree when hacking on Mercurial itself.
I left the mercurial.cext stubs out because it seems very low level, that 3rd
parties shouldn't be using directly.
Matt Harbison <matt_harbison@yahoo.com> [Wed, 23 Nov 2022 20:59:53 -0500] rev 49770
ci: bump pytype to 2022.11.18
No particular reason, other than the current build is fairly old. It flagged a
few more things (that weren't errors based on the logic around them), but OTOH,
some of the pyi stubs it generates are less specific.
Matt Harbison <matt_harbison@yahoo.com> [Wed, 23 Nov 2022 20:56:22 -0500] rev 49769
ci: run the script to add vendored type stubs to typeshed
Since CI runs from docker images, it doesn't matter that this mucks with the
typeshed bundled with pytype.
Matt Harbison <matt_harbison@yahoo.com> [Wed, 23 Nov 2022 20:50:39 -0500] rev 49768
contrib: add a script for adding vendored type stubs to typeshed
I really hate this, but pytype doesn't support PEP 561 and doesn't seem to have
the equivalent of `MYPYPATH` to point to custom stubs. Ignoring the vendored
stubs isn't necessarily harmful, but pytype has been choking on the vendored
attr package after pytype 2022.03.29 with errors like this:
File "/mnt/c/Users/Matt/hg/mercurial/linelog.py", line 52, in __iter__: Built-in function iter was called with the wrong arguments [wrong-arg-types]
Expected: (collection: bytearray)
Actually passed: (collection: mercurial.thirdparty.attr._make._CountingAttr)
File "/mnt/c/Users/Matt/hg/mercurial/dirstateutils/v2.py", line 143, in pack: Built-in function len was called with the wrong arguments [wrong-arg-types]
Expected: (obj: Sized)
Actually passed: (obj: mercurial.thirdparty.attr._make._CountingAttr)
Attributes of protocol Sized are not implemented on mercurial.thirdparty.attr._make._CountingAttr: __len__
File "/mnt/c/Users/Matt/hg/mercurial/dirstateutils/v2.py", line 144, in pack: No attribute 'rfind' on mercurial.thirdparty.attr._make._CountingAttr [attribute-error]
File "/mnt/c/Users/Matt/hg/mercurial/dirstateutils/v2.py", line 146, in pack: Built-in function len was called with the wrong arguments [wrong-arg-types]
Expected: (obj: Sized)
Actually passed: (obj: mercurial.thirdparty.attr._make._CountingAttr)
Attributes of protocol Sized are not implemented on mercurial.thirdparty.attr._make._CountingAttr: __len__
File "/mnt/c/Users/Matt/hg/mercurial/dirstateutils/v2.py", line 152, in pack: No attribute 'v2_data' on mercurial.thirdparty.attr._make._CountingAttr [attribute-error]
File "/mnt/c/Users/Matt/hg/mercurial/util.py", line 2817, in go: unsupported operand type(s) for /: 'count: mercurial.thirdparty.attr._make._CountingAttr' and 'float: float' [unsupported-operands]
No attribute '__truediv__' on 'count: mercurial.thirdparty.attr._make._CountingAttr' or '__rtruediv__' on 'float: float'
Called from (traceback):
line 2981, in __bytes__
This is essentially the same hack we've been using in TortoiseHg to add the
vendored PyQt5 stubs. What I don't understand is pytype *still* generates *.pyi
files under .pytype/pyi/mercurial/thirdparty/attr, even when the package is
explicitly ignored in the pytype command line args. But it avoids the errors,
which means we aren't stuck on pytype==2022.03.29.
https://github.com/google/pytype/issues/151
Matt Harbison <matt_harbison@yahoo.com> [Wed, 23 Nov 2022 20:23:26 -0500] rev 49767
contrib: update check-pytype.sh to list stubs that caused pytype to crash
The same logic is in the TortoiseHg tests for running pytype, and it's useful to
know if a new version of pytype is better or worse.
Matt Harbison <matt_harbison@yahoo.com> [Wed, 23 Nov 2022 16:11:20 -0500] rev 49766
typing: add py.typed to mercurial.cext for PEP 561 support
Unfortunately, pytype doesn't support this yet. But it was included with the
attr package, so we might as well do it here for consistency. Unlike the attr
package, these type hints are only partial, so they are marked as such[1] (but
who knows if it matters, given these are C extensions, so no local source code
to scan).
[1] https://peps.python.org/pep-0561/#partial-stub-packages