Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 07 Nov 2022 20:02:32 -0500] rev 49789
find-delta: minor preparatory change
We are about to add more item in the cachedelta object, so lets access the item by index instead of doing a full extensions.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 07 Nov 2022 17:57:28 -0500] rev 49788
find-delta: rename _isgooddeltainfo
Lets move to a more readable name now that we are allowed to. This cannot hurt.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 07 Nov 2022 18:06:17 -0500] rev 49787
test-revlog-raw: drop the overwrite of dead code
The revlog class no longer have a _isgooddeltainfo method for a long time. So
overwriting it does not get us anything. The test have been wrapping the right
code since then anyway.
Pierre-Yves David <pierre-yves.david@octobus.net> [Sat, 12 Nov 2022 00:18:41 +0100] rev 49786
emitrevision: consider ancestors revision to emit as available base
This should make more delta base valid. This notably affects:
* case where we skipped some parent with empty delta to directly delta against
an ancestors
* case where an intermediate snapshots is stored.
This change means we could sent largish intermediate snapshots over the wire.
However this is actually a sub goal here. Sending snapshots over the wire means
the client have a high odd of simply storing the pre-computed delta instead of
doing a lengthy process that will… end up doing the same intermediate snapshot.
In addition the overall size of snapshot (or any level) is "only" some or the
overall delta size. (0.17% for my mercurial clone, 20% for my clone of Mozilla
try). So Sending them other the wire is unlikely to change large impact on the
bandwidth used.
If we decide that minimising the bandwidth is an explicit goal, we should
introduce new logic to filter-out snapshot as delta. The current code has no
notion explicite of snapshot so far, they just tended to fall into the wobbly
filtering options.
In some cases, this patch can yield large improvement to the bundling time:
### data-env-vars.name = mozilla-try-2019-02-18-zstd-sparse-revlog
# benchmark.name = perf-bundle
# benchmark.variants.revs = last-100000
before: 68.787066 seconds
after: 47.552677 seconds (-30.87%)
That translate to large improvement to the pull time :
### data-env-vars.name = mozilla-try-2019-02-18-zstd-sparse-revlog
# benchmark.name = pull
# benchmark.variants.issue6528 = disabled
# benchmark.variants.revs = last-100000
before: 142.186625 seconds
after: 75.897745 seconds (-46.62%)
No significant negative impact have been observed.
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 09 Nov 2022 13:54:15 -0500] rev 49785
sqlitestore: add an `ancestors` method
We will need it during bundling.
The implementation mirror the one in revlog.
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 24 Nov 2022 04:04:19 +0100] rev 49784
emitrevision: if we need to compute a delta on the fly, try p1 or p2 first
Falling back to `prev` does not yield any real value on modern storage and
result in pathological changes to be created on the other side. Doing a delta
against a parent will likely be smaller (helping the network) and will be safer
to apply on the client (helping future pulls by Triggering intermediate
snapshop where they will be needed by later deltas).
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 28 Nov 2022 16:27:23 +0100] rev 49783
emitrevision: simplify the fallback to computed delta
Not using the stored delta, or having a full snapshot on disk behave the same
ways, so lets use the same code path for that, this is simpler, and it update
will be simpler.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 28 Nov 2022 15:59:52 +0100] rev 49782
emitrevision: also check the parents in the availability closure
One of the point of having a closure is to gather the logic in it. So we gather
the logic.
The `parents[:]` part is a bit ugly but will be replaced by better code soon
anyway.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 28 Nov 2022 15:48:51 +0100] rev 49781
emitrevision: add a small closure to check if a base is usable
We will make more use of this and make it more complex too.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 17 Oct 2022 16:26:16 +0200] rev 49780
reuse-delta-base: improves some documentation
The current code got me a bit confused initially. So a bit more documentation
around it cannot hurt.
Raphaël Gomès <rgomes@octobus.net> [Fri, 25 Nov 2022 15:14:40 +0100] rev 49779
branching: merge stable into default
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 22 Nov 2022 12:44:22 +0100] rev 49778
changelog-v2: add a configuration to disable rank computation
We encountered a graph where rank computation was pathologically slow. We add an
option to disable this computation while this is getting fixed.
Disabling the rank computation should allow for testing other changelog-v2
features undisturbed (like changeset-based copy tracing).
I am purposely not adding a test for the new non-default code path, as this is
a temporary work around of an experimental feature.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 21 Nov 2022 15:04:19 +0100] rev 49777
debugrevlog: display total stored information
This is an interesting statistis, so let's display it.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 07 Nov 2022 14:38:52 -0500] rev 49776
debug-revlog: details about non-ancestors delta-bases
Deltas against a base that is not an ancestor of the revision that owns this
delta are notable.
For example, they introduce complexity during the bundling process as the base
might not exist on the unbundling side.
We detect them in `hg debugrevlog` and print information about them.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 07 Nov 2022 14:24:52 -0500] rev 49775
debug-revlog: move the code in revlogutils module
We have a module dedicated to debug code, let us use it.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 07 Nov 2022 14:13:59 -0500] rev 49774
debug-revlog: move the --dump code in `revlogutils` module
We have a module dedicated to debug code, let us use it.
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 23 Nov 2022 19:08:27 +0100] rev 49773
delta-find: set the default candidate chunk size to 10
I ran performance and storage tests on repositories of various sizes and shapes
for the following values of the config : 5, 10, 20, 50, 100, no-chunking
The performance tests do not show any statistical impact on computation
times for large pushes and pulls.
For searching for an individual delta, this can provide a significant
performance improvement with a minor degradation of space-quality on the
result. (see data at the end of the commit).
For overall store size, the change :
- does not have any impact on many small repositories,
- has an observable, but very negligible impact on most larger repositories.
- One private repository we use for testing sees a small increase in size
(1%) in the narrower version.
We will try to get more numbers on a larger version of that repository to
make sure nothing pathological happens.
We pick "10" as the limit as "5" seems a bit more risky.
There are room to improve the current code, by using more aggressive filtering
and better (i.e any) sorting of the candidates. However this is already a large
improvement for pathological cases, with little impact in the common
situations.
The initial motivation for this change is to fix performance of delta
computation for a file where the previous code ended up testing 20 000 possible
candidate-bases in one go, which is… slow. This affected about ½ of the file
revisions leading to atrocious performance, especially during some push/pull
operations.
Details about individual delta finding timing:
----------------------------------------------
The vast majority of benchmark cases are unchanged but the three below. The first
two do not see any impact on the final delta. The last one sees a change in
delta-size that is negligible compared to the full text size.
### data-env-vars.name = mozilla-try-2019-02-18-zstd-sparse-revlog
# benchmark.name = perf-delta-find
# benchmark.variants.rev = manifest-snapshot-many-tries-a (revision 756096)
∞: 5.844783
5: 4.473523 (-23.46%)
10: 4.970053 (-14.97%)
20: 5.770386 (-1.27%)
50 5.821358
100: 5.834887
MANIFESTLOG: rev = 756096: (no-limit)
delta-base = 301840
search-rounds = 6
try-count = 60
delta-type = snapshot
snap-depth = 7
delta-size = 179
MANIFESTLOG: rev=756096: (limit = 10)
delta-base=301840
search-rounds=9
try-count=51
delta-type=snapshot
snap-depth=7
delta-size=179
### data-env-vars.name = mozilla-try-2019-02-18-zstd-sparse-revlog
# benchmark.name = perf-delta-find
# benchmark.variants.rev = manifest-snapshot-many-tries-d (revision 754060)
∞: 5.017663
5: 3.655931 (-27.14%)
10: 4.095436 (-18.38%)
20: 4.828949 (-3.76%)
50 4.987574
100: 4.994889
MANIFESTLOG: rev=754060: (no limit)
delta-base=301840
search-rounds=5
try-count=53
delta-type=snapshot
snap-depth=7
delta-size = 179
MANIFESTLOG: rev=754060: (limite = 10)
delta-base=301840
search-rounds=8
try-count=45
delta-type=snapshot
snap-depth=7
delta-size = 179
### data-env-vars.name = mozilla-try-2019-02-18-zstd-sparse-revlog
# benchmark.name = perf-delta-find
# bin-env-vars.hg.flavor = rust
# benchmark.variants.rev = manifest-snapshot-many-tries-e (revision 693368)
∞: 4.869282
5: 2.039732 (-58.11%)
10: 2.413537 (-50.43%)
20: 4.449639 (-8.62%)
50 4.865863
100: 4.882649
MANIFESTLOG: rev=693368:
delta-base=693336
search-rounds=6
try-count=53
delta-type=snapshot
snap-depth=6
full-test-size=131065
delta-size=199
MANIFESTLOG: rev=693368:
delta-base=278023
search-rounds=5
try-count=21
delta-type=snapshot
snap-depth=4
full-test-size=131065
delta-size=278
Raw data for store size (in bytes) for various chunk size value below:
----------------------------------------------------------------------
440 134 384 5 pypy/.hg/store/
440 134 384 10 pypy/.hg/store/
440 134 384 20 pypy/.hg/store/
440 134 384 50 pypy/.hg/store/
440 134 384 100 pypy/.hg/store/
440 134 384 ... pypy/.hg/store/
666 987 471 5 netbsd-xsrc-2022-11-15/.hg/store/
666 987 471 10 netbsd-xsrc-2022-11-15/.hg/store/
666 987 471 20 netbsd-xsrc-2022-11-15/.hg/store/
666 987 471 50 netbsd-xsrc-2022-11-15/.hg/store/
666 987 471 100 netbsd-xsrc-2022-11-15/.hg/store/
666 987 471 ... netbsd-xsrc-2022-11-15/.hg/store/
852 844 884 5 netbsd-pkgsrc-2022-11-15/.hg/store/
852 844 884 10 netbsd-pkgsrc-2022-11-15/.hg/store/
852 844 884 20 netbsd-pkgsrc-2022-11-15/.hg/store/
852 844 884 50 netbsd-pkgsrc-2022-11-15/.hg/store/
852 844 884 100 netbsd-pkgsrc-2022-11-15/.hg/store/
852 844 884 ... netbsd-pkgsrc-2022-11-15/.hg/store/
1 504 227 981 5 netbeans-2018-08-01-sparse-zstd/.hg/store/
1 504 227 871 10 netbeans-2018-08-01-sparse-zstd/.hg/store/
1 504 227 813 20 netbeans-2018-08-01-sparse-zstd/.hg/store/
1 504 227 813 50 netbeans-2018-08-01-sparse-zstd/.hg/store/
1 504 227 813 100 netbeans-2018-08-01-sparse-zstd/.hg/store/
1 504 227 813 ... netbeans-2018-08-01-sparse-zstd/.hg/store/
3 875 801 068 5 netbsd-src-2022-11-15/.hg/store/
3 875 696 767 10 netbsd-src-2022-11-15/.hg/store/
3 875 696 757 20 netbsd-src-2022-11-15/.hg/store/
3 875 696 653 50 netbsd-src-2022-11-15/.hg/store/
3 875 696 653 100 netbsd-src-2022-11-15/.hg/store/
3 875 696 653 ... netbsd-src-2022-11-15/.hg/store/
4 531 441 314 5 mozilla-central/.hg/store/
4 531 435 157 10 mozilla-central/.hg/store/
4 531 432 045 20 mozilla-central/.hg/store/
4 531 429 119 50 mozilla-central/.hg/store/
4 531 429 119 100 mozilla-central/.hg/store/
4 531 429 119 ... mozilla-central/.hg/store/
4 875 861 390 5 mozilla-unified/.hg/store/
4 875 855 155 10 mozilla-unified/.hg/store/
4 875 852 027 20 mozilla-unified/.hg/store/
4 875 848 851 50 mozilla-unified/.hg/store/
4 875 848 851 100 mozilla-unified/.hg/store/
4 875 848 851 ... mozilla-unified/.hg/store/
11 498 764 601 5 mozilla-try/.hg/store/
11 497 968 858 10 mozilla-try/.hg/store/
11 497 958 730 20 mozilla-try/.hg/store/
11 497 927 156 50 mozilla-try/.hg/store/
11 497 925 963 100 mozilla-try/.hg/store/
11 497 923 428 ... mozilla-try/.hg/store/
10 047 914 031 5 private-repo
9 969 132 101 10 private-repo
9 944 745 015 20 private-repo
9 939 756 703 50 private-repo
9 939 833 016 100 private-repo
9 939 822 035 ... private-repo
Pierre-Yves David <pierre-yves.david@octobus.net> [Sun, 06 Nov 2022 14:47:17 -0500] rev 49772
delta-find: add a way to control the number of bases tested at the same time
See inline comment for details.
The feature is currently disabled, but should be enabled by default to mitigate
some existing pathological cases.
Also see the next changeset for details.
Matt Harbison <matt_harbison@yahoo.com> [Wed, 23 Nov 2022 21:11:46 -0500] rev 49771
setup: include vendored 3rd party type stubs
While pytype may not support PEP 561, PyCharm does, so having the stubs
available means it can determine `foo = attr.ib(type=int)` means `foo` is an
int. This only applies when using Mercurial as a library, like with TortoiseHg
development- PyCharm is already smart enough to use the *.pyi files in the
Mercurial source tree when hacking on Mercurial itself.
I left the mercurial.cext stubs out because it seems very low level, that 3rd
parties shouldn't be using directly.
Matt Harbison <matt_harbison@yahoo.com> [Wed, 23 Nov 2022 20:59:53 -0500] rev 49770
ci: bump pytype to 2022.11.18
No particular reason, other than the current build is fairly old. It flagged a
few more things (that weren't errors based on the logic around them), but OTOH,
some of the pyi stubs it generates are less specific.
Matt Harbison <matt_harbison@yahoo.com> [Wed, 23 Nov 2022 20:56:22 -0500] rev 49769
ci: run the script to add vendored type stubs to typeshed
Since CI runs from docker images, it doesn't matter that this mucks with the
typeshed bundled with pytype.
Matt Harbison <matt_harbison@yahoo.com> [Wed, 23 Nov 2022 20:50:39 -0500] rev 49768
contrib: add a script for adding vendored type stubs to typeshed
I really hate this, but pytype doesn't support PEP 561 and doesn't seem to have
the equivalent of `MYPYPATH` to point to custom stubs. Ignoring the vendored
stubs isn't necessarily harmful, but pytype has been choking on the vendored
attr package after pytype 2022.03.29 with errors like this:
File "/mnt/c/Users/Matt/hg/mercurial/linelog.py", line 52, in __iter__: Built-in function iter was called with the wrong arguments [wrong-arg-types]
Expected: (collection: bytearray)
Actually passed: (collection: mercurial.thirdparty.attr._make._CountingAttr)
File "/mnt/c/Users/Matt/hg/mercurial/dirstateutils/v2.py", line 143, in pack: Built-in function len was called with the wrong arguments [wrong-arg-types]
Expected: (obj: Sized)
Actually passed: (obj: mercurial.thirdparty.attr._make._CountingAttr)
Attributes of protocol Sized are not implemented on mercurial.thirdparty.attr._make._CountingAttr: __len__
File "/mnt/c/Users/Matt/hg/mercurial/dirstateutils/v2.py", line 144, in pack: No attribute 'rfind' on mercurial.thirdparty.attr._make._CountingAttr [attribute-error]
File "/mnt/c/Users/Matt/hg/mercurial/dirstateutils/v2.py", line 146, in pack: Built-in function len was called with the wrong arguments [wrong-arg-types]
Expected: (obj: Sized)
Actually passed: (obj: mercurial.thirdparty.attr._make._CountingAttr)
Attributes of protocol Sized are not implemented on mercurial.thirdparty.attr._make._CountingAttr: __len__
File "/mnt/c/Users/Matt/hg/mercurial/dirstateutils/v2.py", line 152, in pack: No attribute 'v2_data' on mercurial.thirdparty.attr._make._CountingAttr [attribute-error]
File "/mnt/c/Users/Matt/hg/mercurial/util.py", line 2817, in go: unsupported operand type(s) for /: 'count: mercurial.thirdparty.attr._make._CountingAttr' and 'float: float' [unsupported-operands]
No attribute '__truediv__' on 'count: mercurial.thirdparty.attr._make._CountingAttr' or '__rtruediv__' on 'float: float'
Called from (traceback):
line 2981, in __bytes__
This is essentially the same hack we've been using in TortoiseHg to add the
vendored PyQt5 stubs. What I don't understand is pytype *still* generates *.pyi
files under .pytype/pyi/mercurial/thirdparty/attr, even when the package is
explicitly ignored in the pytype command line args. But it avoids the errors,
which means we aren't stuck on pytype==2022.03.29.
https://github.com/google/pytype/issues/151
Matt Harbison <matt_harbison@yahoo.com> [Wed, 23 Nov 2022 20:23:26 -0500] rev 49767
contrib: update check-pytype.sh to list stubs that caused pytype to crash
The same logic is in the TortoiseHg tests for running pytype, and it's useful to
know if a new version of pytype is better or worse.
Matt Harbison <matt_harbison@yahoo.com> [Wed, 23 Nov 2022 16:11:20 -0500] rev 49766
typing: add py.typed to mercurial.cext for PEP 561 support
Unfortunately, pytype doesn't support this yet. But it was included with the
attr package, so we might as well do it here for consistency. Unlike the attr
package, these type hints are only partial, so they are marked as such[1] (but
who knows if it matters, given these are C extensions, so no local source code
to scan).
[1] https://peps.python.org/pep-0561/#partial-stub-packages