Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 02 Apr 2019 11:03:46 -0700] rev 42045
compression: display compression level in debugformat
Now that we have options to control the compression level, we teach `hg
debugformat` about them. This is a useful information when comparing
repositories.
Note that we have no trace of the compression level used to store existing
deltas. Actually, it would even varies from one delta to another. So we display
the currently set value.
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 27 Mar 2019 18:35:59 +0100] rev 42044
compression: introduce a `storage.revlog.zstd.level` configuration
This option control the zstd compression level used when compressing revlog
chunk. The usage of zstd for revlog compression has not graduated from
experimental yet, but we intend to fix that soon.
The option name for the compression level is more straight forward to pick, so
this changesets comes first. Having a dedicated option for each compression
engine is useful because they don't support the same range of values.
I ran the same measurement as for the zlib compression level (in the parent
changesets). The variation in repository size is stay mostly in the same (small)
range. The "read/write" performance see smallish variation, but are overall much
better than zlib. Write performance show the same tend of having better write
performance for when reaching high-end compression.
Again, we don't intend to change the default zstd compression level (currently:
3) in this series. However this is worth investigating in the future.
The Performance comparison of zlib vs zstd is quite impressive. The repository
size stay in the same range, but the performance are much better in all
situations.
Comparison summary
==================
We are looking at:
- performance range for zlib
- performance range for zstd
- comparison of default zstd (level-3) to default zlib (level 6)
- comparison of the slowest zstd time to the fastest zlib time
Read performance:
-----------------
| zlib | zstd | cmp | f2s
mercurial | 0.170159 - 0.189219 | 0.144127 - 0.149624 | 80% | 88%
pypy | 2.679217 - 2.768691 | 1.532317 - 1.705044 | 60% | 63%
netbeans | 122.477027 - 141.620281 | 72.996346 - 89.731560 | 58% | 73%
mozilla | 147.867662 - 170.572118 | 91.700995 - 105.853099 | 56% | 71%
Write performance:
------------------
| zlib | zstd | cmp | f2s
mercurial | 53.250304 - 56.2936129 | 40.877025 - 45.677286 | 75% | 86%
pypy | 460.721984 - 476.589918 | 270.545409 - 301.002219 | 63% | 65%
netbeans | 520.560316 - 715.930400 | 370.356311 - 428.329652 | 55% | 82%
mozilla | 739.803002 - 987.056093 | 505.152906 - 591.930683 | 57% | 80%
Raw data
--------
repo alg lvl .hg/store size 00manifest.d read write
mercurial zlib 1 49,402,813 5,963,475 0.170159 53.250304
mercurial zlib 6 47,197,397 5,875,730 0.182820 56.264320
mercurial zlib 9 47,121,596 5,849,781 0.189219 56.293612
mercurial zstd 1 49,737,084 5,966,355 0.144127 40.877025
mercurial zstd 3 48,961,867 5,895,208 0.146376 42.268142
mercurial zstd 5 48,200,592 5,938,676 0.149624 43.162875
mercurial zstd 10 47,833,520 5,913,353 0.145185 44.012489
mercurial zstd 15 47,314,604 5,728,679 0.147686 45.677286
mercurial zstd 20 47,330,502 5,830,539 0.145789 45.025407
mercurial zstd 22 47,330,076 5,830,539 0.143996 44.690460
pypy zlib 1 370,830,572 28,462,425 2.679217 460.721984
pypy zlib 6 340,112,317 27,648,747 2.768691 467.537158
pypy zlib 9 338,360,736 27,639,003 2.763495 476.589918
pypy zstd 1 362,377,479 27,916,214 1.532317 270.545409
pypy zstd 3 354,137,693 27,905,988 1.686718 294.951509
pypy zstd 5 342,640,043 27,655,774 1.705044 301.002219
pypy zstd 10 334,224,327 27,164,493 1.567287 285.186239
pypy zstd 15 329,000,363 26,645,965 1.637729 299.561332
pypy zstd 20 324,534,039 26,199,547 1.526813 302.149827
pypy zstd 22 324,530,595 26,198,932 1.525718 307.821218
netbeans zlib 1 1,281,847,810 165,495,457 122.477027 520.560316
netbeans zlib 6 1,205,284,353 159,161,207 139.876147 715.930400
netbeans zlib 9 1,197,135,671 155,034,586 141.620281 678.297064
netbeans zstd 1 1,259,581,737 160,840,613 72.996346 370.356311
netbeans zstd 3 1,232,978,122 157,691,551 81.622317 396.733087
netbeans zstd 5 1,208,034,075 160,246,880 83.080549 364.342626
netbeans zstd 10 1,188,624,176 156,083,417 79.323935 403.594602
netbeans zstd 15 1,176,973,589 153,859,477 89.731560 428.329652
netbeans zstd 20 1,162,958,258 151,147,535 82.842667 392.335349
netbeans zstd 22 1,162,707,029 151,150,220 82.565695 402.840655
mozilla zlib 1 2,775,497,186 298,527,987 147.867662 751.263721
mozilla zlib 6 2,596,856,420 286,597,671 170.572118 987.056093
mozilla zlib 9 2,587,542,494 287,018,264 163.622338 739.803002
mozilla zstd 1 2,723,159,348 286,617,532 91.700995 570.042751
mozilla zstd 3 2,665,055,001 286,152,013 95.240155 561.412805
mozilla zstd 5 2,607,819,817 288,060,030 101.978048 505.152906
mozilla zstd 10 2,558,761,085 283,967,648 104.113481 497.771202
mozilla zstd 15 2,526,216,060 275,581,300 105.853099 591.930683
mozilla zstd 20 2,485,114,806 266,478,859 95.268795 576.515389
mozilla zstd 22 2,484,869,080 266,456,505 94.429282 572.785537
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 27 Mar 2019 18:35:27 +0100] rev 42043
compression: introduce a `storage.revlog.zlib.level` configuration
This option control the zlib compression level used when compression revlog
chunk.
This is also a good excuse to pave the way for a similar configuration option
for the zstd compression engine. Having a dedicated option for each compression
algorithm is useful because they don't support the same range of values.
Using a higher zlib compression impact CPU consumption at compression time, but
does not directly affected decompression time. However dealing with small
compressed chunk can directly help decompression and indirectly help other
revlog logic.
I ran some basic test on repositories using different level. I am using the
mercurial, pypy, netbeans and mozilla-central clone from our benchmark suite.
All tested repository use sparse-revlog and got all their delta recomputed.
The different compression level has a small effect on the repository size
(about 10% variation in the total range). My quick analysis is that revlog
mostly store small delta, that are not affected by the compression level much.
So the variation probably mostly comes from better compression of the snapshots
revisions, and snapshot revision only represent a small portion of the
repository content.
I also made some basic timings measurements. The "read" timings are gathered using
simple run of `hg perfrevlogrevisions`, the "write" timings using `hg
perfrevlogwrite` (restricted to the last 5000 revisions for netbeans and
mozilla central). The timings are gathered on a generic machine, (not one of
our performance locked machine), so small variation might not be meaningful.
However large trend remains relevant.
Keep in mind that these numbers are not pure compression/decompression time.
They also involve the full revlog logic. In particular the difference in chunk
size has an impact on the delta chain structure, affecting performance when
writing or reading them.
On read/write performance, the compression level has a bigger impact.
Counter-intuitively, the higher compression levels improve "write" performance
for the large repositories in our tested setting. Maybe because the last 5000
delta chain end up having a very different shape in this specific spot? Or maybe
because of a more general trend of better delta chains thanks to the smaller
chunk and snapshot.
This series does not intend to change the default compression level. However,
these result call for a deeper analysis of this performance difference in the
future.
Full data
=========
repo level .hg/store size 00manifest.d read write
----------------------------------------------------------------
mercurial 1 49,402,813 5,963,475 0.170159 53.250304
mercurial 6 47,197,397 5,875,730 0.182820 56.264320
mercurial 9 47,121,596 5,849,781 0.189219 56.293612
pypy 1 370,830,572 28,462,425 2.679217 460.721984
pypy 6 340,112,317 27,648,747 2.768691 467.537158
pypy 9 338,360,736 27,639,003 2.763495 476.589918
netbeans 1 1,281,847,810 165,495,457 122.477027 520.560316
netbeans 6 1,205,284,353 159,161,207 139.876147 715.930400
netbeans 9 1,197,135,671 155,034,586 141.620281 678.297064
mozilla 1 2,775,497,186 298,527,987 147.867662 751.263721
mozilla 6 2,596,856,420 286,597,671 170.572118 987.056093
mozilla 9 2,587,542,494 287,018,264 163.622338 739.803002
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 27 Mar 2019 19:34:10 +0100] rev 42042
compression: accept level management for zlib compression
We update the zlib related class to be support setting the compression level.
This changeset focus on updating the internal only. A way to configure this
level will be introduced in the next changeset.
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 27 Mar 2019 16:45:14 +0100] rev 42041
util: extract compression code in `mercurial.utils.compression`
The code seems large enough to be worth extracting. This is similar to what was
done for various module in `mercurial/utils/`.
Since None of the compression logic takes a `ui` objet, issuing deprecation
warning is tricky. Luckly the logic does not seems to have many external users.
Martin von Zweigbergk <martinvonz@google.com> [Sat, 30 Mar 2019 13:13:10 -0700] rev 42040
merge: make "labels" argument to graft() optional, like it is for update()
graft() just passes the argument on to update(), and update() doesn't
require it, so graft() shouldn't either.
Differential Revision: https://phab.mercurial-scm.org/D6175
Martin von Zweigbergk <martinvonz@google.com> [Sun, 31 Mar 2019 09:39:02 -0700] rev 42039
revset: remove comment about linkrev workaround from user-facing docs
I think the code is clear enough so we don't need to keep the comment
at all (by now, most Mercurial developers are probably familiar with
the linkrevs issues).
Differential Revision: https://phab.mercurial-scm.org/D6176
Martin von Zweigbergk <martinvonz@google.com> [Fri, 29 Mar 2019 11:32:02 -0700] rev 42038
shelve: let cmdutil.revert() take care of backing up untracked files
cmdutil.revert() backs up untracked files, so I don't see a reason to
do it shelve.mergefiles(). We have tests for this and they still pass.
Differential Revision: https://phab.mercurial-scm.org/D6174
Martin von Zweigbergk <martinvonz@google.com> [Fri, 29 Mar 2019 11:31:42 -0700] rev 42037
shelve: stop passing list of files to revert
It seems to work just fine to not specify any files here. I suspect it
looked the way it did for historical reasons. It apparently used to
use merge instead of rebase until 1d7a36ff2615 (shelve: use rebase
instead of merge (issue4068), 2013-10-23) and it makes sense to want
to restrict the set of files then.
I noticed this because of the files.extend(shelvectx.p1().files()). If
the working copy was clean before, then shelvectx.p1() will be the
working copy parent and that ended up adding all the files in that
set. In our Google-internal Mercurial setup (including a FUSE) that
was very noticeably slow when the working copy parent happened to have
many files in large directories.
This patch doesn't yet remove the call to shelvectx.p1().files(). We
also use that set for deciding what to back up. I'm pretty sure it's
safe to back up only the set of files we already back even if we no
longer restrict the set of files to revert, so this patch should be
safe on its own. Regardless, the next patch will delegate the
backing-up to cmdutil.revert().
Incidentally, this also gets rid of a repo.pathto() that I had earlier
wanted to get rid of.
Differential Revision: https://phab.mercurial-scm.org/D6173
Martin von Zweigbergk <martinvonz@google.com> [Wed, 27 Mar 2019 14:55:46 -0700] rev 42036
remotefilelog: prefetch files in deterministic order
I have been troubleshooting some slowness in this area (it's unclear
if it's the client or server that's to blame, but that's beside the
point) and it's a lot easier to do troubleshoot if the files are
prefetched in the same order each time.
Differential Revision: https://phab.mercurial-scm.org/D6172
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 26 Mar 2019 17:35:28 +0100] rev 42035
debugdiscovery: display time elapsed during the discovery step
This is a useful information. Now that we perform more analysing after the
discovery is done, it is worth have a more precise measurement. For serious
timing analysis use `hg perfdiscovery`.
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 26 Mar 2019 17:26:54 +0100] rev 42034
debugdiscovery: only list common heads on verbose
The list of common heads is only part of the useful information. In addition on
repository with many heads, the information is very not helpful (just fill a
couple of screen with hash). As a result we hide it behind a --verbose flag.
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 26 Mar 2019 17:26:11 +0100] rev 42033
debugdiscovery: drop duplicated information
The old line informing about the local being a superset or subset of the remote
is redundant with the newly introduced data. So we drop it.
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 26 Mar 2019 17:25:22 +0100] rev 42032
debugdiscovery: display more statistic about the common set
We display a lot more information now. Especially, we display the overlap
between the common heads and the local/remote heads. There are various
optimization geared toward heads, as a result, the less common the heads the
more complex the discovery. Having this information easily accessible help when
working on discovery.