view tests/test-convert-bzr-merges.t @ 42044:bb271ec2fbfb

compression: introduce a `storage.revlog.zstd.level` configuration

This option control the zstd compression level used when compressing revlog
chunk. The usage of zstd for revlog compression has not graduated from
experimental yet, but we intend to fix that soon.

The option name for the compression level is more straight forward to pick, so
this changesets comes first.  Having a dedicated option for each compression
engine is useful because they don't support the same range of values.

I ran the same measurement as for the zlib compression level (in the parent
changesets). The variation in repository size is stay mostly in the same (small)
range. The "read/write" performance see smallish variation, but are overall much
better than zlib. Write performance show the same tend of having better write
performance for when reaching high-end compression.

Again, we don't intend to change the default zstd compression level (currently:
3) in this series. However this is worth investigating in the future.

The Performance comparison of zlib vs zstd is quite impressive. The repository
size stay in the same range, but the performance are much better in all
situations.

Comparison summary
==================

We are looking at:
- performance range for zlib
- performance range for zstd
- comparison of default zstd (level-3) to default zlib (level 6)
- comparison of the slowest zstd time to the fastest zlib time

Read performance:
-----------------
          |           zlib          |           zstd          | cmp | f2s
mercurial |   0.170159 -   0.189219 |   0.144127 -   0.149624 | 80% | 88%
pypy      |   2.679217 -   2.768691 |   1.532317 -   1.705044 | 60% | 63%
netbeans  | 122.477027 - 141.620281 |  72.996346 -  89.731560 | 58% | 73%
mozilla   | 147.867662 - 170.572118 |  91.700995 - 105.853099 | 56% | 71%

Write performance:
------------------
          |           zlib          |           zstd          | cmp | f2s
mercurial |  53.250304 - 56.2936129 |  40.877025 -  45.677286 | 75% | 86%
pypy      | 460.721984 - 476.589918 | 270.545409 - 301.002219 | 63% | 65%
netbeans  | 520.560316 - 715.930400 | 370.356311 - 428.329652 | 55% | 82%
mozilla   | 739.803002 - 987.056093 | 505.152906 - 591.930683 | 57% | 80%

Raw data
--------

repo      alg lvl  .hg/store size  00manifest.d read       write

mercurial zlib  1      49,402,813     5,963,475   0.170159  53.250304
mercurial zlib  6      47,197,397     5,875,730   0.182820  56.264320
mercurial zlib  9      47,121,596     5,849,781   0.189219  56.293612

mercurial zstd  1      49,737,084     5,966,355   0.144127  40.877025
mercurial zstd  3      48,961,867     5,895,208   0.146376  42.268142
mercurial zstd  5      48,200,592     5,938,676   0.149624  43.162875
mercurial zstd 10      47,833,520     5,913,353   0.145185  44.012489
mercurial zstd 15      47,314,604     5,728,679   0.147686  45.677286
mercurial zstd 20      47,330,502     5,830,539   0.145789  45.025407
mercurial zstd 22      47,330,076     5,830,539   0.143996  44.690460


pypy      zlib  1     370,830,572    28,462,425   2.679217 460.721984
pypy      zlib  6     340,112,317    27,648,747   2.768691 467.537158
pypy      zlib  9     338,360,736    27,639,003   2.763495 476.589918

pypy      zstd  1     362,377,479    27,916,214   1.532317 270.545409
pypy      zstd  3     354,137,693    27,905,988   1.686718 294.951509
pypy      zstd  5     342,640,043    27,655,774   1.705044 301.002219
pypy      zstd 10     334,224,327    27,164,493   1.567287 285.186239
pypy      zstd 15     329,000,363    26,645,965   1.637729 299.561332
pypy      zstd 20     324,534,039    26,199,547   1.526813 302.149827
pypy      zstd 22     324,530,595    26,198,932   1.525718 307.821218


netbeans  zlib  1   1,281,847,810   165,495,457 122.477027 520.560316
netbeans  zlib  6   1,205,284,353   159,161,207 139.876147 715.930400
netbeans  zlib  9   1,197,135,671   155,034,586 141.620281 678.297064

netbeans  zstd  1   1,259,581,737   160,840,613  72.996346 370.356311
netbeans  zstd  3   1,232,978,122   157,691,551  81.622317 396.733087
netbeans  zstd  5   1,208,034,075   160,246,880  83.080549 364.342626
netbeans  zstd 10   1,188,624,176   156,083,417  79.323935 403.594602
netbeans  zstd 15   1,176,973,589   153,859,477  89.731560 428.329652
netbeans  zstd 20   1,162,958,258   151,147,535  82.842667 392.335349
netbeans  zstd 22   1,162,707,029   151,150,220  82.565695 402.840655


mozilla   zlib  1   2,775,497,186   298,527,987 147.867662 751.263721
mozilla   zlib  6   2,596,856,420   286,597,671 170.572118 987.056093
mozilla   zlib  9   2,587,542,494   287,018,264 163.622338 739.803002

mozilla   zstd  1   2,723,159,348   286,617,532  91.700995 570.042751
mozilla   zstd  3   2,665,055,001   286,152,013  95.240155 561.412805
mozilla   zstd  5   2,607,819,817   288,060,030 101.978048 505.152906
mozilla   zstd 10   2,558,761,085   283,967,648 104.113481 497.771202
mozilla   zstd 15   2,526,216,060   275,581,300 105.853099 591.930683
mozilla   zstd 20   2,485,114,806   266,478,859  95.268795 576.515389
mozilla   zstd 22   2,484,869,080   266,456,505  94.429282 572.785537

#require bzr

N.B. bzr 1.13 has a bug that breaks this test.  If you see this
test fail, check your bzr version.  Upgrading to bzr 1.13.1
should fix it.

  $ . "$TESTDIR/bzr-definitions"

test multiple merges at once

  $ mkdir test-multimerge
  $ cd test-multimerge
  $ bzr init -q source
  $ cd source
  $ echo content > file
  $ echo text > rename_me
  $ bzr add -q file rename_me
  $ bzr commit -q -m 'Initial add' '--commit-time=2009-10-10 08:00:00 +0100'
  $ cd ..
  $ bzr branch -q source source-branch1
  $ cd source-branch1
  $ echo morecontent >> file
  $ echo evenmorecontent > file-branch1
  $ bzr add -q file-branch1
  $ bzr commit -q -m 'Added branch1 file' '--commit-time=2009-10-10 08:00:01 +0100'
  $ cd ../source
  $ sleep 1
  $ echo content > file-parent
  $ bzr add -q file-parent
  $ bzr commit -q -m 'Added parent file' '--commit-time=2009-10-10 08:00:02 +0100'
  $ cd ..
  $ bzr branch -q source source-branch2
  $ cd source-branch2
  $ echo somecontent > file-branch2
  $ bzr add -q file-branch2
  $ bzr mv -q rename_me renamed
  $ echo change > renamed
  $ bzr commit -q -m 'Added brach2 file' '--commit-time=2009-10-10 08:00:03 +0100'
  $ sleep 1
  $ cd ../source
  $ bzr merge -q ../source-branch1
  $ bzr merge -q --force ../source-branch2
  $ bzr commit -q -m 'Merged branches' '--commit-time=2009-10-10 08:00:04 +0100'
  $ cd ..

BUG: file-branch2 should not be added in rev 4, and the rename_me -> renamed
move should be recorded in the fixup merge.
  $ hg convert --datesort --config convert.bzr.saverev=False source source-hg
  initializing destination source-hg repository
  scanning source...
  sorting...
  converting...
  4 Initial add
  3 Added branch1 file
  2 Added parent file
  1 Added brach2 file
  0 Merged branches
  warning: can't find ancestor for 'renamed' copied from 'rename_me'!
  $ glog -R source-hg
  o    5@source "(octopus merge fixup)" files+: [], files-: [], files: [renamed]
  |\
  | o    4@source "Merged branches" files+: [file-branch1 file-branch2 renamed], files-: [rename_me], files: [file]
  | |\
  o---+  3@source-branch2 "Added brach2 file" files+: [file-branch2 renamed], files-: [rename_me], files: []
   / /
  | o  2@source "Added parent file" files+: [file-parent], files-: [], files: []
  | |
  o |  1@source-branch1 "Added branch1 file" files+: [file-branch1], files-: [], files: [file]
  |/
  o  0@source "Initial add" files+: [file rename_me], files-: [], files: []
  
  $ manifest source-hg tip
  % manifest of tip
  644   file
  644   file-branch1
  644   file-branch2
  644   file-parent
  644   renamed

  $ hg convert source-hg hg2hg
  initializing destination hg2hg repository
  scanning source...
  sorting...
  converting...
  5 Initial add
  4 Added branch1 file
  3 Added parent file
  2 Added brach2 file
  1 Merged branches
  0 (octopus merge fixup)

BUG: The manifest entries should be the same for matching revisions, and
nothing should be outgoing

  $ hg -R source-hg manifest --debug -r tip | grep renamed
  67109fdebf6c556eb0a9d5696dd98c8420520405 644   renamed
  $ hg -R hg2hg manifest --debug -r tip | grep renamed
  27c968376d7c3afd095ecb9c7697919b933448c8 644   renamed
  $ hg -R source-hg manifest --debug -r 'tip^' | grep renamed
  27c968376d7c3afd095ecb9c7697919b933448c8 644   renamed
  $ hg -R hg2hg manifest --debug -r 'tip^' | grep renamed
  27c968376d7c3afd095ecb9c7697919b933448c8 644   renamed

BUG: The revisions found should be the same in both repos

  $ hg --cwd source-hg log -r 'file("renamed")' -G -Tcompact
  o    5[tip]:4,3   6652429c300a   2009-10-10 08:00 +0100   foo
  |\     (octopus merge fixup)
  | |
  | o    4:2,1   e0ae8af3503a   2009-10-10 08:00 +0100   foo
  | |\     Merged branches
  | ~ ~
  o  3   138bed2e14be   2009-10-10 08:00 +0100   foo
  |    Added brach2 file
  ~
  $ hg --cwd hg2hg log -r 'file("renamed")' -G -Tcompact
  o    4:2,1   e0ae8af3503a   2009-10-10 08:00 +0100   foo
  |\     Merged branches
  ~ ~
  o  3   138bed2e14be   2009-10-10 08:00 +0100   foo
  |    Added brach2 file
  ~

BUG(?): The move seems to be recorded in rev 4, so it should probably show up
there.  It's not recorded as a move in rev 5, even in source-hg.

  $ hg -R source-hg up -q tip
  $ hg -R hg2hg up -q tip
  $ hg --cwd source-hg log -r 'follow("renamed")' -G -Tcompact
  @    5[tip]:4,3   6652429c300a   2009-10-10 08:00 +0100   foo
  |\     (octopus merge fixup)
  | :
  o :  3   138bed2e14be   2009-10-10 08:00 +0100   foo
  :/     Added brach2 file
  :
  o  0   18b86f5df51b   2009-10-10 08:00 +0100   foo
       Initial add
  
  $ hg --cwd hg2hg log -r 'follow("renamed")' -G -Tcompact
  o  3   138bed2e14be   2009-10-10 08:00 +0100   foo
  :    Added brach2 file
  :
  o  0   18b86f5df51b   2009-10-10 08:00 +0100   foo
       Initial add
  

  $ hg -R hg2hg out source-hg -T compact
  comparing with source-hg
  searching for changes
  5[tip]:4,3   3be2299ccd31   2009-10-10 08:00 +0100   foo
    (octopus merge fixup)
  

  $ glog -R hg2hg
  @    5@source "(octopus merge fixup)" files+: [], files-: [], files: []
  |\
  | o    4@source "Merged branches" files+: [file-branch1 file-branch2 renamed], files-: [rename_me], files: [file]
  | |\
  o---+  3@source-branch2 "Added brach2 file" files+: [file-branch2 renamed], files-: [rename_me], files: []
   / /
  | o  2@source "Added parent file" files+: [file-parent], files-: [], files: []
  | |
  o |  1@source-branch1 "Added branch1 file" files+: [file-branch1], files-: [], files: [file]
  |/
  o  0@source "Initial add" files+: [file rename_me], files-: [], files: []
  

  $ hg -R source-hg log --debug -r tip
  changeset:   5:6652429c300ab66fdeaf2e730945676a00b53231
  branch:      source
  tag:         tip
  phase:       draft
  parent:      4:e0ae8af3503af9bbffb0b29268a02744cc61a561
  parent:      3:138bed2e14be415a2692b02e41405b2864f758b4
  manifest:    5:1eabd5f5d4b985784cf2c45c717ff053eca14b0d
  user:        Foo Bar <[e-mail]>
  date:        Sat Oct 10 08:00:04 2009 +0100
  files:       renamed
  extra:       branch=source
  description:
  (octopus merge fixup)
  
  
  $ hg -R hg2hg log --debug -r tip
  changeset:   5:3be2299ccd315ff9aab2b49bdb0d14e3244435e8
  branch:      source
  tag:         tip
  phase:       draft
  parent:      4:e0ae8af3503af9bbffb0b29268a02744cc61a561
  parent:      3:138bed2e14be415a2692b02e41405b2864f758b4
  manifest:    4:3ece3c7f2cc6df15b3cbbf3273c69869fc7c3ab0
  user:        Foo Bar <[e-mail]>
  date:        Sat Oct 10 08:00:04 2009 +0100
  extra:       branch=source
  description:
  (octopus merge fixup)
  
  
  $ hg -R source-hg manifest --debug -r tip
  cdf31ed9242b209cd94697112160e2c5b37a667d 644   file
  5108144f585149b29779d7c7e51d61dd22303ffe 644   file-branch1
  80753c4a9ac3806858405b96b24a907b309e3616 644   file-branch2
  7108421418404a937c684d2479a34a24d2ce4757 644   file-parent
  67109fdebf6c556eb0a9d5696dd98c8420520405 644   renamed
  $ hg -R source-hg manifest --debug -r 'tip^'
  cdf31ed9242b209cd94697112160e2c5b37a667d 644   file
  5108144f585149b29779d7c7e51d61dd22303ffe 644   file-branch1
  80753c4a9ac3806858405b96b24a907b309e3616 644   file-branch2
  7108421418404a937c684d2479a34a24d2ce4757 644   file-parent
  27c968376d7c3afd095ecb9c7697919b933448c8 644   renamed

  $ hg -R hg2hg manifest --debug -r tip
  cdf31ed9242b209cd94697112160e2c5b37a667d 644   file
  5108144f585149b29779d7c7e51d61dd22303ffe 644   file-branch1
  80753c4a9ac3806858405b96b24a907b309e3616 644   file-branch2
  7108421418404a937c684d2479a34a24d2ce4757 644   file-parent
  27c968376d7c3afd095ecb9c7697919b933448c8 644   renamed
  $ hg -R hg2hg manifest --debug -r 'tip^'
  cdf31ed9242b209cd94697112160e2c5b37a667d 644   file
  5108144f585149b29779d7c7e51d61dd22303ffe 644   file-branch1
  80753c4a9ac3806858405b96b24a907b309e3616 644   file-branch2
  7108421418404a937c684d2479a34a24d2ce4757 644   file-parent
  27c968376d7c3afd095ecb9c7697919b933448c8 644   renamed

  $ cd ..