view tests/test-releasenotes-formatting.t @ 51681:522b4d729e89

mmap: populate the mapping by default

Without pre-population, accessing all data through a mmap can result in many
pagefault, reducing performance significantly. If the mmap is prepopulated, the
performance can no longer get slower than a full read.
(See benchmark number below)

In some cases were very few data is read, prepopulating can be overkill and
slower than populating on access (through page fault). So that behavior can be
controlled when the caller can pre-determine the best behavior.
(See benchmark number below)

In addition, testing with populating in a secondary thread yield great result
combining the best of each approach. This might be implemented in later
changesets.

In all cases, using mmap has a great effect on memory usage when many processes
run in parallel on the same machine.

### Benchmarks

# What did I run

A couple of month back I ran a large benchmark campaign to assess the impact of
various approach for using mmap with the revlog (and other files), it
highlighted a few benchmarks that capture the impact of the changes well. So to
validate this change I checked the following:

- log command displaying various revisions
  (read the changelog index)
- log command displaying the patch of listed revisions
  (read the changelog index, the manifest index and a few files indexes)
- unbundling a few revisions
  (read and write changelog, manifest and few files indexes, and walk the graph
  to update some cache)
- pushing a few revisions
  (read and write changelog, manifest and few files indexes, walk the graph to
  update some cache, performs various accesses locally and remotely during
  discovery)

Benchmarks were run using the default module policy (c+py) and the rust one. No
significant difference were found between the two implementation, so we will
present result using the default policy (unless otherwise specified).

I ran them on a few repositories :

- mercurial: a "public changeset only" copy of mercurial from 2018-08-01 using
  zstd compression and sparse-revlog

- pypy: a copy of pypy from 2018-08-01 using zstd compression and sparse-revlog

- netbeans: a copy of netbeans from 2018-08-01 using zstd compression and
  sparse-revlog

- mozilla-try: a copy of mozilla-try from 2019-02-18 using zstd compression and
  sparse-revlog

- mozilla-try persistent-nodemap: Same as the above but with a persistent
  nodemap. Used for the log --patch benchmark only

# Results

For the smaller repositories (mercurial, pypy), the impact of mmap is almost
imperceptible, other cost dominating the operation. The impact of prepopulating
is undiscernible in the benchmark we ran.

For larger repositories the benchmark support explanation given above:

On netbeans, the log can be about 1% faster without repopulation (for a
difference < 100ms) but unbundle becomes a bit slower, even when small.

### data-env-vars.name                             = netbeans-2018-08-01-zstd-sparse-revlog
  # benchmark.name                                 = hg.command.unbundle
  # benchmark.variants.issue6528                   = disabled
  # benchmark.variants.reuse-external-delta-parent = yes
  # benchmark.variants.revs                        = any-1-extra-rev
  # benchmark.variants.source                      = unbundle
  # benchmark.variants.verbosity                   = quiet
with-populate: 0.240157
no-populate:   0.265087  (+10.38%, +0.02)
  # benchmark.variants.revs                        = any-100-extra-rev
with-populate: 1.459518
no-populate:   1.481290   (+1.49%, +0.02)
 ## benchmark.name                                 = hg.command.push
  # benchmark.variants.explicit-rev                = none
  # benchmark.variants.issue6528                   = disabled
  # benchmark.variants.protocol                    = ssh
  # benchmark.variants.reuse-external-delta-parent = yes
  # benchmark.variants.revs                        = any-1-extra-rev
with-populate: 0.771919
no-populate:   0.792025   (+2.60%, +0.02)
  # benchmark.variants.revs                        = any-100-extra-rev
with-populate: 1.459518
no-populate:   1.481290   (+1.49%, +0.02)


For mozilla-try, the "slow down" from pre-populate for small `hg log` is more
visible, but still small in absolute time. (using rust value for the persistent
nodemap value to be relevant).

### data-env-vars.name            = mozilla-try-2019-02-18-ds2-pnm
  # benchmark.name                = hg.command.log
  # bin-env-vars.hg.flavor        = rust
  # benchmark.variants.patch      = yes
  # benchmark.variants.limit-rev  = 1
with-populate: 0.237813
no-populate:   0.229452   (-3.52%, -0.01)
  # benchmark.variants.limit-rev  = 10
  # benchmark.variants.patch      = yes
with-populate: 1.213578
no-populate:   1.205189
### data-env-vars.name            = mozilla-try-2019-02-18-zstd-sparse-revlog
  # benchmark.variants.limit-rev  = 1000
  # benchmark.variants.patch      = no
  # benchmark.variants.rev        = tip
with-populate: 0.198607
no-populate:   0.195038   (-1.80%, -0.00)

However pre-populating provide a significant boost on more complex operations
like unbundle or push:

### data-env-vars.name                             = mozilla-try-2019-02-18-zstd-sparse-revlog
  # benchmark.name                                 = hg.command.push
  # benchmark.variants.explicit-rev                = none
  # benchmark.variants.issue6528                   = disabled
  # benchmark.variants.protocol                    = ssh
  # benchmark.variants.reuse-external-delta-parent = yes
  # benchmark.variants.revs                        = any-1-extra-rev
with-populate: 4.798632
no-populate:   4.953295   (+3.22%, +0.15)
  # benchmark.variants.revs                        = any-100-extra-rev
with-populate: 4.903618
no-populate:   5.014963   (+2.27%, +0.11)
 ## benchmark.name                                 = hg.command.unbundle
  # benchmark.variants.revs                        = any-1-extra-rev
with-populate: 1.423411
no-populate:   1.585365  (+11.38%, +0.16)
  # benchmark.variants.revs                        = any-100-extra-rev
with-populate: 1.537909
no-populate:   1.688489   (+9.79%, +0.15)

#require fuzzywuzzy

  $ cat >> $HGRCPATH << EOF
  > [extensions]
  > releasenotes=
  > EOF

  $ hg init simple-repo
  $ cd simple-repo

A fix with a single line results in a bullet point in the appropriate section

  $ touch fix1
  $ hg -q commit -A -l - << EOF
  > single line fix
  > 
  > .. fix::
  > 
  >    Simple fix with a single line content entry.
  > EOF

  $ hg releasenotes -r . $TESTTMP/relnotes-single-line

  $ cat $TESTTMP/relnotes-single-line
  Bug Fixes
  =========
  
  * Simple fix with a single line content entry.

A fix with multiple lines is handled correctly

  $ touch fix2
  $ hg -q commit -A -l - << EOF
  > multi line fix
  > 
  > .. fix::
  > 
  >    First line of fix entry.
  >    A line after it without a space.
  > 
  >    A new paragraph in the fix entry. And this is a really long line. It goes on for a while.
  >    And it wraps around to a new paragraph.
  > EOF

  $ hg releasenotes -r . $TESTTMP/relnotes-multi-line
  $ cat $TESTTMP/relnotes-multi-line
  Bug Fixes
  =========
  
  * First line of fix entry. A line after it without a space.
  
    A new paragraph in the fix entry. And this is a really long line. It goes on
    for a while. And it wraps around to a new paragraph.

A release note with a title results in a sub-section being written

  $ touch fix3
  $ hg -q commit -A -l - << EOF
  > fix with title
  > 
  > .. fix:: Fix Title
  > 
  >    First line of fix with title.
  > 
  >    Another paragraph of fix with title. But this is a paragraph
  >    with multiple lines.
  > EOF

  $ hg releasenotes -r . $TESTTMP/relnotes-fix-with-title
  $ cat $TESTTMP/relnotes-fix-with-title
  Bug Fixes
  =========
  
  Fix Title
  ---------
  
  First line of fix with title.
  
  Another paragraph of fix with title. But this is a paragraph with multiple
  lines.

  $ cd ..

Formatting of multiple bullet points works

  $ hg init multiple-bullets
  $ cd multiple-bullets
  $ touch fix1
  $ hg -q commit -A -l - << EOF
  > commit 1
  > 
  > .. fix::
  > 
  >    first fix
  > EOF

  $ touch fix2
  $ hg -q commit -A -l - << EOF
  > commit 2
  > 
  > .. fix::
  > 
  >    second fix
  > 
  >    Second paragraph of second fix.
  > EOF

  $ touch fix3
  $ hg -q commit -A -l - << EOF
  > commit 3
  > 
  > .. fix::
  > 
  >    third fix
  > EOF

  $ hg releasenotes -r 'all()' $TESTTMP/relnotes-multiple-bullets
  $ cat $TESTTMP/relnotes-multiple-bullets
  Bug Fixes
  =========
  
  * first fix
  
  * second fix
  
    Second paragraph of second fix.
  
  * third fix

  $ cd ..

Formatting of multiple sections works

  $ hg init multiple-sections
  $ cd multiple-sections
  $ touch fix1
  $ hg -q commit -A -l - << EOF
  > commit 1
  > 
  > .. fix::
  > 
  >    first fix
  > EOF

  $ touch feature1
  $ hg -q commit -A -l - << EOF
  > commit 2
  > 
  > .. feature::
  > 
  >    description of the new feature
  > EOF

  $ touch fix2
  $ hg -q commit -A -l - << EOF
  > commit 3
  > 
  > .. fix::
  > 
  >    second fix
  > EOF

  $ hg releasenotes -r 'all()' $TESTTMP/relnotes-multiple-sections
  $ cat $TESTTMP/relnotes-multiple-sections
  New Features
  ============
  
  * description of the new feature
  
  Bug Fixes
  =========
  
  * first fix
  
  * second fix

  $ cd ..

Section with subsections and bullets

  $ hg init multiple-subsections
  $ cd multiple-subsections

  $ touch fix1
  $ hg -q commit -A -l - << EOF
  > commit 1
  > 
  > .. fix:: Title of First Fix
  > 
  >    First paragraph of first fix.
  > 
  >    Second paragraph of first fix.
  > EOF

  $ touch fix2
  $ hg -q commit -A -l - << EOF
  > commit 2
  > 
  > .. fix:: Title of Second Fix
  > 
  >    First paragraph of second fix.
  > 
  >    Second paragraph of second fix.
  > EOF

  $ hg releasenotes -r 'all()' $TESTTMP/relnotes-multiple-subsections
  $ cat $TESTTMP/relnotes-multiple-subsections
  Bug Fixes
  =========
  
  Title of First Fix
  ------------------
  
  First paragraph of first fix.
  
  Second paragraph of first fix.
  
  Title of Second Fix
  -------------------
  
  First paragraph of second fix.
  
  Second paragraph of second fix.

Now add bullet points to sections having sub-sections

  $ touch fix3
  $ hg -q commit -A -l - << EOF
  > commit 3
  > 
  > .. fix::
  > 
  >    Short summary of fix 3
  > EOF

  $ hg releasenotes -r 'all()' $TESTTMP/relnotes-multiple-subsections-with-bullets
  $ cat $TESTTMP/relnotes-multiple-subsections-with-bullets
  Bug Fixes
  =========
  
  Title of First Fix
  ------------------
  
  First paragraph of first fix.
  
  Second paragraph of first fix.
  
  Title of Second Fix
  -------------------
  
  First paragraph of second fix.
  
  Second paragraph of second fix.
  
  Other Changes
  -------------
  
  * Short summary of fix 3

  $ cd ..

Multiple 'Other Changes' sub-sections for every section

  $ hg init multiple-otherchanges
  $ cd multiple-otherchanges

  $ touch fix1
  $ hg -q commit -A -l - << EOF
  > commit 1
  > 
  > .. fix:: Title of First Fix
  > 
  >    First paragraph of fix 1.
  > EOF

  $ touch feature1
  $ hg -q commit -A -l - << EOF
  > commit 2
  > 
  > .. feature:: Title of First Feature
  > 
  >    First paragraph of feature 1.
  > EOF

  $ touch feature2
  $ hg -q commit -A -l - << EOF
  > commit 3
  > 
  > .. feature::
  > 
  >    Short summary of feature 2.
  > EOF

  $ touch fix2
  $ hg -q commit -A -l - << EOF
  > commit 4
  > 
  > .. fix::
  > 
  >    Short summary of fix 2
  > EOF

  $ hg releasenotes -r 'all()' $TESTTMP/relnotes-multiple-otherchanges
  $ cat $TESTTMP/relnotes-multiple-otherchanges
  New Features
  ============
  
  Title of First Feature
  ----------------------
  
  First paragraph of feature 1.
  
  Other Changes
  -------------
  
  * Short summary of feature 2.
  
  Bug Fixes
  =========
  
  Title of First Fix
  ------------------
  
  First paragraph of fix 1.
  
  Other Changes
  -------------
  
  * Short summary of fix 2

  $ cd ..

Using custom sections in notes

  $ hg init custom-section
  $ cd custom-section
  $ cat >> .hgreleasenotes << EOF
  > [sections]
  > testsection=Name of Section
  > EOF

  $ touch a
  $ hg -q commit -A -l - << EOF
  > commit 1
  > 
  > .. testsection::
  > 
  >    First paragraph under this admonition.
  > EOF

  $ hg releasenotes -r . $TESTTMP/relnotes-custom-section
  $ cat $TESTTMP/relnotes-custom-section
  Name of Section
  ===============
  
  * First paragraph under this admonition.

Overriding default sections (For eg. by default feature = New Features)

  $ cat >> .hgreleasenotes << EOF
  > [sections]
  > feature=Feature Additions
  > EOF

  $ touch b
  $ hg -q commit -A -l - << EOF
  > commit 2
  > 
  > .. feature::
  > 
  >    Adds a new feature.
  > EOF

  $ hg releasenotes -r . $TESTTMP/relnotes-override-section
  $ cat $TESTTMP/relnotes-override-section
  Feature Additions
  =================
  
  * Adds a new feature.

  $ cd ..

Testing output for the --check (-c) flag

  $ hg init check-flag
  $ cd check-flag

  $ touch a
  $ hg -q commit -A -l - << EOF
  > commit 2
  > 
  > .. asf::
  > 
  >    First paragraph under this admonition.
  > EOF

Suggest similar admonition in place of the invalid one.

  $ hg releasenotes -r . -c
  Invalid admonition 'asf' present in changeset 99fa3c800c5e

  $ touch b
  $ hg -q commit -A -l - << EOF
  > .. fixes::
  > 
  >    First paragraph under this admonition.
  > EOF

  $ hg releasenotes -r . -c
  Invalid admonition 'fixes' present in changeset 4737b1b5afd1
  (did you mean fix?)

  $ cd ..

Usage of --list flag

  $ hg init relnotes-list
  $ cd relnotes-list
  $ hg releasenotes -l
  feature: New Features
  bc: Backwards Compatibility Changes
  fix: Bug Fixes
  perf: Performance Improvements
  api: API Changes

  $ cd ..

Raise error on simultaneous usage of flags

  $ hg init relnotes-raise-error
  $ cd relnotes-raise-error
  $ hg releasenotes -r . -l
  abort: cannot specify both --list and --rev
  [10]

  $ hg releasenotes -l -c
  abort: cannot specify both --list and --check
  [10]

Display release notes for specified revs if no file is mentioned

  $ hg init relnotes-nofile
  $ cd relnotes-nofile

  $ touch fix1
  $ hg -q commit -A -l - << EOF
  > commit 1
  > 
  > .. fix:: Title of First Fix
  > 
  >    First paragraph of fix 1.
  > EOF

  $ hg releasenote -r .
  Bug Fixes
  =========
  
  Title of First Fix
  ------------------
  
  First paragraph of fix 1.

  $ cd ..

Using multiple admonitions in same changeset

  $ hg init relnotes-multiadmon
  $ cd relnotes-multiadmon

  $ touch file1
  $ hg -q commit -A -l - << EOF
  > commit 1
  > 
  > .. feature::
  > 
  >    Details about new feature.
  > 
  > .. perf::
  > 
  >    Improves the execution by 2x
  > EOF

  $ hg releasenotes -r . $TESTTMP/relnotes-multiple-admonitions
  $ cat $TESTTMP/relnotes-multiple-admonitions
  New Features
  ============
  
  * Details about new feature.
  
  Performance Improvements
  ========================
  
  * Improves the execution by 2x