view tests/test-hgweb-removed.t @ 42469:4a3abb33380a

deltas: set estimated compression upper bound to "3x" instead of "10x"

In pratice, we very rarely observer compression better than "3x" on manifest
deltas. Having a more aggressive estimate significantly helps our pathological
use case on a private repository. Here are a comparison of timings using
different upper bound.

Estimated compression |    ø   |  ×10 |  ×5  |  ×3  |
timing                |  14.11 | 2.61 | 1.96 | 1.53 |


We also tested the impact of this series on an array of public repositories.
This shown no impact in either size nor timing.

Full data set below for those interested.

Size
----

Regarding size, not significant impact have been noticed on neither public nor
private repositories. Here are the number we gathered on public repositories:

zlib/upperbound | no            | 10x           | 5x            | 3x
mercurial       |     5 875 730 |     5 875 730 |     5 875 730 |     5 875 730
pypy            |    27 782 913 |    27 782 913 |    27 782 913 |    27 782 913
netbeans        |   159 161 207 |   159 161 207 |   159 161 207 |   159 959 879 (+0.5%)
mozilla-central |   323 841 642 |   323 841 642 |   323 841 642 |   319 867 519 (-2.5%)
mozilla-try     |   746 649 123 |   746 649 123 |   746 649 123 |   741 155 568 (-0.7%)
private-repo    | 1 485 287 294 | 1 485 287 294 | 1 485 287 294 | 1 409 248 382 (-5.1%)

zstd/upperbound | no            | 10x           | 5x            | 3x
mercurial       |     5 895 206 |     5 895 206 |     5 895 206 |     5 895 206
pypy            |    28 689 230 |    28 689 230 |    28 689 230 |    28 689 230
netbeans        |   157 636 387 |   157 636 387 |   157 636 387 |   159 692 678 (+1.3%)
mozilla-central |   317 650 281 |   317 650 281 |   317 650 281 |   319 613 603 (+0.6%)
mozilla-try     |   737 555 275 |   737 555 275 |   737 555 275 |   738 079 473 (+0.1%)
private-repo    | 1 352 362 982 | 1 352 362 982 | 1 346 961 880 | 1 361 327 384 (+0.7%)


Speed
------

Timing gathered using `hg perfrevlogwrite -m`. Value are in seconds.

mercurial

zlib   | no        | 10x       | 5x        | 3x        |
total  | 65.551783 | 65.388887 | 65.260658 | 65.321199 |
max    |  0.034544 |  0.034571 |  0.034659 |  0.034521 |
99.99% |  0.034544 |  0.034571 |  0.034659 |  0.034521 |

zstd   | no        | 10x       | 5x        | 3x        |
total  | 49.118449 | 49.054062 | 48.753588 | 48.740230 |
max    |  0.009338 |  0.009239 |  0.009202 |  0.009178 |
99.99% |  0.007618 |  0.007639 |  0.007626 |  0.007621 |

pypy

zlib   | no         | 10x        | 5x         | 3x         |
total  | 560.865984 | 558.983817 | 559.083815 | 559.349152 |
max    |   0.219614 |   0.215922 |   0.218112 |   0.218107 |
99.99% |   0.219614 |   0.215922 |   0.218112 |   0.218107 |

zstd   | no         | 10x        | 5x         | 3x         |
total  | 349.393280 | 347.395819 | 347.185407 | 345.643985 |
max    |   0.084143 |   0.083536 |   0.081834 |   0.082178 |
99.99% |   0.039445 |   0.039639 |   0.039612 |   0.039175 |

netbeans
zlib   | no           | 10x          | 5x           | 3x           |
total  | 33103.327727 | 33314.932260 | 33211.745233 | 33345.891778 |
max    |     2.666852 |     2.672059 |     2.662453 |     2.662936 |
99.99% |     2.058772 |     2.070429 |     2.069569 |     2.064653 |

zstd   | no           | 10x         | 5x            | 3x           |
total  | 20112.102708 | 20095.879719 | 20083.390300 | 20123.221859 |
max    |     2.063482 |     2.062851 |     2.065229 |     2.060147 |
99.99% |     1.146647 |     1.143794 |     1.142933 |     1.146529 |

mozilla
zlib   | no           | 10x          | 5x           | 3x           |
total  | 41374.102138 | 41418.816773 | 41381.956370 | 41334.280732 |
max    |     3.383474 |     3.387400 |     3.405711 |     3.387316 |
99.99% |     1.006755 |     1.005954 |     1.007700 |     1.007373 |

zstd   | no           | 10x          | 5x           | 3x           |
total  | 24689.691520 | 24643.939662 | 24664.630027 | 24664.512714 |
max    |     1.460822 |     1.449640 |     1.439747 |     1.465304 |
99.99% |     0.527111 |     0.527377 |     0.527807 |     0.527226 |

#require serve

setting up repo

  $ hg init test
  $ cd test
  $ echo a > a
  $ hg ci -Ama
  adding a
  $ hg rm a
  $ hg ci -mdel

set up hgweb

  $ hg serve -n test -p $HGPORT -d --pid-file=hg.pid -A access.log -E errors.log
  $ cat hg.pid >> $DAEMON_PIDS

revision

  $ get-with-headers.py localhost:$HGPORT 'rev/tip'
  200 Script output follows
  
  <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
  <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en-US">
  <head>
  <link rel="icon" href="/static/hgicon.png" type="image/png" />
  <meta name="robots" content="index, nofollow" />
  <link rel="stylesheet" href="/static/style-paper.css" type="text/css" />
  <script type="text/javascript" src="/static/mercurial.js"></script>
  
  <title>test: c78f6c5cbea9</title>
  </head>
  <body>
  <div class="container">
  <div class="menu">
  <div class="logo">
  <a href="https://mercurial-scm.org/">
  <img src="/static/hglogo.png" alt="mercurial" /></a>
  </div>
  <ul>
   <li><a href="/shortlog/tip">log</a></li>
   <li><a href="/graph/tip">graph</a></li>
   <li><a href="/tags">tags</a></li>
   <li><a href="/bookmarks">bookmarks</a></li>
   <li><a href="/branches">branches</a></li>
  </ul>
  <ul>
   <li class="active">changeset</li>
   <li><a href="/raw-rev/tip">raw</a></li>
   <li><a href="/file/tip">browse</a></li>
  </ul>
  <ul>
   
  </ul>
  <ul>
   <li><a href="/help">help</a></li>
  </ul>
  </div>
  
  <div class="main">
  
  <h2 class="breadcrumb"><a href="/">Mercurial</a> </h2>
  <h3>
   changeset 1:<a href="/rev/c78f6c5cbea9">c78f6c5cbea9</a>
   <span class="phase">draft</span> <span class="branchhead">default</span> <span class="tag">tip</span> 
  </h3>
  
  
  <form class="search" action="/log">
  
  <p><input name="rev" id="search1" type="text" size="30" value="" /></p>
  <div id="hint">Find changesets by keywords (author, files, the commit message), revision
  number or hash, or <a href="/help/revsets">revset expression</a>.</div>
  </form>
  
  <div class="description">del</div>
  
  <table id="changesetEntry">
  <tr>
   <th class="author">author</th>
   <td class="author">&#116;&#101;&#115;&#116;</td>
  </tr>
  <tr>
   <th class="date">date</th>
   <td class="date age">Thu, 01 Jan 1970 00:00:00 +0000</td>
  </tr>
  
  
  <tr>
   <th class="author">parents</th>
   <td class="author"><a href="/rev/cb9a9f314b8b">cb9a9f314b8b</a> </td>
  </tr>
  <tr>
   <th class="author">children</th>
   <td class="author"></td>
  </tr>
  <tr>
   <th class="files">files</th>
   <td class="files">a </td>
  </tr>
  <tr>
    <th class="diffstat">diffstat</th>
    <td class="diffstat">
       1 files changed, 0 insertions(+), 1 deletions(-)
  
      <a id="diffstatexpand" class="diffstattoggle" href="#">[<tt>+</tt>]</a>
      <div id="diffstatdetails" style="display:none;">
        <a class="diffstattoggle" href="#">[<tt>-</tt>]</a>
        <table class="diffstat-table stripes2">  <tr>
      <td class="diffstat-file"><a href="#l1.1">a</a></td>
      <td class="diffstat-total" align="right">1</td>
      <td class="diffstat-graph">
        <span class="diffstat-add" style="width:0.0%;">&nbsp;</span>
        <span class="diffstat-remove" style="width:100.0%;">&nbsp;</span>
      </td>
    </tr>
  </table>
      </div>
    </td>
  </tr>
  </table>
  
  <div class="overflow">
  <div class="sourcefirst linewraptoggle">line wrap: <a class="linewraplink" href="#">on</a></div>
  <div class="sourcefirst"> line diff</div>
  <div class="stripes2 diffblocks">
  <div class="bottomline inc-lineno"><pre class="sourcelines wrap">
  <span id="l1.1" class="minusline">--- a/a	Thu Jan 01 00:00:00 1970 +0000</span><a href="#l1.1"></a>
  <span id="l1.2" class="plusline">+++ /dev/null	Thu Jan 01 00:00:00 1970 +0000</span><a href="#l1.2"></a>
  <span id="l1.3" class="atline">@@ -1,1 +0,0 @@</span><a href="#l1.3"></a>
  <span id="l1.4" class="minusline">-a</span><a href="#l1.4"></a></pre></div>
  </div>
  </div>
  
  </div>
  </div>
  
  
  </body>
  </html>
  

diff removed file

  $ get-with-headers.py localhost:$HGPORT 'diff/tip/a'
  200 Script output follows
  
  <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
  <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en-US">
  <head>
  <link rel="icon" href="/static/hgicon.png" type="image/png" />
  <meta name="robots" content="index, nofollow" />
  <link rel="stylesheet" href="/static/style-paper.css" type="text/css" />
  <script type="text/javascript" src="/static/mercurial.js"></script>
  
  <title>test: a diff</title>
  </head>
  <body>
  
  <div class="container">
  <div class="menu">
  <div class="logo">
  <a href="https://mercurial-scm.org/">
  <img src="/static/hglogo.png" alt="mercurial" /></a>
  </div>
  <ul>
  <li><a href="/shortlog/tip">log</a></li>
  <li><a href="/graph/tip">graph</a></li>
  <li><a href="/tags">tags</a></li>
  <li><a href="/bookmarks">bookmarks</a></li>
  <li><a href="/branches">branches</a></li>
  </ul>
  <ul>
  <li><a href="/rev/tip">changeset</a></li>
  <li><a href="/file/tip">browse</a></li>
  </ul>
  <ul>
  <li><a href="/file/tip/a">file</a></li>
  <li><a href="/file/tip/a">latest</a></li>
  <li class="active">diff</li>
  <li><a href="/comparison/tip/a">comparison</a></li>
  <li><a href="/annotate/tip/a">annotate</a></li>
  <li><a href="/log/tip/a">file log</a></li>
  <li><a href="/raw-file/tip/a">raw</a></li>
  </ul>
  <ul>
  <li><a href="/help">help</a></li>
  </ul>
  </div>
  
  <div class="main">
  <h2 class="breadcrumb"><a href="/">Mercurial</a> </h2>
  <h3>
   diff a @ 1:<a href="/rev/c78f6c5cbea9">c78f6c5cbea9</a>
   <span class="phase">draft</span> <span class="branchhead">default</span> <span class="tag">tip</span> 
  </h3>
  
  
  <form class="search" action="/log">
  
  <p><input name="rev" id="search1" type="text" size="30" value="" /></p>
  <div id="hint">Find changesets by keywords (author, files, the commit message), revision
  number or hash, or <a href="/help/revsets">revset expression</a>.</div>
  </form>
  
  <div class="description">del</div>
  
  <table id="changesetEntry">
  <tr>
   <th>author</th>
   <td>&#116;&#101;&#115;&#116;</td>
  </tr>
  <tr>
   <th>date</th>
   <td class="date age">Thu, 01 Jan 1970 00:00:00 +0000</td>
  </tr>
  <tr>
   <th>parents</th>
   <td><a href="/file/cb9a9f314b8b/a">cb9a9f314b8b</a> </td>
  </tr>
  <tr>
   <th>children</th>
   <td></td>
  </tr>
  </table>
  
  <div class="overflow">
  <div class="sourcefirst linewraptoggle">line wrap: <a class="linewraplink" href="#">on</a></div>
  <div class="sourcefirst"> line diff</div>
  <div class="stripes2 diffblocks">
  <div class="bottomline inc-lineno"><pre class="sourcelines wrap">
  <span id="l1.1" class="minusline">--- a/a	Thu Jan 01 00:00:00 1970 +0000</span><a href="#l1.1"></a>
  <span id="l1.2" class="plusline">+++ /dev/null	Thu Jan 01 00:00:00 1970 +0000</span><a href="#l1.2"></a>
  <span id="l1.3" class="atline">@@ -1,1 +0,0 @@</span><a href="#l1.3"></a>
  <span id="l1.4" class="minusline">-a</span><a href="#l1.4"></a></pre></div>
  </div>
  </div>
  </div>
  </div>
  
  
  
  </body>
  </html>
  

  $ cd ..