Mercurial > hg
view mercurial/minirst.py @ 39506:b66ea3fc3a86
sparse-revlog: set max delta chain length to on thousand
The new snapshot system used in the sparse-revlog case gave us some small size
benefit so far. However its most important property is to gracefully handle
harder limit on delta chainlength.
Long delta chain has a very detrimental impact on read (and write) performance
in revlog. Being able to shorter them provide a great boost. However, shorting
delta used to result significantly lower compression ratio. The intermediate
snapshots effectively suppress most of this effect (even all in some case).
# Effect on the test repository
The repository we use for test is not "realistic" but can still show this in
action using an unreasonably low chain limit. Limiting the chain length show a
sizeable increase but stay under control: +6% for limit=15; +15% for limit=10.
Without the snapshot system the increase is significantly bigger: +45% for
limit=15; +80% for limit=10. Even slightly larger than without delta chain
limit, the resulting size is still smaller than before we started doing
snapshots.
Here is a table for comparison. *Since the repository is not branchy, the
initial sparse-revlog version does not bring much benefit compare to the
non-sparse one):
chain length limit | none | limit=15 | limit=10 |
without sparse-revlog | 62 818 987 | 112 664 615 | 131 222 574 |
without snapshot | 74 365 490 | 108 211 410 | 133 857 764 |
with snapshot | 59 230 936 | 63 002 924 | 68 415 329 |
# Effect On Real Life Repositories
The series provides significant benefits on all kind of repositories.
Using `hg debugupgraderepo -o redeltaparent --run`, we recomputed delta chain
for various repositories with different settings:
- delta chain length: unlimited or 1000 limit
- sparse-revlog: enabled or disabled
- this series: applied or not applied
We can observe multiple types of effect:
- On very branchy repositories:
* The delta chain limit as low impact on the repo size.
* Intermediate snapshot greatly reduces manifest size:
- pypy: -80%
- netbeans: -95%
* The delta chain limit is effective, without a size impact:
- netbeans average: 613 -> 282
- private #1 average: 1 068 -> 307
- On more linear repository:
* Intermediate snapshot limit the impact of delta chain limit:
- mozilla:
without the series: +360%
with the series: +25%
* The delta chain limit provides large improvement:
- mozilla's average chain length:
unlimited: 15 338
limited: 469
* Despite the chain length limit, the manifest size is reduced:
- mercurial: -25%
- mozilla: -30%
It is clear that the use of chains of intermediate snapshots provide large
benefits both in storage size and delta chains quality. We should now switch our
effort toward making sure the write performance are acceptable. Then,
`sparse-revlog` will be a suitable format for all new repository.
# Raw Statistic
* no-sparse: general delta repository not using sparse-revlog
* no-snapshot: sparse-revlog repository not using this series
* snapshot: sparse-revlog repository using this series
mercurial
Manifest Size:
limit | none | 1000
------------|-------------|------------
no-sparse | 8 021 373 | 8 199 366
no-snapshot | 8 103 561 | 8 259 719
snapshot | 6 137 116 | 6 126 433
Manifest Chain length data
limit || none || 1000 ||
value || average | max || average | max ||
------------||---------|---------||---------|---------||
no-sparse || 307 | 1456 || 279 | 1000 ||
no-snapshot || 312 | 1456 || 283 | 1000 ||
snapshot || 248 | 1208 || 241 | 1000 ||
Full Store Size
limit | none | 1000
------------|-------------|------------
no-sparse | 51 013 198 | 51 201 574
no-snapshot | 50 930 795 | 51 141 006
snapshot | 48 072 037 | 48 093 572
pypy
Manifest Size:
limit | none | 1000
------------|-------------|------------
no-sparse | 193 987 784 | 193 987 784
no-snapshot | 163 171 745 | 163 312 229
snapshot | 34 605 900 | 34 600 750
Manifest Chain length data
limit || none || 1000 ||
value || average | max || average | max ||
------------||---------|---------||---------|---------||
no-sparse || 101 | 692 || 101 | 692 ||
no-snapshot || 151 | 1307 || 148 | 1000 ||
snapshot || 128 | 1309 || 125 | 1000 ||
Full Store Size
limit | none | 1000
------------|-------------|------------
no-sparse | 495 931 473 | 495 931 473
no-snapshot | 465 441 017 | 465 581 501
snapshot | 355 467 301 | 355 472 451
Mozilla
Manifest Size:
limit | none | 1000
------------|----------------|---------------
no-sparse | 416 757 148 | 1 869 009 668
no-snapshot | 401 592 370 | 1 843 493 795
snapshot | 224 359 521 | 284 615 500
Manifest Chain length data
limit || none || 1000 ||
value || average | max || average | max ||
------------||---------|---------||---------|---------||
no-sparse || 15 333 | 58 980 || 468 | 1 000 ||
no-snapshot || 15 336 | 58 980 || 469 | 1 000 ||
snapshot || 15 338 | 58 983 || 469 | 1 000 ||
Full Store Size
limit | none | 1000
------------|----------------|---------------
no-sparse | 2 712 477 887 | 4 164 995 451
no-snapshot | 2 698 887 835 | 4 141 054 304
snapshot | 2 518 130 385 | 2 578 587 596
Netbeans
Manifest Size:
limit | none | 1000
------------|----------------|---------------
no-sparse | 4 766 794 101 | 4 870 642 687
no-snapshot | 4 334 806 082 | 4 428 681 309
snapshot | 232 659 666 | 240 330 665
Manifest Chain length data
limit || none || 1000 ||
value || average | max || average | max ||
------------||---------|---------||---------|---------||
no-sparse || 597 | 6802 || 254 | 1 000 ||
no-snapshot || 648 | 6 802 || 305 | 1 000 ||
snapshot || 613 | 6 804 || 282 | 1 000 ||
Full Store Size
limit | none | 1000
------------|----------------|---------------
no-sparse | 5 807 347 998 | 5 911 196 584
no-snapshot | 5 375 398 602 | 5 469 273 829
snapshot | 1 282 519 928 | 1 290 190 927
Private repo #1
Manifest Size:
limit | none | 1000
------------|-----------------|---------------
no-sparse | 41 389 010 840 | 41 398 162 091
no-snapshot | 9 737 319 435 | 10 223 773 150
snapshot | 744 215 807 | 747 961 822
Manifest Chain length data
limit || none || 1000 ||
value || average | max || average | max ||
------------||---------|---------||---------|---------||
no-sparse || 245 | 8 885 || 81 | 1 000 ||
no-snapshot || 1 225 | 8 885 || 336 | 1 000 ||
snapshot || 1 068 | 7 909 || 307 | 1 000 ||
Full Store Size
limit | none | 1000
------------|----------------|---------------
no-sparse | 49 646 065 126 | 49 655 216 377
no-snapshot | 17 924 862 856 | 18 411 316 571
snapshot | 9 009 024 710 | 9 012 770 725
Private repo #2
We currently have less data available for that repository.
* Before is a sparse-revlog repository without this series
* After is a sparse-revlog repository with this series + 1000 chain limit
Manifest Size:
Before: 1 531 485 040 bytes
After: 1 091 422 451 bytes
Manifest Chain:
Before: 2 218 avg; 6 575 Max
After: 442 avg; 1 000 Max
Full Store Size
Before: 15 203 955 615
after: 8 207 180 693
author | Boris Feld <boris.feld@octobus.net> |
---|---|
date | Fri, 07 Sep 2018 11:18:45 -0400 |
parents | ca2f4dabf51d |
children | 876494fd967d |
line wrap: on
line source
# minirst.py - minimal reStructuredText parser # # Copyright 2009, 2010 Matt Mackall <mpm@selenic.com> and others # # This software may be used and distributed according to the terms of the # GNU General Public License version 2 or any later version. """simplified reStructuredText parser. This parser knows just enough about reStructuredText to parse the Mercurial docstrings. It cheats in a major way: nested blocks are not really nested. They are just indented blocks that look like they are nested. This relies on the user to keep the right indentation for the blocks. Remember to update https://mercurial-scm.org/wiki/HelpStyleGuide when adding support for new constructs. """ from __future__ import absolute_import import re from .i18n import _ from . import ( encoding, pycompat, url, ) from .utils import ( stringutil, ) def section(s): return "%s\n%s\n\n" % (s, "\"" * encoding.colwidth(s)) def subsection(s): return "%s\n%s\n\n" % (s, '=' * encoding.colwidth(s)) def subsubsection(s): return "%s\n%s\n\n" % (s, "-" * encoding.colwidth(s)) def subsubsubsection(s): return "%s\n%s\n\n" % (s, "." * encoding.colwidth(s)) def replace(text, substs): ''' Apply a list of (find, replace) pairs to a text. >>> replace(b"foo bar", [(b'f', b'F'), (b'b', b'B')]) 'Foo Bar' >>> encoding.encoding = b'latin1' >>> replace(b'\\x81\\\\', [(b'\\\\', b'/')]) '\\x81/' >>> encoding.encoding = b'shiftjis' >>> replace(b'\\x81\\\\', [(b'\\\\', b'/')]) '\\x81\\\\' ''' # some character encodings (cp932 for Japanese, at least) use # ASCII characters other than control/alphabet/digit as a part of # multi-bytes characters, so direct replacing with such characters # on strings in local encoding causes invalid byte sequences. utext = text.decode(pycompat.sysstr(encoding.encoding)) for f, t in substs: utext = utext.replace(f.decode("ascii"), t.decode("ascii")) return utext.encode(pycompat.sysstr(encoding.encoding)) _blockre = re.compile(br"\n(?:\s*\n)+") def findblocks(text): """Find continuous blocks of lines in text. Returns a list of dictionaries representing the blocks. Each block has an 'indent' field and a 'lines' field. """ blocks = [] for b in _blockre.split(text.lstrip('\n').rstrip()): lines = b.splitlines() if lines: indent = min((len(l) - len(l.lstrip())) for l in lines) lines = [l[indent:] for l in lines] blocks.append({'indent': indent, 'lines': lines}) return blocks def findliteralblocks(blocks): """Finds literal blocks and adds a 'type' field to the blocks. Literal blocks are given the type 'literal', all other blocks are given type the 'paragraph'. """ i = 0 while i < len(blocks): # Searching for a block that looks like this: # # +------------------------------+ # | paragraph | # | (ends with "::") | # +------------------------------+ # +---------------------------+ # | indented literal block | # +---------------------------+ blocks[i]['type'] = 'paragraph' if blocks[i]['lines'][-1].endswith('::') and i + 1 < len(blocks): indent = blocks[i]['indent'] adjustment = blocks[i + 1]['indent'] - indent if blocks[i]['lines'] == ['::']: # Expanded form: remove block del blocks[i] i -= 1 elif blocks[i]['lines'][-1].endswith(' ::'): # Partially minimized form: remove space and both # colons. blocks[i]['lines'][-1] = blocks[i]['lines'][-1][:-3] elif len(blocks[i]['lines']) == 1 and \ blocks[i]['lines'][0].lstrip(' ').startswith('.. ') and \ blocks[i]['lines'][0].find(' ', 3) == -1: # directive on its own line, not a literal block i += 1 continue else: # Fully minimized form: remove just one colon. blocks[i]['lines'][-1] = blocks[i]['lines'][-1][:-1] # List items are formatted with a hanging indent. We must # correct for this here while we still have the original # information on the indentation of the subsequent literal # blocks available. m = _bulletre.match(blocks[i]['lines'][0]) if m: indent += m.end() adjustment -= m.end() # Mark the following indented blocks. while i + 1 < len(blocks) and blocks[i + 1]['indent'] > indent: blocks[i + 1]['type'] = 'literal' blocks[i + 1]['indent'] -= adjustment i += 1 i += 1 return blocks _bulletre = re.compile(br'(\*|-|[0-9A-Za-z]+\.|\(?[0-9A-Za-z]+\)|\|) ') _optionre = re.compile(br'^(-([a-zA-Z0-9]), )?(--[a-z0-9-]+)' br'((.*) +)(.*)$') _fieldre = re.compile(br':(?![: ])([^:]*)(?<! ):[ ]+(.*)') _definitionre = re.compile(br'[^ ]') _tablere = re.compile(br'(=+\s+)*=+') def splitparagraphs(blocks): """Split paragraphs into lists.""" # Tuples with (list type, item regexp, single line items?). Order # matters: definition lists has the least specific regexp and must # come last. listtypes = [('bullet', _bulletre, True), ('option', _optionre, True), ('field', _fieldre, True), ('definition', _definitionre, False)] def match(lines, i, itemre, singleline): """Does itemre match an item at line i? A list item can be followed by an indented line or another list item (but only if singleline is True). """ line1 = lines[i] line2 = i + 1 < len(lines) and lines[i + 1] or '' if not itemre.match(line1): return False if singleline: return line2 == '' or line2[0:1] == ' ' or itemre.match(line2) else: return line2.startswith(' ') i = 0 while i < len(blocks): if blocks[i]['type'] == 'paragraph': lines = blocks[i]['lines'] for type, itemre, singleline in listtypes: if match(lines, 0, itemre, singleline): items = [] for j, line in enumerate(lines): if match(lines, j, itemre, singleline): items.append({'type': type, 'lines': [], 'indent': blocks[i]['indent']}) items[-1]['lines'].append(line) blocks[i:i + 1] = items break i += 1 return blocks _fieldwidth = 14 def updatefieldlists(blocks): """Find key for field lists.""" i = 0 while i < len(blocks): if blocks[i]['type'] != 'field': i += 1 continue j = i while j < len(blocks) and blocks[j]['type'] == 'field': m = _fieldre.match(blocks[j]['lines'][0]) key, rest = m.groups() blocks[j]['lines'][0] = rest blocks[j]['key'] = key j += 1 i = j + 1 return blocks def updateoptionlists(blocks): i = 0 while i < len(blocks): if blocks[i]['type'] != 'option': i += 1 continue optstrwidth = 0 j = i while j < len(blocks) and blocks[j]['type'] == 'option': m = _optionre.match(blocks[j]['lines'][0]) shortoption = m.group(2) group3 = m.group(3) longoption = group3[2:].strip() desc = m.group(6).strip() longoptionarg = m.group(5).strip() blocks[j]['lines'][0] = desc noshortop = '' if not shortoption: noshortop = ' ' opt = "%s%s" % (shortoption and "-%s " % shortoption or '', ("%s--%s %s") % (noshortop, longoption, longoptionarg)) opt = opt.rstrip() blocks[j]['optstr'] = opt optstrwidth = max(optstrwidth, encoding.colwidth(opt)) j += 1 for block in blocks[i:j]: block['optstrwidth'] = optstrwidth i = j + 1 return blocks def prunecontainers(blocks, keep): """Prune unwanted containers. The blocks must have a 'type' field, i.e., they should have been run through findliteralblocks first. """ pruned = [] i = 0 while i + 1 < len(blocks): # Searching for a block that looks like this: # # +-------+---------------------------+ # | ".. container ::" type | # +---+ | # | blocks | # +-------------------------------+ if (blocks[i]['type'] == 'paragraph' and blocks[i]['lines'][0].startswith('.. container::')): indent = blocks[i]['indent'] adjustment = blocks[i + 1]['indent'] - indent containertype = blocks[i]['lines'][0][15:] prune = True for c in keep: if c in containertype.split('.'): prune = False if prune: pruned.append(containertype) # Always delete "..container:: type" block del blocks[i] j = i i -= 1 while j < len(blocks) and blocks[j]['indent'] > indent: if prune: del blocks[j] else: blocks[j]['indent'] -= adjustment j += 1 i += 1 return blocks, pruned _sectionre = re.compile(br"""^([-=`:.'"~^_*+#])\1+$""") def findtables(blocks): '''Find simple tables Only simple one-line table elements are supported ''' for block in blocks: # Searching for a block that looks like this: # # === ==== === # A B C # === ==== === <- optional # 1 2 3 # x y z # === ==== === if (block['type'] == 'paragraph' and len(block['lines']) > 2 and _tablere.match(block['lines'][0]) and block['lines'][0] == block['lines'][-1]): block['type'] = 'table' block['header'] = False div = block['lines'][0] # column markers are ASCII so we can calculate column # position in bytes columns = [x for x in pycompat.xrange(len(div)) if div[x:x + 1] == '=' and (x == 0 or div[x - 1:x] == ' ')] rows = [] for l in block['lines'][1:-1]: if l == div: block['header'] = True continue row = [] # we measure columns not in bytes or characters but in # colwidth which makes things tricky pos = columns[0] # leading whitespace is bytes for n, start in enumerate(columns): if n + 1 < len(columns): width = columns[n + 1] - start v = encoding.getcols(l, pos, width) # gather columns pos += len(v) # calculate byte position of end row.append(v.strip()) else: row.append(l[pos:].strip()) rows.append(row) block['table'] = rows return blocks def findsections(blocks): """Finds sections. The blocks must have a 'type' field, i.e., they should have been run through findliteralblocks first. """ for block in blocks: # Searching for a block that looks like this: # # +------------------------------+ # | Section title | # | ------------- | # +------------------------------+ if (block['type'] == 'paragraph' and len(block['lines']) == 2 and encoding.colwidth(block['lines'][0]) == len(block['lines'][1]) and _sectionre.match(block['lines'][1])): block['underline'] = block['lines'][1][0:1] block['type'] = 'section' del block['lines'][1] return blocks def inlineliterals(blocks): substs = [('``', '"')] for b in blocks: if b['type'] in ('paragraph', 'section'): b['lines'] = [replace(l, substs) for l in b['lines']] return blocks def hgrole(blocks): substs = [(':hg:`', "'hg "), ('`', "'")] for b in blocks: if b['type'] in ('paragraph', 'section'): # Turn :hg:`command` into "hg command". This also works # when there is a line break in the command and relies on # the fact that we have no stray back-quotes in the input # (run the blocks through inlineliterals first). b['lines'] = [replace(l, substs) for l in b['lines']] return blocks def addmargins(blocks): """Adds empty blocks for vertical spacing. This groups bullets, options, and definitions together with no vertical space between them, and adds an empty block between all other blocks. """ i = 1 while i < len(blocks): if (blocks[i]['type'] == blocks[i - 1]['type'] and blocks[i]['type'] in ('bullet', 'option', 'field')): i += 1 elif not blocks[i - 1]['lines']: # no lines in previous block, do not separate i += 1 else: blocks.insert(i, {'lines': [''], 'indent': 0, 'type': 'margin'}) i += 2 return blocks def prunecomments(blocks): """Remove comments.""" i = 0 while i < len(blocks): b = blocks[i] if b['type'] == 'paragraph' and (b['lines'][0].startswith('.. ') or b['lines'] == ['..']): del blocks[i] if i < len(blocks) and blocks[i]['type'] == 'margin': del blocks[i] else: i += 1 return blocks def findadmonitions(blocks, admonitions=None): """ Makes the type of the block an admonition block if the first line is an admonition directive """ admonitions = admonitions or _admonitiontitles.keys() admonitionre = re.compile(br'\.\. (%s)::' % '|'.join(sorted(admonitions)), flags=re.IGNORECASE) i = 0 while i < len(blocks): m = admonitionre.match(blocks[i]['lines'][0]) if m: blocks[i]['type'] = 'admonition' admonitiontitle = blocks[i]['lines'][0][3:m.end() - 2].lower() firstline = blocks[i]['lines'][0][m.end() + 1:] if firstline: blocks[i]['lines'].insert(1, ' ' + firstline) blocks[i]['admonitiontitle'] = admonitiontitle del blocks[i]['lines'][0] i = i + 1 return blocks _admonitiontitles = { 'attention': _('Attention:'), 'caution': _('Caution:'), 'danger': _('!Danger!'), 'error': _('Error:'), 'hint': _('Hint:'), 'important': _('Important:'), 'note': _('Note:'), 'tip': _('Tip:'), 'warning': _('Warning!'), } def formatoption(block, width): desc = ' '.join(map(bytes.strip, block['lines'])) colwidth = encoding.colwidth(block['optstr']) usablewidth = width - 1 hanging = block['optstrwidth'] initindent = '%s%s ' % (block['optstr'], ' ' * ((hanging - colwidth))) hangindent = ' ' * (encoding.colwidth(initindent) + 1) return ' %s\n' % (stringutil.wrap(desc, usablewidth, initindent=initindent, hangindent=hangindent)) def formatblock(block, width): """Format a block according to width.""" if width <= 0: width = 78 indent = ' ' * block['indent'] if block['type'] == 'admonition': admonition = _admonitiontitles[block['admonitiontitle']] if not block['lines']: return indent + admonition + '\n' hang = len(block['lines'][-1]) - len(block['lines'][-1].lstrip()) defindent = indent + hang * ' ' text = ' '.join(map(bytes.strip, block['lines'])) return '%s\n%s\n' % (indent + admonition, stringutil.wrap(text, width=width, initindent=defindent, hangindent=defindent)) if block['type'] == 'margin': return '\n' if block['type'] == 'literal': indent += ' ' return indent + ('\n' + indent).join(block['lines']) + '\n' if block['type'] == 'section': underline = encoding.colwidth(block['lines'][0]) * block['underline'] return "%s%s\n%s%s\n" % (indent, block['lines'][0],indent, underline) if block['type'] == 'table': table = block['table'] # compute column widths widths = [max([encoding.colwidth(e) for e in c]) for c in zip(*table)] text = '' span = sum(widths) + len(widths) - 1 indent = ' ' * block['indent'] hang = ' ' * (len(indent) + span - widths[-1]) for row in table: l = [] for w, v in zip(widths, row): pad = ' ' * (w - encoding.colwidth(v)) l.append(v + pad) l = ' '.join(l) l = stringutil.wrap(l, width=width, initindent=indent, hangindent=hang) if not text and block['header']: text = l + '\n' + indent + '-' * (min(width, span)) + '\n' else: text += l + "\n" return text if block['type'] == 'definition': term = indent + block['lines'][0] hang = len(block['lines'][-1]) - len(block['lines'][-1].lstrip()) defindent = indent + hang * ' ' text = ' '.join(map(bytes.strip, block['lines'][1:])) return '%s\n%s\n' % (term, stringutil.wrap(text, width=width, initindent=defindent, hangindent=defindent)) subindent = indent if block['type'] == 'bullet': if block['lines'][0].startswith('| '): # Remove bullet for line blocks and add no extra # indentation. block['lines'][0] = block['lines'][0][2:] else: m = _bulletre.match(block['lines'][0]) subindent = indent + m.end() * ' ' elif block['type'] == 'field': key = block['key'] subindent = indent + _fieldwidth * ' ' if len(key) + 2 > _fieldwidth: # key too large, use full line width key = key.ljust(width) else: # key fits within field width key = key.ljust(_fieldwidth) block['lines'][0] = key + block['lines'][0] elif block['type'] == 'option': return formatoption(block, width) text = ' '.join(map(bytes.strip, block['lines'])) return stringutil.wrap(text, width=width, initindent=indent, hangindent=subindent) + '\n' def formathtml(blocks): """Format RST blocks as HTML""" out = [] headernest = '' listnest = [] def escape(s): return url.escape(s, True) def openlist(start, level): if not listnest or listnest[-1][0] != start: listnest.append((start, level)) out.append('<%s>\n' % start) blocks = [b for b in blocks if b['type'] != 'margin'] for pos, b in enumerate(blocks): btype = b['type'] level = b['indent'] lines = b['lines'] if btype == 'admonition': admonition = escape(_admonitiontitles[b['admonitiontitle']]) text = escape(' '.join(map(bytes.strip, lines))) out.append('<p>\n<b>%s</b> %s\n</p>\n' % (admonition, text)) elif btype == 'paragraph': out.append('<p>\n%s\n</p>\n' % escape('\n'.join(lines))) elif btype == 'margin': pass elif btype == 'literal': out.append('<pre>\n%s\n</pre>\n' % escape('\n'.join(lines))) elif btype == 'section': i = b['underline'] if i not in headernest: headernest += i level = headernest.index(i) + 1 out.append('<h%d>%s</h%d>\n' % (level, escape(lines[0]), level)) elif btype == 'table': table = b['table'] out.append('<table>\n') for row in table: out.append('<tr>') for v in row: out.append('<td>') out.append(escape(v)) out.append('</td>') out.append('\n') out.pop() out.append('</tr>\n') out.append('</table>\n') elif btype == 'definition': openlist('dl', level) term = escape(lines[0]) text = escape(' '.join(map(bytes.strip, lines[1:]))) out.append(' <dt>%s\n <dd>%s\n' % (term, text)) elif btype == 'bullet': bullet, head = lines[0].split(' ', 1) if bullet in ('*', '-'): openlist('ul', level) else: openlist('ol', level) out.append(' <li> %s\n' % escape(' '.join([head] + lines[1:]))) elif btype == 'field': openlist('dl', level) key = escape(b['key']) text = escape(' '.join(map(bytes.strip, lines))) out.append(' <dt>%s\n <dd>%s\n' % (key, text)) elif btype == 'option': openlist('dl', level) opt = escape(b['optstr']) desc = escape(' '.join(map(bytes.strip, lines))) out.append(' <dt>%s\n <dd>%s\n' % (opt, desc)) # close lists if indent level of next block is lower if listnest: start, level = listnest[-1] if pos == len(blocks) - 1: out.append('</%s>\n' % start) listnest.pop() else: nb = blocks[pos + 1] ni = nb['indent'] if (ni < level or (ni == level and nb['type'] not in 'definition bullet field option')): out.append('</%s>\n' % start) listnest.pop() return ''.join(out) def parse(text, indent=0, keep=None, admonitions=None): """Parse text into a list of blocks""" pruned = [] blocks = findblocks(text) for b in blocks: b['indent'] += indent blocks = findliteralblocks(blocks) blocks = findtables(blocks) blocks, pruned = prunecontainers(blocks, keep or []) blocks = findsections(blocks) blocks = inlineliterals(blocks) blocks = hgrole(blocks) blocks = splitparagraphs(blocks) blocks = updatefieldlists(blocks) blocks = updateoptionlists(blocks) blocks = findadmonitions(blocks, admonitions=admonitions) blocks = addmargins(blocks) blocks = prunecomments(blocks) return blocks, pruned def formatblocks(blocks, width): text = ''.join(formatblock(b, width) for b in blocks) return text def formatplain(blocks, width): """Format parsed blocks as plain text""" return ''.join(formatblock(b, width) for b in blocks) def format(text, width=80, indent=0, keep=None, style='plain', section=None): """Parse and format the text according to width.""" blocks, pruned = parse(text, indent, keep or []) if section: blocks = filtersections(blocks, section) if style == 'html': return formathtml(blocks) else: return formatplain(blocks, width=width) def filtersections(blocks, section): """Select parsed blocks under the specified section The section name is separated by a dot, and matches the suffix of the full section path. """ parents = [] sections = _getsections(blocks) blocks = [] i = 0 lastparents = [] synthetic = [] collapse = True while i < len(sections): path, nest, b = sections[i] del parents[nest:] parents.append(i) if path == section or path.endswith('.' + section): if lastparents != parents: llen = len(lastparents) plen = len(parents) if llen and llen != plen: collapse = False s = [] for j in pycompat.xrange(3, plen - 1): parent = parents[j] if (j >= llen or lastparents[j] != parent): s.append(len(blocks)) sec = sections[parent][2] blocks.append(sec[0]) blocks.append(sec[-1]) if s: synthetic.append(s) lastparents = parents[:] blocks.extend(b) ## Also show all subnested sections while i + 1 < len(sections) and sections[i + 1][1] > nest: i += 1 blocks.extend(sections[i][2]) i += 1 if collapse: synthetic.reverse() for s in synthetic: path = [blocks[syn]['lines'][0] for syn in s] real = s[-1] + 2 realline = blocks[real]['lines'] realline[0] = ('"%s"' % '.'.join(path + [realline[0]]).replace('"', '')) del blocks[s[0]:real] return blocks def _getsections(blocks): '''return a list of (section path, nesting level, blocks) tuples''' nest = "" names = () level = 0 secs = [] def getname(b): if b['type'] == 'field': x = b['key'] else: x = b['lines'][0] x = encoding.lower(x).strip('"') if '(' in x: x = x.split('(')[0] return x for b in blocks: if b['type'] == 'section': i = b['underline'] if i not in nest: nest += i level = nest.index(i) + 1 nest = nest[:level] names = names[:level] + (getname(b),) secs.append(('.'.join(names), level, [b])) elif b['type'] in ('definition', 'field'): i = ' ' if i not in nest: nest += i level = nest.index(i) + 1 nest = nest[:level] for i in range(1, len(secs) + 1): sec = secs[-i] if sec[1] < level: break siblings = [a for a in sec[2] if a['type'] == 'definition'] if siblings: siblingindent = siblings[-1]['indent'] indent = b['indent'] if siblingindent < indent: level += 1 break elif siblingindent == indent: level = sec[1] break names = names[:level] + (getname(b),) secs.append(('.'.join(names), level, [b])) else: if not secs: # add an initial empty section secs = [('', 0, [])] if b['type'] != 'margin': pointer = 1 bindent = b['indent'] while pointer < len(secs): section = secs[-pointer][2][0] if section['type'] != 'margin': sindent = section['indent'] if len(section['lines']) > 1: sindent += len(section['lines'][1]) - \ len(section['lines'][1].lstrip(' ')) if bindent >= sindent: break pointer += 1 if pointer > 1: blevel = secs[-pointer][1] if section['type'] != b['type']: blevel += 1 secs.append(('', blevel, [])) secs[-1][2].append(b) return secs def maketable(data, indent=0, header=False): '''Generate an RST table for the given table data as a list of lines''' widths = [max(encoding.colwidth(e) for e in c) for c in zip(*data)] indent = ' ' * indent div = indent + ' '.join('=' * w for w in widths) + '\n' out = [div] for row in data: l = [] for w, v in zip(widths, row): if '\n' in v: # only remove line breaks and indentation, long lines are # handled by the next tool v = ' '.join(e.lstrip() for e in v.split('\n')) pad = ' ' * (w - encoding.colwidth(v)) l.append(v + pad) out.append(indent + ' '.join(l) + "\n") if header and len(data) > 1: out.insert(2, div) out.append(div) return out