mercurial/revlogutils/sidedata.py

resourceutil: don't limit resources to the `mercurial` package

This should make things a little clearer, in that it now requires the full
package name to access a resource.  But the real motivation is that
`extensions._disabledpaths()` walks the `hgext` directory looking for bundled
extensions.  This in turn feeds, among other things:

    1) Listing disabled extensions in `hg help extensions`
    2) Indicating that an unknown command is in a non-enabled extension
    3) Displaying help for non-enabled extensions
    4) Generating documentation
    5) Announcing LFS is auto-enabled (or not) when cloning from an LFS source

The filesystem based ResourceReader will happily return *.py and *.pyc, but the
one supplied by PyOxidizer doesn't.  Presumably we can change that.  The only
other idea I had here is for setup.py to generate a text file containing the
list of extensions, but that doesn't seem great when running from source.

Differential Revision: https://phab.mercurial-scm.org/D7772

# sidedata.py - Logic around store extra data alongside revlog revisions
#
# Copyright 2019 Pierre-Yves David <[e-mail])
#
# This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version.
"""core code for "sidedata" support

The "sidedata" are stored alongside the revision without actually being part of
its content and not affecting its hash. It's main use cases is to cache
important information related to a changesets.

The current implementation is experimental and subject to changes. Do not rely
on it in production.

Sidedata are stored in the revlog itself, withing the revision rawtext. They
are inserted, removed from it using the flagprocessors mechanism. The following
format is currently used::

    initial header:
        <number of sidedata; 2 bytes>
    sidedata (repeated N times):
        <sidedata-key; 2 bytes>
        <sidedata-entry-length: 4 bytes>
        <sidedata-content-sha1-digest: 20 bytes>
        <sidedata-content; X bytes>
    normal raw text:
        <all bytes remaining in the rawtext>

This is a simple and effective format. It should be enought to experiment with
the concept.
"""

from __future__ import absolute_import

import hashlib
import struct

from .. import error

## sidedata type constant
# reserve a block for testing purposes.
SD_TEST1 = 1
SD_TEST2 = 2
SD_TEST3 = 3
SD_TEST4 = 4
SD_TEST5 = 5
SD_TEST6 = 6
SD_TEST7 = 7

# key to store copies related information
SD_P1COPIES = 8
SD_P2COPIES = 9
SD_FILESADDED = 10
SD_FILESREMOVED = 11

# internal format constant
SIDEDATA_HEADER = struct.Struct('>H')
SIDEDATA_ENTRY = struct.Struct('>HL20s')


def sidedatawriteprocessor(rl, text, sidedata):
    sidedata = list(sidedata.items())
    sidedata.sort()
    rawtext = [SIDEDATA_HEADER.pack(len(sidedata))]
    for key, value in sidedata:
        digest = hashlib.sha1(value).digest()
        rawtext.append(SIDEDATA_ENTRY.pack(key, len(value), digest))
    for key, value in sidedata:
        rawtext.append(value)
    rawtext.append(bytes(text))
    return b''.join(rawtext), False


def sidedatareadprocessor(rl, text):
    sidedata = {}
    offset = 0
    (nbentry,) = SIDEDATA_HEADER.unpack(text[: SIDEDATA_HEADER.size])
    offset += SIDEDATA_HEADER.size
    dataoffset = SIDEDATA_HEADER.size + (SIDEDATA_ENTRY.size * nbentry)
    for i in range(nbentry):
        nextoffset = offset + SIDEDATA_ENTRY.size
        key, size, storeddigest = SIDEDATA_ENTRY.unpack(text[offset:nextoffset])
        offset = nextoffset
        # read the data associated with that entry
        nextdataoffset = dataoffset + size
        entrytext = text[dataoffset:nextdataoffset]
        readdigest = hashlib.sha1(entrytext).digest()
        if storeddigest != readdigest:
            raise error.SidedataHashError(key, storeddigest, readdigest)
        sidedata[key] = entrytext
        dataoffset = nextdataoffset
    text = text[dataoffset:]
    return text, True, sidedata


def sidedatarawprocessor(rl, text):
    # side data modifies rawtext and prevent rawtext hash validation
    return False


processors = (
    sidedatareadprocessor,
    sidedatawriteprocessor,
    sidedatarawprocessor,
)