view contrib/benchmarks/__init__.py @ 39772:ae531f5e583c

testing: add interface unit tests for file storage Our strategy for supporting alternate storage backends is to define interfaces for everything then "code to the interface." We already have interfaces for various primitives, including file and manifest storage. What we don't have is generic unit tests for those interfaces. Up to this point we've been relying on high-level integration tests (mainly in the form of existing .t tests) to test alternate storage backends. And my experience with developing the "simple store" test extension is that such testing is very tedious: it takes several minutes to run all tests and when you find a failure, it is often non-trivial to debug. This commit starts to change that. This commit introduces the mercurial.testing.storage module. It contains testing code for storage. Currently, it defines some unittest.TestCase classes for testing the file storage interfaces. It also defines some factory functions that allow a caller to easily spawn a custom TestCase "bound" to a specific file storage backend implementation. A new .py test has been added. It simply defines a callable to produce filelog and transaction instances on demand and then "registers" the various test classes so the filelog class can be tested with the storage interface unit tests. As part of writing the tests, I identified a couple of apparent bugs in revlog.py and filelog.py! These are tracked with inline TODO comments. Writing the tests makes it more obvious where the storage interface is lacking. For example, we raise either IndexError or error.LookupError for missing revisions depending on whether we use an integer revision or a node. Also, we raise error.RevlogError in various places when we should be raising a storage-agnostic error type. The storage interfaces are currently far from perfect and there is much work to be done to improve them. But at least with this commit we finally have the start of unit tests that can be used to "qualify" the behavior of a storage backend. And when implementing and debugging new storage backends, we now have an obvious place to define new tests and have obvious places to insert breakpoints to facilitate debugging. This should be invaluable when implementing new storage backends. I added the mercurial.testing package because these interface conformance tests are generic and need to be usable by all storage backends. Having the code live in tests/ would make it difficult for storage backends implemented in extensions to test their interface conformance. First, it would require obtaining a copy of Mercurial's storage test code in order to test. Second, it would make testing against multiple Mercurial versions difficult, as you would need to import N copies of the storage testing code in order to achieve test coverage. By making the test code part of the Mercurial distribution itself, extensions can `import mercurial.testing.*` to access and run the test code. The test will run against whatever Mercurial version is active. FWIW I've always wanted to move parts of run-tests.py into the mercurial.* package to make the testing story simpler (e.g. imagine an `hg debugruntests` command that could invoke the test harness). While I have no plans to do that in the near future, establishing the mercurial.testing package does provide a natural home for that code should someone do this in the future. Differential Revision: https://phab.mercurial-scm.org/D4650
author Gregory Szorc <gregory.szorc@gmail.com>
date Tue, 18 Sep 2018 16:52:11 -0700
parents be0e7af80543
children 2372284d9457
line wrap: on
line source

# __init__.py - asv benchmark suite
#
# Copyright 2016 Logilab SA <contact@logilab.fr>
#
# This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version.

# "historical portability" policy of contrib/benchmarks:
#
# We have to make this code work correctly with current mercurial stable branch
# and if possible with reasonable cost with early Mercurial versions.

'''ASV (https://asv.readthedocs.io) benchmark suite

Benchmark are parameterized against reference repositories found in the
directory pointed by the REPOS_DIR environment variable.

Invocation example:

    $ export REPOS_DIR=~/hgperf/repos
    # run suite on given revision
    $ asv --config contrib/asv.conf.json run REV
    # run suite on new changesets found in stable and default branch
    $ asv --config contrib/asv.conf.json run NEW
    # display a comparative result table of benchmark results between two given
    # revisions
    $ asv --config contrib/asv.conf.json compare REV1 REV2
    # compute regression detection and generate ASV static website
    $ asv --config contrib/asv.conf.json publish
    # serve the static website
    $ asv --config contrib/asv.conf.json preview
'''

from __future__ import absolute_import

import functools
import os
import re

from mercurial import (
    extensions,
    hg,
    ui as uimod,
    util,
)

basedir = os.path.abspath(os.path.join(os.path.dirname(__file__),
                          os.path.pardir, os.path.pardir))
reposdir = os.environ['REPOS_DIR']
reposnames = [name for name in os.listdir(reposdir)
              if os.path.isdir(os.path.join(reposdir, name, ".hg"))]
if not reposnames:
    raise ValueError("No repositories found in $REPO_DIR")
outputre = re.compile((r'! wall (\d+.\d+) comb \d+.\d+ user \d+.\d+ sys '
                       r'\d+.\d+ \(best of \d+\)'))

def runperfcommand(reponame, command, *args, **kwargs):
    os.environ["HGRCPATH"] = os.environ.get("ASVHGRCPATH", "")
    # for "historical portability"
    # ui.load() has been available since d83ca85
    if util.safehasattr(uimod.ui, "load"):
        ui = uimod.ui.load()
    else:
        ui = uimod.ui()
    repo = hg.repository(ui, os.path.join(reposdir, reponame))
    perfext = extensions.load(ui, 'perfext',
                              os.path.join(basedir, 'contrib', 'perf.py'))
    cmd = getattr(perfext, command)
    ui.pushbuffer()
    cmd(ui, repo, *args, **kwargs)
    output = ui.popbuffer()
    match = outputre.search(output)
    if not match:
        raise ValueError("Invalid output {0}".format(output))
    return float(match.group(1))

def perfbench(repos=reposnames, name=None, params=None):
    """decorator to declare ASV benchmark based on contrib/perf.py extension

    An ASV benchmark is a python function with the given attributes:

    __name__: should start with track_, time_ or mem_ to be collected by ASV
    params and param_name: parameter matrix to display multiple graphs on the
    same page.
    pretty_name: If defined it's displayed in web-ui instead of __name__
    (useful for revsets)
    the module name is prepended to the benchmark name and displayed as
    "category" in webui.

    Benchmarks are automatically parameterized with repositories found in the
    REPOS_DIR environment variable.

    `params` is the param matrix in the form of a list of tuple
    (param_name, [value0, value1])

    For example [(x, [a, b]), (y, [c, d])] declare benchmarks for
    (a, c), (a, d), (b, c) and (b, d).
    """
    params = list(params or [])
    params.insert(0, ("repo", repos))

    def decorator(func):
        @functools.wraps(func)
        def wrapped(repo, *args):
            def perf(command, *a, **kw):
                return runperfcommand(repo, command, *a, **kw)
            return func(perf, *args)

        wrapped.params = [p[1] for p in params]
        wrapped.param_names = [p[0] for p in params]
        wrapped.pretty_name = name
        return wrapped
    return decorator