view mercurial/wireprotov2peer.py @ 38956:a3cabe9415e1

dirstate: use visitchildrenset in traverse This speeds up `hg status` a fair amount when there is a very large directory and narrow is in use. Timing numbers according to command: hyperfine --warmup 1 'hg status' HGRCPATH points to a file with the following contents: [extensions] narrow = mozilla-unified (called m-u below) was at revision #468856. regular hash: eb39298e432d treemanifests hash: 0553b7f29eaf large-dir-repo (called l-d-r below) was generated with the following script: #!/bin/bash hg init large-dir-repo mkdir -p large-dir-repo/third_party/rust/log touch large-dir-repo/third_party/rust/log/foo.txt for i in $(seq 1 30000); do d=$(mktemp -d large-dir-repo/third_party/XXXXXXXXX) touch $d/file.txt done hg -R large-dir-repo ci -Am 'rev0' --user test --date '0 0' for repos that use narrow, the narrowspec was this: [includes] rootfilesin:third_party/rust/log [excludes] This narrowspec was chosen due to the size of the third_party/rust directory; this directory was *not* modified in revision #468856 in mozilla-unified. Importantly, when using narrow, these repos had everything checked out (in the case of large-dir-repo, that means all 30,001 directories), *before* adding the narrowspec. This is to simulate the behavior when using a virtual filesystem that shows everything for the user even if they haven't added it to the narrowspec yet. This is not a supported configuration, and `hg update` will not really do the "correct" thing, but non-mutating commands should behave correctly. There are two repos below that do not follow the setup above, 'citc1' and 'citc2', which are using a virtual filesystem and can not be reproduced upstream; these numbers are here mostly to indicate that these performance improvements are not hypothetical, and show the benefits we're hoping to achieve on our real workloads. 'citc1' is closest to large-dir-repo with one of our pathological cases, 'citc2' is an arbitrary repo and closer to "average". I'm not claiming anything less than a 5% speed win as improvements due to this change; these are probably eiter measurement artifacts or constant time improvements. The numbers that aren't changing are shown primarily to prove that this doesn't make anything worse in any case I plan on testing during this series. 'before' is hg from commit c83ad576. 'N' indicates narrow in use, 'T' indicates treemanifest in use. hg status: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 2.284 s +- 0.022 s | 2.274 s +- 0.021 s | 99.6% m-u | | x | 2.289 s +- 0.008 s | 2.284 s +- 0.028 s | 99.8% m-u | x | | 430.8 ms +- 3.1 ms | 424.5 ms +- 3.2 ms | 98.5% m-u | x | x | 429.8 ms +- 2.5 ms | 425.8 ms +- 3.7 ms | 99.1% l-d-r | | | 681.3 ms +- 5.5 ms | 689.6 ms +- 8.0 ms | 101.2% l-d-r | | x | 666.8 ms +- 21.8 ms | 672.5 ms +- 14.9 ms | 100.9% l-d-r | x | | 282.6 ms +- 1.8 ms | 203.0 ms +- 1.2 ms | 71.8% <-- l-d-r | x | x | 275.2 ms +- 3.9 ms | 199.3 ms +- 3.5 ms | 72.4% <-- citc1 | x | x | 1.023 s +- 0.011 s | 398.6 ms +- 9.2 ms | 39.0% <-- citc2 | x | x | 297.9 ms +- 4.4 ms | 289.6 ms +- 4.2 ms | 97.2% hg status --change .: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 478.2 ms +- 2.0 ms | 476.9 ms +- 3.7 ms | 99.7% m-u | | x | 169.5 ms +- 2.7 ms | 169.5 ms +- 2.5 ms | 100.0% m-u | x | | 477.0 ms +- 2.4 ms | 476.1 ms +- 1.4 ms | 99.8% m-u | x | x | 124.7 ms +- 1.9 ms | 124.2 ms +- 3.3 ms | 99.6% l-d-r | | | 97.4 ms +- 1.2 ms | 96.5 ms +- 1.2 ms | 99.1% l-d-r | | x | 4.778 s +- 0.018 s | 4.774 s +- 0.011 s | 99.9% l-d-r | x | | 99.9 ms +- 1.1 ms | 98.8 ms +- 1.3 ms | 98.9% l-d-r | x | x | 848.7 ms +- 7.1 ms | 849.4 ms +- 6.5 ms | 100.1% citc1 | x | x | 4.250 s +- 0.051 s | 4.283 s +- 0.042 s | 100.8% citc2 | x | x | 341.5 ms +- 4.7 ms | 341.5 ms +- 4.1 ms | 100.0% hg update $rev^; hg update $rev: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 4.357 s +- 0.032 s | 4.312 s +- 0.093 s | 99.0% m-u | | x | 3.599 s +- 0.061 s | 3.592 s +- 0.071 s | 99.8% m-u | x | | 1.815 s +- 0.012 s | 1.816 s +- 0.013 s | 100.1% m-u | x | x | 1.110 s +- 0.009 s | 1.106 s +- 0.005 s | 99.6% l-d-r | | | 527.1 ms +- 7.8 ms | 523.3 ms +- 6.5 ms | 99.3% l-d-r | | x | 8.835 s +- 0.067 s | 8.825 s +- 0.064 s | 99.9% l-d-r | x | | 313.0 ms +- 2.2 ms | 312.1 ms +- 1.2 ms | 99.7% l-d-r | x | x | 1.780 s +- 0.011 s | 1.799 s +- 0.013 s | 101.1% citc1 | x | x | 6.825 s +- 0.262 s | 6.707 s +- 0.353 s | 98.3% citc2 | x | x | 776.4 ms +- 4.5 ms | 781.3 ms +- 6.3 ms | 100.6% hg diff: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 1.519 s +- 0.015 s | 1.525 s +- 0.017 s | 100.4% m-u | | x | 1.512 s +- 0.010 s | 1.517 s +- 0.027 s | 100.3% m-u | x | | 420.0 ms +- 3.2 ms | 417.1 ms +- 1.9 ms | 99.3% m-u | x | x | 415.0 ms +- 3.8 ms | 415.7 ms +- 2.7 ms | 100.2% l-d-r | | | 220.8 ms +- 4.0 ms | 220.8 ms +- 3.7 ms | 100.0% l-d-r | | x | 216.6 ms +- 7.5 ms | 211.4 ms +- 2.1 ms | 97.6% l-d-r | x | | 111.9 ms +- 1.8 ms | 112.0 ms +- 1.5 ms | 100.1% l-d-r | x | x | 111.4 ms +- 1.4 ms | 110.2 ms +- 1.0 ms | 98.9% citc1 | x | x | 268.7 ms +- 2.3 ms | 269.6 ms +- 2.8 ms | 100.3% citc2 | x | x | 273.5 ms +- 5.5 ms | 273.9 ms +- 3.7 ms | 100.1% hg diff -c .: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+--------------------------+-----------------------+---------- m-u | | | 497.1 ms +- 1.4 ms | 500.1 ms +- 2.4 ms | 100.6% m-u | | x | 195.3 ms +- 13.2 ms | 191.6 ms +- 3.0 ms | 98.1% m-u | x | | 476.8 ms +- 1.9 ms | 476.7 ms +- 2.3 ms | 100.0% m-u | x | x | 122.8 ms +- 2.1 ms | 122.9 ms +- 2.0 ms | 100.1% l-d-r | | | 99.3 ms +- 2.3 ms | 98.8 ms +- 1.7 ms | 99.5% l-d-r | | x | 4.875 s +- 0.041 s | 4.847 s +- 0.038 s | 99.4% l-d-r | x | | 98.5 ms +- 1.2 ms | 98.9 ms +- 1.3 ms | 100.4% l-d-r | x | x | 864.6 ms +- 7.4 ms | 855.4 ms +- 6.6 ms | 98.9% citc1 | x | x | 4.505 s +- 0.060 s | 4.466 s +- 0.036 s | 99.1% citc2 | x | x | 368.0 ms +- 4.0 ms | 365.5 ms +- 6.3 ms | 99.3% Differential Revision: https://phab.mercurial-scm.org/D4131
author Kyle Lippincott <spectral@google.com>
date Tue, 31 Jul 2018 16:47:43 -0700
parents 3ea8323d6f95
children 1467b6c27ff9
line wrap: on
line source

# wireprotov2peer.py - client side code for wire protocol version 2
#
# Copyright 2018 Gregory Szorc <gregory.szorc@gmail.com>
#
# This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version.

from __future__ import absolute_import

from .i18n import _
from .thirdparty import (
    cbor,
)
from . import (
    encoding,
    error,
    util,
    wireprotoframing,
)

def formatrichmessage(atoms):
    """Format an encoded message from the framing protocol."""

    chunks = []

    for atom in atoms:
        msg = _(atom[b'msg'])

        if b'args' in atom:
            msg = msg % atom[b'args']

        chunks.append(msg)

    return b''.join(chunks)

class commandresponse(object):
    """Represents the response to a command request."""

    def __init__(self, requestid, command):
        self.requestid = requestid
        self.command = command

        self.b = util.bytesio()

    def cborobjects(self):
        """Obtain decoded CBOR objects from this response."""
        size = self.b.tell()
        self.b.seek(0)

        decoder = cbor.CBORDecoder(self.b)

        while self.b.tell() < size:
            yield decoder.decode()

class clienthandler(object):
    """Object to handle higher-level client activities.

    The ``clientreactor`` is used to hold low-level state about the frame-based
    protocol, such as which requests and streams are active. This type is used
    for higher-level operations, such as reading frames from a socket, exposing
    and managing a higher-level primitive for representing command responses,
    etc. This class is what peers should probably use to bridge wire activity
    with the higher-level peer API.
    """

    def __init__(self, ui, clientreactor):
        self._ui = ui
        self._reactor = clientreactor
        self._requests = {}
        self._futures = {}
        self._responses = {}

    def callcommand(self, command, args, f):
        """Register a request to call a command.

        Returns an iterable of frames that should be sent over the wire.
        """
        request, action, meta = self._reactor.callcommand(command, args)

        if action != 'noop':
            raise error.ProgrammingError('%s not yet supported' % action)

        rid = request.requestid
        self._requests[rid] = request
        self._futures[rid] = f
        self._responses[rid] = commandresponse(rid, command)

        return iter(())

    def flushcommands(self):
        """Flush all queued commands.

        Returns an iterable of frames that should be sent over the wire.
        """
        action, meta = self._reactor.flushcommands()

        if action != 'sendframes':
            raise error.ProgrammingError('%s not yet supported' % action)

        return meta['framegen']

    def readframe(self, fh):
        """Attempt to read and process a frame.

        Returns None if no frame was read. Presumably this means EOF.
        """
        frame = wireprotoframing.readframe(fh)
        if frame is None:
            # TODO tell reactor?
            return

        self._ui.note(_('received %r\n') % frame)
        self._processframe(frame)

        return True

    def _processframe(self, frame):
        """Process a single read frame."""

        action, meta = self._reactor.onframerecv(frame)

        if action == 'error':
            e = error.RepoError(meta['message'])

            if frame.requestid in self._futures:
                self._futures[frame.requestid].set_exception(e)
            else:
                raise e

        if frame.requestid not in self._requests:
            raise error.ProgrammingError(
                'received frame for unknown request; this is either a bug in '
                'the clientreactor not screening for this or this instance was '
                'never told about this request: %r' % frame)

        response = self._responses[frame.requestid]

        if action == 'responsedata':
            response.b.write(meta['data'])

            if meta['eos']:
                # If the command has a decoder, resolve the future to the
                # decoded value. Otherwise resolve to the rich response object.
                decoder = COMMAND_DECODERS.get(response.command)

                # TODO consider always resolving the overall status map.
                if decoder:
                    objs = response.cborobjects()

                    overall = next(objs)

                    if overall['status'] == 'ok':
                        self._futures[frame.requestid].set_result(decoder(objs))
                    else:
                        e = error.RepoError(
                            formatrichmessage(overall['error']['message']))
                        self._futures[frame.requestid].set_exception(e)
                else:
                    self._futures[frame.requestid].set_result(response)

                del self._requests[frame.requestid]
                del self._futures[frame.requestid]

        else:
            raise error.ProgrammingError(
                'unhandled action from clientreactor: %s' % action)

def decodebranchmap(objs):
    # Response should be a single CBOR map of branch name to array of nodes.
    bm = next(objs)

    return {encoding.tolocal(k): v for k, v in bm.items()}

def decodeheads(objs):
    # Array of node bytestrings.
    return next(objs)

def decodeknown(objs):
    # Bytestring where each byte is a 0 or 1.
    raw = next(objs)

    return [True if c == '1' else False for c in raw]

def decodelistkeys(objs):
    # Map with bytestring keys and values.
    return next(objs)

def decodelookup(objs):
    return next(objs)

def decodepushkey(objs):
    return next(objs)

COMMAND_DECODERS = {
    'branchmap': decodebranchmap,
    'heads': decodeheads,
    'known': decodeknown,
    'listkeys': decodelistkeys,
    'lookup': decodelookup,
    'pushkey': decodepushkey,
}