comparison mercurial/manifest.py @ 39668:24870f1be088

narrow: when writing treemanifests, skip inspecting directories outside narrow

This provides significant speed benefits when narrow and treemanifests are in
use, see the timing numbers below. Note that like previously, differences of <5%
are considered noise.

The below timing numbers are in the same style as previously (example:
ee7ee0c516ca). 'before' is 9db85644, and does not include that example commit's
improvements.

    diff --git:
    repo  | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before
    ------+---+---+------------------------+-----------------------+------------
    m-u   |   |   | 1.327 s +-  0.051 s    | 1.296 s +-  0.009 s   |  97.7%
    m-u   |   | x | 1.310 s +-  0.020 s    | 1.295 s +-  0.015 s   |  98.9%
    m-u   | x |   | 1.295 s +-  0.018 s    | 1.296 s +-  0.007 s   | 100.1%
    m-u   | x | x | 83.5 ms +-   0.8 ms    | 84.1 ms +-   0.8 ms   | 100.7%
    l-d-r |   |   | 205.1 ms +-   3.5 ms   | 205.0 ms +-   3.8 ms  | 100.0%
    l-d-r |   | x | 194.2 ms +-   5.6 ms   | 192.3 ms +-   4.3 ms  |  99.0%
    l-d-r | x |   | 99.1 ms +-   2.2 ms    | 97.8 ms +-   0.9 ms   |  98.7%
    l-d-r | x | x | 66.2 ms +-   1.0 ms    | 67.2 ms +-   2.7 ms   | 101.5%

    diff -c . --git:
    repo  | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before
    ------+---+---+------------------------+-----------------------+------------
    m-u   |   |   | 233.9 ms +-   1.9 ms   | 235.6 ms +-   5.1 ms  | 100.7%
    m-u   |   | x | 151.4 ms +-   1.2 ms   | 152.2 ms +-   2.0 ms  | 100.5%
    m-u   | x |   | 234.8 ms +-   2.7 ms   | 235.0 ms +-   2.7 ms  | 100.1%
    m-u   | x | x | 127.8 ms +-   2.1 ms   | 126.0 ms +-   1.1 ms  |  98.6%
    l-d-r |   |   | 82.5 ms +-   1.6 ms    | 82.3 ms +-   2.0 ms   |  99.8%
    l-d-r |   | x | 3.742 s +-  0.017 s    | 3.819 s +-  0.208 s   | 102.1%
    l-d-r | x |   | 84.4 ms +-   1.5 ms    | 83.2 ms +-   1.0 ms   |  98.6%
    l-d-r | x | x | 751.2 ms +-   5.0 ms   | 755.8 ms +-  12.9 ms  | 100.6%

    rebase -r . --keep -d .^^:
    repo  | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before
    ------+---+---+------------------------+-----------------------+------------
    m-u   |   |   | 5.519 s +-  0.038 s    | 5.526 s +-  0.057 s   | 100.1%
    m-u   |   | x | 5.588 s +-  0.048 s    | 5.607 s +-  0.061 s   | 100.3%
    m-u   | x |   | 5.520 s +-  0.044 s    | 5.546 s +-  0.059 s   | 100.5%
    m-u   | x | x | 586.6 ms +-  12.8 ms   | 554.9 ms +-  21.2 ms  |  94.6% <--
    l-d-r |   |   | 629.8 ms +-   5.5 ms   | 627.4 ms +-   6.6 ms  |  99.6%
    l-d-r |   | x | 6.165 s +-  0.058 s    | 6.255 s +-  0.303 s   | 101.5%
    l-d-r | x |   | 270.2 ms +-   2.3 ms   | 271.4 ms +-   2.7 ms  | 100.4%
    l-d-r | x | x | 4.700 s +-  0.025 s    | 1.651 s +-  0.016 s   |  35.1% <--

    status --change . --copies:
    repo  | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before
    ------+---+---+------------------------+-----------------------+------------
    m-u   |   |   | 215.4 ms +-   2.3 ms   | 216.5 ms +-   4.2 ms  | 100.5%
    m-u   |   | x | 132.9 ms +-   1.2 ms   | 132.0 ms +-   1.4 ms  |  99.3%
    m-u   | x |   | 217.0 ms +-   1.9 ms   | 215.4 ms +-   1.9 ms  |  99.3%
    m-u   | x | x | 108.6 ms +-   1.0 ms   | 108.2 ms +-   1.5 ms  |  99.6%
    l-d-r |   |   | 80.0 ms +-   1.3 ms    | 80.5 ms +-   1.1 ms   | 100.6%
    l-d-r |   | x | 3.916 s +-  0.187 s    | 3.966 s +-  0.236 s   | 101.3%
    l-d-r | x |   | 84.4 ms +-   3.1 ms    | 83.9 ms +-   1.1 ms   |  99.4%
    l-d-r | x | x | 758.0 ms +-   8.2 ms   | 753.5 ms +-   5.0 ms  |  99.4%

    status --copies:
    repo  | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before
    ------+---+---+------------------------+-----------------------+------------
    m-u   |   |   | 1.905 s +-  0.025 s    | 1.910 s +-  0.044 s   | 100.3%
    m-u   |   | x | 1.892 s +-  0.009 s    | 1.895 s +-  0.012 s   | 100.2%
    m-u   | x |   | 1.891 s +-  0.012 s    | 1.902 s +-  0.018 s   | 100.6%
    m-u   | x | x | 93.3 ms +-   0.9 ms    | 93.4 ms +-   0.8 ms   | 100.1%
    l-d-r |   |   | 570.7 ms +-   7.8 ms   | 571.9 ms +-  18.5 ms  | 100.2%
    l-d-r |   | x | 561.5 ms +-   5.2 ms   | 562.9 ms +-   6.1 ms  | 100.2%
    l-d-r | x |   | 171.7 ms +-   2.6 ms   | 171.9 ms +-   1.2 ms  | 100.1%
    l-d-r | x | x | 142.7 ms +-   2.0 ms   | 140.3 ms +-   1.0 ms  |  98.3%

    update $rev^; ~/src/hg/hg{hg}/hg update $rev:
    repo  | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before
    ------+---+---+------------------------+-----------------------+------------
    m-u   |   |   | 3.126 s +-  0.016 s    | 3.128 s +-  0.015 s   | 100.1%
    m-u   |   | x | 3.014 s +-  0.068 s    | 3.008 s +-  0.031 s   |  99.8%
    m-u   | x |   | 3.143 s +-  0.037 s    | 3.184 s +-  0.086 s   | 101.3%
    m-u   | x | x | 308.0 ms +-   1.8 ms   | 308.1 ms +-   5.7 ms  | 100.0%
    l-d-r |   |   | 430.8 ms +-   4.5 ms   | 436.4 ms +-   8.7 ms  | 101.3%
    l-d-r |   | x | 9.676 s +-  0.127 s    | 9.945 s +-  0.272 s   | 102.8%
    l-d-r | x |   | 254.2 ms +-   3.3 ms   | 255.7 ms +-   3.1 ms  | 100.6%
    l-d-r | x | x | 1.571 s +-  0.030 s    | 1.555 s +-  0.014 s   |  99.0%

Differential Revision: https://phab.mercurial-scm.org/D4606

39667:0b7594ada0db

        def _load_for_read(s):
            s.parse(gettext(), readsubtree)
            s._dirty = False
        self._loadfunc = _load_for_read

    def writesubtrees(self, m1, m2, writesubtree):
        self._load() # for consistency; should never have any effect here
        m1._load()
        m2._load()
        emptytree = treemanifest()
        def getnode(m, d):
            ld = m._lazydirs.get(d)
            if ld:
                return ld[1]
            return m._dirs.get(d, emptytree)._node

        for d, subm in self._dirs.iteritems():
            subp1 = getnode(m1, d)
            subp2 = getnode(m2, d)
            if subp1 == nullid:
                subp1, subp2 = subp2, subp1
            writesubtree(subm, subp1, subp2)

    def walksubtrees(self, matcher=None):
        """Returns an iterator of the subtrees of this manifest, including this
        manifest itself.

                                      self._dirlogcache,
                                      treemanifest=self._treeondisk)
            self._dirlogcache[d] = mfrevlog
        return self._dirlogcache[d]

    def add(self, m, transaction, link, p1, p2, added, removed, readtree=None):
        if p1 in self.fulltextcache and util.safehasattr(m, 'fastdelta'):
            # If our first parent is in the manifest cache, we can
            # compute a delta here using properties we know about the
            # manifest up-front, which may save time later for the
            # revlog layer.

            # just encode a fulltext of the manifest and pass that
            # through to the revlog layer, and let it handle the delta
            # process.
            if self._treeondisk:
                assert readtree, "readtree must be set for treemanifest writes"
                m1 = readtree(self.tree, p1)
                m2 = readtree(self.tree, p2)
                n = self._addtree(m, transaction, link, m1, m2, readtree)
                arraytext = None
            else:
                text = m.text()
                n = self._revlog.addrevision(text, transaction, link, p1, p2)
                arraytext = bytearray(text)

        if arraytext is not None:
            self.fulltextcache[n] = arraytext

        return n

    def _addtree(self, m, transaction, link, m1, m2, readtree):
        # If the manifest is unchanged compared to one parent,
        # don't write a new revision
        if self.tree != '' and (m.unmodifiedsince(m1) or m.unmodifiedsince(
            m2)):
            return m.node()
        def writesubtree(subm, subp1, subp2):
            sublog = self.dirlog(subm.dir())
            sublog.add(subm, transaction, link, subp1, subp2, None, None,
                       readtree=readtree)
        m.writesubtrees(m1, m2, writesubtree)
        text = m.dirtext()
        n = None
        if self.tree != '':
            # Double-check whether contents are unchanged to one parent
            if text == m1.dirtext():

        return memmf

    def read(self):
        return self._manifestdict

    def write(self, transaction, link, p1, p2, added, removed):
        return self._storage().add(self._manifestdict, transaction, link,
                                   p1, p2, added, removed)

@interfaceutil.implementer(repository.imanifestrevisionstored)
class manifestctx(object):
    """A class representing a single revision of a manifest, including its
    contents, its parent revs, and its linkrev.

        return memmf

    def read(self):
        return self._treemanifest

    def write(self, transaction, link, p1, p2, added, removed):
        def readtree(dir, node):
            return self._manifestlog.get(dir, node).read()
        return self._storage().add(self._treemanifest, transaction, link,
                                   p1, p2, added, removed, readtree=readtree)

@interfaceutil.implementer(repository.imanifestrevisionstored)
class treemanifestctx(object):
    def __init__(self, manifestlog, dir, node):
        self._manifestlog = manifestlog

39668:24870f1be088

        def _load_for_read(s):
            s.parse(gettext(), readsubtree)
            s._dirty = False
        self._loadfunc = _load_for_read

    def writesubtrees(self, m1, m2, writesubtree, match):
        self._load() # for consistency; should never have any effect here
        m1._load()
        m2._load()
        emptytree = treemanifest()
        def getnode(m, d):
            ld = m._lazydirs.get(d)
            if ld:
                return ld[1]
            return m._dirs.get(d, emptytree)._node

        # we should have always loaded everything by the time we get here for
        # `self`, but possibly not in `m1` or `m2`.
        assert not self._lazydirs
        # let's skip investigating things that `match` says we do not need.
        visit = match.visitchildrenset(self._dir[:-1] or '.')
        if visit == 'this' or visit == 'all':
            visit = None
        for d, subm in self._dirs.iteritems():
            if visit and d[:-1] not in visit:
                continue
            subp1 = getnode(m1, d)
            subp2 = getnode(m2, d)
            if subp1 == nullid:
                subp1, subp2 = subp2, subp1
            writesubtree(subm, subp1, subp2, match)

    def walksubtrees(self, matcher=None):
        """Returns an iterator of the subtrees of this manifest, including this
        manifest itself.

                                      self._dirlogcache,
                                      treemanifest=self._treeondisk)
            self._dirlogcache[d] = mfrevlog
        return self._dirlogcache[d]

    def add(self, m, transaction, link, p1, p2, added, removed, readtree=None,
            match=None):
        if p1 in self.fulltextcache and util.safehasattr(m, 'fastdelta'):
            # If our first parent is in the manifest cache, we can
            # compute a delta here using properties we know about the
            # manifest up-front, which may save time later for the
            # revlog layer.

            # just encode a fulltext of the manifest and pass that
            # through to the revlog layer, and let it handle the delta
            # process.
            if self._treeondisk:
                assert readtree, "readtree must be set for treemanifest writes"
                assert match, "match must be specified for treemanifest writes"
                m1 = readtree(self.tree, p1)
                m2 = readtree(self.tree, p2)
                n = self._addtree(m, transaction, link, m1, m2, readtree,
                                  match=match)
                arraytext = None
            else:
                text = m.text()
                n = self._revlog.addrevision(text, transaction, link, p1, p2)
                arraytext = bytearray(text)

        if arraytext is not None:
            self.fulltextcache[n] = arraytext

        return n

    def _addtree(self, m, transaction, link, m1, m2, readtree, match):
        # If the manifest is unchanged compared to one parent,
        # don't write a new revision
        if self.tree != '' and (m.unmodifiedsince(m1) or m.unmodifiedsince(
            m2)):
            return m.node()
        def writesubtree(subm, subp1, subp2, match):
            sublog = self.dirlog(subm.dir())
            sublog.add(subm, transaction, link, subp1, subp2, None, None,
                       readtree=readtree, match=match)
        m.writesubtrees(m1, m2, writesubtree, match)
        text = m.dirtext()
        n = None
        if self.tree != '':
            # Double-check whether contents are unchanged to one parent
            if text == m1.dirtext():

        return memmf

    def read(self):
        return self._manifestdict

    def write(self, transaction, link, p1, p2, added, removed, match=None):
        return self._storage().add(self._manifestdict, transaction, link,
                                   p1, p2, added, removed, match=match)

@interfaceutil.implementer(repository.imanifestrevisionstored)
class manifestctx(object):
    """A class representing a single revision of a manifest, including its
    contents, its parent revs, and its linkrev.

        return memmf

    def read(self):
        return self._treemanifest

    def write(self, transaction, link, p1, p2, added, removed, match=None):
        def readtree(dir, node):
            return self._manifestlog.get(dir, node).read()
        return self._storage().add(self._treemanifest, transaction, link,
                                   p1, p2, added, removed, readtree=readtree,
                                   match=match)

@interfaceutil.implementer(repository.imanifestrevisionstored)
class treemanifestctx(object):
    def __init__(self, manifestlog, dir, node):
        self._manifestlog = manifestlog