view mercurial/subrepoutil.py @ 39515:93486cc46125

treemanifest: introduce lazy loading of subdirs

An earlier patch series made it so that what to load was up to the calling code,
which works fine until manifests are copied - when they're copied, they're
loaded completely and thus we lose the entire benefit.

By lazy loading everything, we can avoid having to pass in the matcher to ~every
manifest function, and handle copies correctly as well. This changeset doesn't
go as far as it could with loading only the necessary subsets, that will happen
in later changes in this series; at the moment, except in a few situations, we
just load everything the moment we want to interact with treemanifest._dirs.
This is thus most likely to be a small slowdown if treemanifests is in use
regardless of whether narrow is in use, but hopefully easier to verify
correctness and review.

This is part of a series of speedups, it is not expected to produce any real speed
improvements itself, but the numbers show that it doesn't produce a large speed
penalty in any common case, and for the cases it does provide a penalty in, it
is not a large absolute amount (even if it is a large percentage amount).

Timing numbers according to command:

    hyperfine --prepare <preparation_script> 'hg status'

HGRCPATH points to a file with the following contents:

    [extensions]
    narrow =
    strip =
    rebase =

mozilla-unified (called m-u below) was at revision #468856.
        regular hash: eb39298e432d
  treemanifests hash: 0553b7f29eaf

large-dir-repo (called l-d-r below) was generated with the following script:

    #!/bin/bash
    hg init large-dir-repo
    mkdir -p large-dir-repo/third_party/rust/log
    touch large-dir-repo/third_party/rust/log/foo.txt
    for i in $(seq 1 30000); do
        d=$(mktemp -d large-dir-repo/third_party/XXXXXXXXX)
        touch $d/file.txt
    done
    hg -R large-dir-repo ci -Am 'rev0' --user test --date '0 0'
    echo hi > large-dir-repo/third_party/rust/log/bar.txt
    hg -R large-dir-repo ci -Am 'rev1' --user test --date '0 0'
    echo hi > large-dir-repo/third_party/rust/log/baz.txt
    hg -R large-dir-repo ci -Am 'rev2' --user test --date '0 0'

for the repos that use narrow, the narrowspec was this:

    [include]
    rootfilesin:accessible/jsat
    rootfilesin:accessible/tests/mochitest/jsat
    rootfilesin:mobile/android/chrome/content
    rootfilesin:mobile/android/modules/geckoview
    rootfilesin:third_party/rust/log
    [exclude]

This narrowspec was chosen due to the size of the third_party/rust directory
(this directory was *not* modified in revision #468856 in mozilla-unified),
plus all the directories that *were* modified in revision #468856 of
mozilla-unified.

Importantly, when using narrow, these repos had everything checked out (in the
case of large-dir-repo, that means all 30,001 directories), *before* adding the
narrowspec. This is to simulate the behavior when using a virtual filesystem
that shows everything for the user even if they haven't added it to the
narrowspec yet. This is not a supported configuration, and `hg update` and `hg
rebase` will not really do the "correct" thing if there are mutations outside
of the narrowspec (which is not the case in these tests, due to a carefully
crafted narrowspec), but non-mutating commands should behave correctly.

I'm not claiming anything less than a 5% speed win as improvements due to this
change; these are probably eiter measurement artifacts or constant time
improvements. The numbers that aren't changing are shown primarily to prove that
this doesn't make anything worse in any case I plan on testing during this
series.

'before' is hg from commit 6268fed3
'N' indicates narrow in use
'T' indicates treemanifest in use

Please note that these commands and the narrowspec are a little different than
the ones in a similar table that I made in a3cabe9415e1.

Important: it is my understanding that these numbers below are *not super reliable*,
the large slowdowns may be artifacts of some odd interaction between GC and
python module/code complexity. Another changeset of mine (D4351) had shown large
timing differences when ~empty, uncalled functions were added to match.py,
though only when using --color=never or redirecting to /dev/null. We seem to be
on some cusp of complexity or code size that is causing, at my best guess
(according to linux `perf` benchmarks) GC to alter behavior and cause a
200-400ms difference in timings. I haven't had a chance to replicate these
results on another machine.

    diff --git:
    repo  | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before
    ------+---+---+------------------------+-----------------------+------------
    m-u   |   |   | 1.580 s +-  0.034 s    | 1.576 s +-  0.022 s   |  99.7%
    m-u   |   | x | 1.568 s +-  0.025 s    | 1.584 s +-  0.044 s   | 101.0%
    m-u   | x |   | 1.569 s +-  0.031 s    | 1.554 s +-  0.025 s   |  99.0%
    m-u   | x | x | 107.3 ms +-   1.6 ms   | 106.3 ms +-   1.5 ms  |  99.1%
    l-d-r |   |   | 232.5 ms +-   5.9 ms   | 233.5 ms +-   5.3 ms  | 100.4%
    l-d-r |   | x | 236.6 ms +-   6.3 ms   | 233.6 ms +-   7.0 ms  |  98.7%
    l-d-r | x |   | 118.4 ms +-   2.1 ms   | 118.4 ms +-   1.4 ms  | 100.0%
    l-d-r | x | x | 116.8 ms +-   1.5 ms   | 118.9 ms +-   1.6 ms  | 101.8%

    diff -c . --git:
    repo  | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before
    ------+---+---+------------------------+-----------------------+------------
    m-u   |   |   | 354.4 ms +-  16.6 ms   | 351.0 ms +-   6.9 ms  |  99.0%
    m-u   |   | x | 207.2 ms +-   3.0 ms   | 206.2 ms +-   2.7 ms  |  99.5%
    m-u   | x |   | 422.0 ms +-  26.0 ms   | 351.2 ms +-   6.4 ms  |  83.2% <--
    m-u   | x | x | 166.7 ms +-   2.1 ms   | 169.5 ms +-   4.1 ms  | 101.7%
    l-d-r |   |   | 98.4 ms +-   4.5 ms    | 98.5 ms +-   2.1 ms   | 100.1%
    l-d-r |   | x | 5.519 s +-  0.060 s    | 5.149 s +-  0.042 s   |  93.3% <--
    l-d-r | x |   | 99.1 ms +-   3.2 ms    | 102.6 ms +-   9.7 ms  | 103.5% <--?
    l-d-r | x | x | 994.9 ms +-  10.7 ms   | 1.026 s +-  0.012 s   | 103.1% <--?

    rebase -r . --keep -d .^^:
    repo  | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before
    ------+---+---+------------------------+-----------------------+------------
    m-u   |   |   | 6.639 s +-  0.168 s    | 6.559 s +-  0.097 s   |  98.8%
    m-u   |   | x | 6.601 s +-  0.143 s    | 6.640 s +-  0.207 s   | 100.6%
    m-u   | x |   | 6.582 s +-  0.098 s    | 6.543 s +-  0.098 s   |  99.4%
    m-u   | x | x | 678.4 ms +-  57.7 ms   | 703.7 ms +-  52.4 ms  | 103.7% <--?
    l-d-r |   |   | 780.0 ms +-  23.9 ms   | 776.0 ms +-  12.6 ms  |  99.5%
    l-d-r |   | x | 7.520 s +-  0.255 s    | 7.395 s +-  0.044 s   |  98.3%
    l-d-r | x |   | 331.9 ms +-  16.5 ms   | 327.0 ms +-   3.4 ms  |  98.5%
    l-d-r | x | x | 6.228 s +-  0.113 s    | 5.924 s +-  0.044 s   |  95.1%

    status --change . --copies:
    repo  | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before
    ------+---+---+------------------------+-----------------------+------------
    m-u   |   |   | 330.8 ms +-   7.2 ms   | 329.0 ms +-   7.1 ms  |  99.5%
    m-u   |   | x | 182.9 ms +-   2.7 ms   | 183.5 ms +-   2.7 ms  | 100.3%
    m-u   | x |   | 330.0 ms +-   7.6 ms   | 327.1 ms +-   5.4 ms  |  99.1%
    m-u   | x | x | 146.2 ms +-   2.4 ms   | 147.1 ms +-   1.3 ms  | 100.6%
    l-d-r |   |   | 95.3 ms +-   1.4 ms    | 95.9 ms +-   1.5 ms   | 100.6%
    l-d-r |   | x | 5.157 s +-  0.035 s    | 5.166 s +-  0.058 s   | 100.2%
    l-d-r | x |   | 99.7 ms +-   3.0 ms    | 100.2 ms +-   4.4 ms  | 100.5%
    l-d-r | x | x | 993.6 ms +-  13.1 ms   | 1.025 s +-  0.015 s   | 103.2% <--?

    status --copies:
    repo  | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before
    ------+---+---+------------------------+-----------------------+------------
    m-u   |   |   | 2.348 s +-  0.031 s    | 2.329 s +-  0.019 s   |  99.2%
    m-u   |   | x | 2.337 s +-  0.026 s    | 2.346 s +-  0.034 s   | 100.4%
    m-u   | x |   | 2.354 s +-  0.015 s    | 2.342 s +-  0.021 s   |  99.5%
    m-u   | x | x | 120.6 ms +-   4.3 ms   | 119.2 ms +-   2.1 ms  |  98.8%
    l-d-r |   |   | 731.5 ms +-  11.1 ms   | 719.6 ms +-   9.8 ms  |  98.4%
    l-d-r |   | x | 729.0 ms +-  15.5 ms   | 725.7 ms +-  10.6 ms  |  99.5%
    l-d-r | x |   | 211.0 ms +-   3.9 ms   | 212.8 ms +-   3.7 ms  | 100.9%
    l-d-r | x | x | 211.5 ms +-   4.2 ms   | 211.0 ms +-   3.3 ms  |  99.8%

    update $rev^; ~/src/hg/hg{hg}/hg update $rev:
    repo  | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before
    ------+---+---+------------------------+-----------------------+------------
    m-u   |   |   | 3.910 s +-  0.055 s    | 3.920 s +-  0.075 s   | 100.3%
    m-u   |   | x | 3.613 s +-  0.056 s    | 3.630 s +-  0.056 s   | 100.5%
    m-u   | x |   | 3.873 s +-  0.055 s    | 3.864 s +-  0.049 s   |  99.8%
    m-u   | x | x | 400.4 ms +-   7.4 ms   | 403.6 ms +-   5.0 ms  | 100.8%
    l-d-r |   |   | 531.6 ms +-  10.0 ms   | 528.8 ms +-   9.6 ms  |  99.5%
    l-d-r |   | x | 10.377 s +-  0.049 s   | 9.955 s +-  0.046 s   |  95.9%
    l-d-r | x |   | 308.3 ms +-   4.4 ms   | 306.8 ms +-   3.7 ms  |  99.5%
    l-d-r | x | x | 1.805 s +-  0.015 s    | 1.834 s +-  0.020 s   | 101.6%

Differential Revision: https://phab.mercurial-scm.org/D4366

# subrepoutil.py - sub-repository operations and substate handling
#
# Copyright 2009-2010 Matt Mackall <[e-mail]>
#
# This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version.

from __future__ import absolute_import

import errno
import os
import posixpath
import re

from .i18n import _
from . import (
    config,
    error,
    filemerge,
    pathutil,
    phases,
    util,
)
from .utils import (
    stringutil,
)

nullstate = ('', '', 'empty')

def state(ctx, ui):
    """return a state dict, mapping subrepo paths configured in .hgsub
    to tuple: (source from .hgsub, revision from .hgsubstate, kind
    (key in types dict))
    """
    p = config.config()
    repo = ctx.repo()
    def read(f, sections=None, remap=None):
        if f in ctx:
            try:
                data = ctx[f].data()
            except IOError as err:
                if err.errno != errno.ENOENT:
                    raise
                # handle missing subrepo spec files as removed
                ui.warn(_("warning: subrepo spec file \'%s\' not found\n") %
                        repo.pathto(f))
                return
            p.parse(f, data, sections, remap, read)
        else:
            raise error.Abort(_("subrepo spec file \'%s\' not found") %
                             repo.pathto(f))
    if '.hgsub' in ctx:
        read('.hgsub')

    for path, src in ui.configitems('subpaths'):
        p.set('subpaths', path, src, ui.configsource('subpaths', path))

    rev = {}
    if '.hgsubstate' in ctx:
        try:
            for i, l in enumerate(ctx['.hgsubstate'].data().splitlines()):
                l = l.lstrip()
                if not l:
                    continue
                try:
                    revision, path = l.split(" ", 1)
                except ValueError:
                    raise error.Abort(_("invalid subrepository revision "
                                       "specifier in \'%s\' line %d")
                                     % (repo.pathto('.hgsubstate'), (i + 1)))
                rev[path] = revision
        except IOError as err:
            if err.errno != errno.ENOENT:
                raise

    def remap(src):
        for pattern, repl in p.items('subpaths'):
            # Turn r'C:\foo\bar' into r'C:\\foo\\bar' since re.sub
            # does a string decode.
            repl = stringutil.escapestr(repl)
            # However, we still want to allow back references to go
            # through unharmed, so we turn r'\\1' into r'\1'. Again,
            # extra escapes are needed because re.sub string decodes.
            repl = re.sub(br'\\\\([0-9]+)', br'\\\1', repl)
            try:
                src = re.sub(pattern, repl, src, 1)
            except re.error as e:
                raise error.Abort(_("bad subrepository pattern in %s: %s")
                                 % (p.source('subpaths', pattern),
                                    stringutil.forcebytestr(e)))
        return src

    state = {}
    for path, src in p[''].items():
        kind = 'hg'
        if src.startswith('['):
            if ']' not in src:
                raise error.Abort(_('missing ] in subrepository source'))
            kind, src = src.split(']', 1)
            kind = kind[1:]
            src = src.lstrip() # strip any extra whitespace after ']'

        if not util.url(src).isabs():
            parent = _abssource(repo, abort=False)
            if parent:
                parent = util.url(parent)
                parent.path = posixpath.join(parent.path or '', src)
                parent.path = posixpath.normpath(parent.path)
                joined = bytes(parent)
                # Remap the full joined path and use it if it changes,
                # else remap the original source.
                remapped = remap(joined)
                if remapped == joined:
                    src = remap(src)
                else:
                    src = remapped

        src = remap(src)
        state[util.pconvert(path)] = (src.strip(), rev.get(path, ''), kind)

    return state

def writestate(repo, state):
    """rewrite .hgsubstate in (outer) repo with these subrepo states"""
    lines = ['%s %s\n' % (state[s][1], s) for s in sorted(state)
                                                if state[s][1] != nullstate[1]]
    repo.wwrite('.hgsubstate', ''.join(lines), '')

def submerge(repo, wctx, mctx, actx, overwrite, labels=None):
    """delegated from merge.applyupdates: merging of .hgsubstate file
    in working context, merging context and ancestor context"""
    if mctx == actx: # backwards?
        actx = wctx.p1()
    s1 = wctx.substate
    s2 = mctx.substate
    sa = actx.substate
    sm = {}

    repo.ui.debug("subrepo merge %s %s %s\n" % (wctx, mctx, actx))

    def debug(s, msg, r=""):
        if r:
            r = "%s:%s:%s" % r
        repo.ui.debug("  subrepo %s: %s %s\n" % (s, msg, r))

    promptssrc = filemerge.partextras(labels)
    for s, l in sorted(s1.iteritems()):
        prompts = None
        a = sa.get(s, nullstate)
        ld = l # local state with possible dirty flag for compares
        if wctx.sub(s).dirty():
            ld = (l[0], l[1] + "+")
        if wctx == actx: # overwrite
            a = ld

        prompts = promptssrc.copy()
        prompts['s'] = s
        if s in s2:
            r = s2[s]
            if ld == r or r == a: # no change or local is newer
                sm[s] = l
                continue
            elif ld == a: # other side changed
                debug(s, "other changed, get", r)
                wctx.sub(s).get(r, overwrite)
                sm[s] = r
            elif ld[0] != r[0]: # sources differ
                prompts['lo'] = l[0]
                prompts['ro'] = r[0]
                if repo.ui.promptchoice(
                    _(' subrepository sources for %(s)s differ\n'
                      'use (l)ocal%(l)s source (%(lo)s)'
                      ' or (r)emote%(o)s source (%(ro)s)?'
                      '$$ &Local $$ &Remote') % prompts, 0):
                    debug(s, "prompt changed, get", r)
                    wctx.sub(s).get(r, overwrite)
                    sm[s] = r
            elif ld[1] == a[1]: # local side is unchanged
                debug(s, "other side changed, get", r)
                wctx.sub(s).get(r, overwrite)
                sm[s] = r
            else:
                debug(s, "both sides changed")
                srepo = wctx.sub(s)
                prompts['sl'] = srepo.shortid(l[1])
                prompts['sr'] = srepo.shortid(r[1])
                option = repo.ui.promptchoice(
                    _(' subrepository %(s)s diverged (local revision: %(sl)s, '
                      'remote revision: %(sr)s)\n'
                      '(M)erge, keep (l)ocal%(l)s or keep (r)emote%(o)s?'
                      '$$ &Merge $$ &Local $$ &Remote')
                    % prompts, 0)
                if option == 0:
                    wctx.sub(s).merge(r)
                    sm[s] = l
                    debug(s, "merge with", r)
                elif option == 1:
                    sm[s] = l
                    debug(s, "keep local subrepo revision", l)
                else:
                    wctx.sub(s).get(r, overwrite)
                    sm[s] = r
                    debug(s, "get remote subrepo revision", r)
        elif ld == a: # remote removed, local unchanged
            debug(s, "remote removed, remove")
            wctx.sub(s).remove()
        elif a == nullstate: # not present in remote or ancestor
            debug(s, "local added, keep")
            sm[s] = l
            continue
        else:
            if repo.ui.promptchoice(
                _(' local%(l)s changed subrepository %(s)s'
                  ' which remote%(o)s removed\n'
                  'use (c)hanged version or (d)elete?'
                  '$$ &Changed $$ &Delete') % prompts, 0):
                debug(s, "prompt remove")
                wctx.sub(s).remove()

    for s, r in sorted(s2.items()):
        prompts = None
        if s in s1:
            continue
        elif s not in sa:
            debug(s, "remote added, get", r)
            mctx.sub(s).get(r)
            sm[s] = r
        elif r != sa[s]:
            prompts = promptssrc.copy()
            prompts['s'] = s
            if repo.ui.promptchoice(
                _(' remote%(o)s changed subrepository %(s)s'
                  ' which local%(l)s removed\n'
                  'use (c)hanged version or (d)elete?'
                  '$$ &Changed $$ &Delete') % prompts, 0) == 0:
                debug(s, "prompt recreate", r)
                mctx.sub(s).get(r)
                sm[s] = r

    # record merged .hgsubstate
    writestate(repo, sm)
    return sm

def precommit(ui, wctx, status, match, force=False):
    """Calculate .hgsubstate changes that should be applied before committing

    Returns (subs, commitsubs, newstate) where
    - subs: changed subrepos (including dirty ones)
    - commitsubs: dirty subrepos which the caller needs to commit recursively
    - newstate: new state dict which the caller must write to .hgsubstate

    This also updates the given status argument.
    """
    subs = []
    commitsubs = set()
    newstate = wctx.substate.copy()

    # only manage subrepos and .hgsubstate if .hgsub is present
    if '.hgsub' in wctx:
        # we'll decide whether to track this ourselves, thanks
        for c in status.modified, status.added, status.removed:
            if '.hgsubstate' in c:
                c.remove('.hgsubstate')

        # compare current state to last committed state
        # build new substate based on last committed state
        oldstate = wctx.p1().substate
        for s in sorted(newstate.keys()):
            if not match(s):
                # ignore working copy, use old state if present
                if s in oldstate:
                    newstate[s] = oldstate[s]
                    continue
                if not force:
                    raise error.Abort(
                        _("commit with new subrepo %s excluded") % s)
            dirtyreason = wctx.sub(s).dirtyreason(True)
            if dirtyreason:
                if not ui.configbool('ui', 'commitsubrepos'):
                    raise error.Abort(dirtyreason,
                        hint=_("use --subrepos for recursive commit"))
                subs.append(s)
                commitsubs.add(s)
            else:
                bs = wctx.sub(s).basestate()
                newstate[s] = (newstate[s][0], bs, newstate[s][2])
                if oldstate.get(s, (None, None, None))[1] != bs:
                    subs.append(s)

        # check for removed subrepos
        for p in wctx.parents():
            r = [s for s in p.substate if s not in newstate]
            subs += [s for s in r if match(s)]
        if subs:
            if (not match('.hgsub') and
                '.hgsub' in (wctx.modified() + wctx.added())):
                raise error.Abort(_("can't commit subrepos without .hgsub"))
            status.modified.insert(0, '.hgsubstate')

    elif '.hgsub' in status.removed:
        # clean up .hgsubstate when .hgsub is removed
        if ('.hgsubstate' in wctx and
            '.hgsubstate' not in (status.modified + status.added +
                                  status.removed)):
            status.removed.insert(0, '.hgsubstate')

    return subs, commitsubs, newstate

def reporelpath(repo):
    """return path to this (sub)repo as seen from outermost repo"""
    parent = repo
    while util.safehasattr(parent, '_subparent'):
        parent = parent._subparent
    return repo.root[len(pathutil.normasprefix(parent.root)):]

def subrelpath(sub):
    """return path to this subrepo as seen from outermost repo"""
    return sub._relpath

def _abssource(repo, push=False, abort=True):
    """return pull/push path of repo - either based on parent repo .hgsub info
    or on the top repo config. Abort or return None if no source found."""
    if util.safehasattr(repo, '_subparent'):
        source = util.url(repo._subsource)
        if source.isabs():
            return bytes(source)
        source.path = posixpath.normpath(source.path)
        parent = _abssource(repo._subparent, push, abort=False)
        if parent:
            parent = util.url(util.pconvert(parent))
            parent.path = posixpath.join(parent.path or '', source.path)
            parent.path = posixpath.normpath(parent.path)
            return bytes(parent)
    else: # recursion reached top repo
        path = None
        if util.safehasattr(repo, '_subtoppath'):
            path = repo._subtoppath
        elif push and repo.ui.config('paths', 'default-push'):
            path = repo.ui.config('paths', 'default-push')
        elif repo.ui.config('paths', 'default'):
            path = repo.ui.config('paths', 'default')
        elif repo.shared():
            # chop off the .hg component to get the default path form.  This has
            # already run through vfsmod.vfs(..., realpath=True), so it doesn't
            # have problems with 'C:'
            return os.path.dirname(repo.sharedpath)
        if path:
            # issue5770: 'C:\' and 'C:' are not equivalent paths.  The former is
            # as expected: an absolute path to the root of the C: drive.  The
            # latter is a relative path, and works like so:
            #
            #   C:\>cd C:\some\path
            #   C:\>D:
            #   D:\>python -c "import os; print os.path.abspath('C:')"
            #   C:\some\path
            #
            #   D:\>python -c "import os; print os.path.abspath('C:relative')"
            #   C:\some\path\relative
            if util.hasdriveletter(path):
                if len(path) == 2 or path[2:3] not in br'\/':
                    path = os.path.abspath(path)
            return path

    if abort:
        raise error.Abort(_("default path for subrepository not found"))

def newcommitphase(ui, ctx):
    commitphase = phases.newcommitphase(ui)
    substate = getattr(ctx, "substate", None)
    if not substate:
        return commitphase
    check = ui.config('phases', 'checksubrepos')
    if check not in ('ignore', 'follow', 'abort'):
        raise error.Abort(_('invalid phases.checksubrepos configuration: %s')
                         % (check))
    if check == 'ignore':
        return commitphase
    maxphase = phases.public
    maxsub = None
    for s in sorted(substate):
        sub = ctx.sub(s)
        subphase = sub.phase(substate[s][1])
        if maxphase < subphase:
            maxphase = subphase
            maxsub = s
    if commitphase < maxphase:
        if check == 'abort':
            raise error.Abort(_("can't commit in %s phase"
                               " conflicting %s from subrepository %s") %
                             (phases.phasenames[commitphase],
                              phases.phasenames[maxphase], maxsub))
        ui.warn(_("warning: changes are committed in"
                  " %s phase from subrepository %s\n") %
                (phases.phasenames[maxphase], maxsub))
        return maxphase
    return commitphase