mercurial/streamclone.py

convert: fix mercurial_sink.putcommit

Changeset 4ebc8693ce72 added some code to putcommit to avoid creating a
revision that touches no files, but this can break regular conversions
from some repositories:

- conceptually, since we're converting a repo, we should try to make
  the new hg repo as similar as possible to the original repo - we
  should create a new changeset, even if the original revision didn't
  touch any files (maybe the commit message had some important bit);

- even if a "regular" revision that doesn't touch any file may seem
  weird (and maybe even broken), it's completely legitimate for a merge
  revision to not touch any file, and, if we just skip it, the
  converted repo will end up with wrong history and possibly an extra
  head.

  As an example, say the crew and main hg repos are sync'ed.  Somebody
  sends an important patch to the mailing list.  Matt quickly applies
  and pushes it.  But at the same time somebody also applies it to crew
  and pushes it.  Suppose the commit message ended up being a bit
  different (say, there was a typo and somebody didn't fix it) or that
  the date ended up being different (because of different patch-applying
  scripts): the changeset hashes will be different, but the manifests
  will be the same.

  Since both changesets were pushed to public repos, it's hard to recall
  them.  If both are merged, the manifest from the resulting merge
  revision will have the exact same contents as its parents - i.e. the
  merge revision really doesn't touch any file at all.

To keep the file filtering stuff "working", the generic code was changed
to skip empty revisions if we're filtering the repo, fixing a bug in the
process (we want parents[0] instead of tip).

# streamclone.py - streaming clone server support for mercurial
#
# Copyright 2006 Vadim Gelfer <[e-mail]>
#
# This software may be used and distributed according to the terms
# of the GNU General Public License, incorporated herein by reference.

from i18n import _
import os, stat, util, lock

# if server supports streaming clone, it advertises "stream"
# capability with value that is version+flags of repo it is serving.
# client only streams if it can read that repo format.

def walkrepo(root):
    '''iterate over metadata files in repository.
    walk in natural (sorted) order.
    yields 2-tuples: name of .d or .i file, size of file.'''

    strip_count = len(root) + len(os.sep)
    def walk(path, recurse):
        ents = os.listdir(path)
        ents.sort()
        for e in ents:
            pe = os.path.join(path, e)
            st = os.lstat(pe)
            if stat.S_ISDIR(st.st_mode):
                if recurse:
                    for x in walk(pe, True):
                        yield x
            else:
                if not stat.S_ISREG(st.st_mode) or len(e) < 2:
                    continue
                sfx = e[-2:]
                if sfx in ('.d', '.i'):
                    yield pe[strip_count:], st.st_size
    # write file data first
    for x in walk(os.path.join(root, 'data'), True):
        yield x
    # write manifest before changelog
    meta = list(walk(root, False))
    meta.sort()
    meta.reverse()
    for x in meta:
        yield x

# stream file format is simple.
#
# server writes out line that says how many files, how many total
# bytes.  separator is ascii space, byte counts are strings.
#
# then for each file:
#
#   server writes out line that says file name, how many bytes in
#   file.  separator is ascii nul, byte count is string.
#
#   server writes out raw file data.

def stream_out(repo, fileobj, untrusted=False):
    '''stream out all metadata files in repository.
    writes to file-like object, must support write() and optional flush().'''

    if not repo.ui.configbool('server', 'uncompressed', untrusted=untrusted):
        fileobj.write('1\n')
        return

    # get consistent snapshot of repo. lock during scan so lock not
    # needed while we stream, and commits can happen.
    lock = None
    try:
        try:
            repolock = repo.lock()
        except (lock.LockHeld, lock.LockUnavailable), inst:
            repo.ui.warn('locking the repository failed: %s\n' % (inst,))
            fileobj.write('2\n')
            return

        fileobj.write('0\n')
        repo.ui.debug('scanning\n')
        entries = []
        total_bytes = 0
        for name, size in walkrepo(repo.spath):
            name = repo.decodefn(util.pconvert(name))
            entries.append((name, size))
            total_bytes += size
    finally:
        del repolock

    repo.ui.debug('%d files, %d bytes to transfer\n' %
                  (len(entries), total_bytes))
    fileobj.write('%d %d\n' % (len(entries), total_bytes))
    for name, size in entries:
        repo.ui.debug('sending %s (%d bytes)\n' % (name, size))
        fileobj.write('%s\0%d\n' % (name, size))
        for chunk in util.filechunkiter(repo.sopener(name), limit=size):
            fileobj.write(chunk)
    flush = getattr(fileobj, 'flush', None)
    if flush: flush()