view mercurial/sshserver.py @ 21424:d13b4ecdb680

test: split test-largefile.t in multiple file

The `test-largefiles.t` unified test is significantly longer (about 30%) than
any other tests in the mercurial test suite. As a result, its is alway the last
test my test runner is waiting for at the end of a run.

In practice, this means that `test-largefile.t` is wasting half a minute of my
life every times I'm running the mercurial test suites. This probably mean more
a few cumulated day by now.

I've finally decided to split it up in multiple smaller tests to bring it back in
reasonable length.

This changeset extracts independent test cases in two files. One dedicated to
wire protocole testing, and another one dedicated to all other tests that could
be independently extracted.

No test case were haltered in the making of this changeset.

Various timing available below. All timing have been done on a with 90 jobs on a
64 cores machine. Similar result are shown on firefly (20 jobs on 12 core).

General timing of the whole run
--------------------------------

We see a 25% real time improvement for no significant cpu time impact.

Before split:

    real   2m1.149s
    user  58m4.662s
    sys   11m28.563s

After split:

    real   1m31.977s
    user  57m45.993s
    sys   11m33.634s


Last test to finish (using run-test.py --time)
----------------------------------------------

test-largefile.t is now finishing at the same time than other slow tests.

Before split:

    Time      Test
    119.280   test-largefiles.t
     93.995   test-mq.t
     89.897   test-subrepo.t
     86.920   test-glog.t
     85.508   test-rename-merge2.t
     83.594   test-revset.t
     79.824   test-keyword.t
     78.077   test-mq-header-date.t

After split:

    Time      Test
    90.414   test-mq.t
    88.594   test-largefiles.t
    85.363   test-subrepo.t
    81.059   test-glog.t
    78.927   test-rename-merge2.t
    78.021   test-revset.t
    77.777   test-command-template.t



Timing of largefile test themself
-----------------------------------

Running only tests prefixed with "test-largefiles".
No significant change in cumulated time.

Before:

    Time     Test
    58.673   test-largefiles.t
     2.931   test-largefiles-cache.t
     0.583   test-largefiles-small-disk.t

After:

    Time     Test
    31.754   test-largefiles.t
    17.460   test-largefiles-misc.t
     8.888   test-largefiles-wireproto.t
     2.864   test-largefiles-cache.t
     0.580   test-largefiles-small-disk.t

# sshserver.py - ssh protocol server support for mercurial
#
# Copyright 2005-2007 Matt Mackall <[e-mail]>
# Copyright 2006 Vadim Gelfer <[e-mail]>
#
# This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version.

import util, hook, wireproto, changegroup
import os, sys

class sshserver(wireproto.abstractserverproto):
    def __init__(self, ui, repo):
        self.ui = ui
        self.repo = repo
        self.lock = None
        self.fin = ui.fin
        self.fout = ui.fout

        hook.redirect(True)
        ui.fout = repo.ui.fout = ui.ferr

        # Prevent insertion/deletion of CRs
        util.setbinary(self.fin)
        util.setbinary(self.fout)

    def getargs(self, args):
        data = {}
        keys = args.split()
        for n in xrange(len(keys)):
            argline = self.fin.readline()[:-1]
            arg, l = argline.split()
            if arg not in keys:
                raise util.Abort("unexpected parameter %r" % arg)
            if arg == '*':
                star = {}
                for k in xrange(int(l)):
                    argline = self.fin.readline()[:-1]
                    arg, l = argline.split()
                    val = self.fin.read(int(l))
                    star[arg] = val
                data['*'] = star
            else:
                val = self.fin.read(int(l))
                data[arg] = val
        return [data[k] for k in keys]

    def getarg(self, name):
        return self.getargs(name)[0]

    def getfile(self, fpout):
        self.sendresponse('')
        count = int(self.fin.readline())
        while count:
            fpout.write(self.fin.read(count))
            count = int(self.fin.readline())

    def redirect(self):
        pass

    def groupchunks(self, changegroup):
        while True:
            d = changegroup.read(4096)
            if not d:
                break
            yield d

    def sendresponse(self, v):
        self.fout.write("%d\n" % len(v))
        self.fout.write(v)
        self.fout.flush()

    def sendstream(self, source):
        write = self.fout.write
        for chunk in source.gen:
            write(chunk)
        self.fout.flush()

    def sendpushresponse(self, rsp):
        self.sendresponse('')
        self.sendresponse(str(rsp.res))

    def sendpusherror(self, rsp):
        self.sendresponse(rsp.res)

    def sendooberror(self, rsp):
        self.ui.ferr.write('%s\n-\n' % rsp.message)
        self.ui.ferr.flush()
        self.fout.write('\n')
        self.fout.flush()

    def serve_forever(self):
        try:
            while self.serve_one():
                pass
        finally:
            if self.lock is not None:
                self.lock.release()
        sys.exit(0)

    handlers = {
        str: sendresponse,
        wireproto.streamres: sendstream,
        wireproto.pushres: sendpushresponse,
        wireproto.pusherr: sendpusherror,
        wireproto.ooberror: sendooberror,
    }

    def serve_one(self):
        cmd = self.fin.readline()[:-1]
        if cmd and cmd in wireproto.commands:
            rsp = wireproto.dispatch(self.repo, self, cmd)
            self.handlers[rsp.__class__](self, rsp)
        elif cmd:
            impl = getattr(self, 'do_' + cmd, None)
            if impl:
                r = impl()
                if r is not None:
                    self.sendresponse(r)
            else: self.sendresponse("")
        return cmd != ''

    def do_lock(self):
        '''DEPRECATED - allowing remote client to lock repo is not safe'''

        self.lock = self.repo.lock()
        return ""

    def do_unlock(self):
        '''DEPRECATED'''

        if self.lock:
            self.lock.release()
        self.lock = None
        return ""

    def do_addchangegroup(self):
        '''DEPRECATED'''

        if not self.lock:
            self.sendresponse("not locked")
            return

        self.sendresponse("")
        cg = changegroup.unbundle10(self.fin, "UN")
        r = changegroup.addchangegroup(self.repo, cg, 'serve', self._client())
        self.lock.release()
        return str(r)

    def _client(self):
        client = os.environ.get('SSH_CLIENT', '').split(' ', 1)[0]
        return 'remote:ssh:' + client