mercurial/dicthelpers.py

test: split test-largefile.t in multiple file

The `test-largefiles.t` unified test is significantly longer (about 30%) than
any other tests in the mercurial test suite. As a result, its is alway the last
test my test runner is waiting for at the end of a run.

In practice, this means that `test-largefile.t` is wasting half a minute of my
life every times I'm running the mercurial test suites. This probably mean more
a few cumulated day by now.

I've finally decided to split it up in multiple smaller tests to bring it back in
reasonable length.

This changeset extracts independent test cases in two files. One dedicated to
wire protocole testing, and another one dedicated to all other tests that could
be independently extracted.

No test case were haltered in the making of this changeset.

Various timing available below. All timing have been done on a with 90 jobs on a
64 cores machine. Similar result are shown on firefly (20 jobs on 12 core).

General timing of the whole run
--------------------------------

We see a 25% real time improvement for no significant cpu time impact.

Before split:

    real   2m1.149s
    user  58m4.662s
    sys   11m28.563s

After split:

    real   1m31.977s
    user  57m45.993s
    sys   11m33.634s


Last test to finish (using run-test.py --time)
----------------------------------------------

test-largefile.t is now finishing at the same time than other slow tests.

Before split:

    Time      Test
    119.280   test-largefiles.t
     93.995   test-mq.t
     89.897   test-subrepo.t
     86.920   test-glog.t
     85.508   test-rename-merge2.t
     83.594   test-revset.t
     79.824   test-keyword.t
     78.077   test-mq-header-date.t

After split:

    Time      Test
    90.414   test-mq.t
    88.594   test-largefiles.t
    85.363   test-subrepo.t
    81.059   test-glog.t
    78.927   test-rename-merge2.t
    78.021   test-revset.t
    77.777   test-command-template.t



Timing of largefile test themself
-----------------------------------

Running only tests prefixed with "test-largefiles".
No significant change in cumulated time.

Before:

    Time     Test
    58.673   test-largefiles.t
     2.931   test-largefiles-cache.t
     0.583   test-largefiles-small-disk.t

After:

    Time     Test
    31.754   test-largefiles.t
    17.460   test-largefiles-misc.t
     8.888   test-largefiles-wireproto.t
     2.864   test-largefiles-cache.t
     0.580   test-largefiles-small-disk.t

# dicthelpers.py - helper routines for Python dicts
#
# Copyright 2013 Facebook
#
# This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version.

def diff(d1, d2, default=None):
    '''Return all key-value pairs that are different between d1 and d2.

    This includes keys that are present in one dict but not the other, and
    keys whose values are different. The return value is a dict with values
    being pairs of values from d1 and d2 respectively, and missing values
    treated as default, so if a value is missing from one dict and the same as
    default in the other, it will not be returned.'''
    res = {}
    if d1 is d2:
        # same dict, so diff is empty
        return res

    for k1, v1 in d1.iteritems():
        v2 = d2.get(k1, default)
        if v1 != v2:
            res[k1] = (v1, v2)

    for k2 in d2:
        if k2 not in d1:
            v2 = d2[k2]
            if v2 != default:
                res[k2] = (default, v2)

    return res

def join(d1, d2, default=None):
    '''Return all key-value pairs from both d1 and d2.

    This is akin to an outer join in relational algebra. The return value is a
    dict with values being pairs of values from d1 and d2 respectively, and
    missing values represented as default.'''
    res = {}

    for k1, v1 in d1.iteritems():
        if k1 in d2:
            res[k1] = (v1, d2[k1])
        else:
            res[k1] = (v1, default)

    if d1 is d2:
        return res

    for k2 in d2:
        if k2 not in d1:
            res[k2] = (default, d2[k2])

    return res