mercurial/changegroup.py
author Matt Mackall <mpm@selenic.com>
Sun, 03 Feb 2008 19:29:05 -0600
changeset 6009 f077815932ce
parent 5906 0136d7f58982
child 6152 c050548307a4
permissions -rw-r--r--
filemerge: remove the hgmerge script This is now better handled using [merge-tools] and [merge-patterns] in hgrc.
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
1981
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
     1
"""
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
     2
changegroup.py - Mercurial changegroup manipulation functions
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
     3
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
     4
 Copyright 2006 Matt Mackall <mpm@selenic.com>
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
     5
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
     6
This software may be used and distributed according to the terms
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
     7
of the GNU General Public License, incorporated herein by reference.
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
     8
"""
3877
abaee83ce0a6 Replace demandload with new demandimport
Matt Mackall <mpm@selenic.com>
parents: 3859
diff changeset
     9
3891
6b4127c7d52a Simplify i18n imports
Matt Mackall <mpm@selenic.com>
parents: 3877
diff changeset
    10
from i18n import _
3877
abaee83ce0a6 Replace demandload with new demandimport
Matt Mackall <mpm@selenic.com>
parents: 3859
diff changeset
    11
import struct, os, bz2, zlib, util, tempfile
1981
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    12
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    13
def getchunk(source):
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    14
    """get a chunk from a changegroup"""
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    15
    d = source.read(4)
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    16
    if not d:
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    17
        return ""
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    18
    l = struct.unpack(">l", d)[0]
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    19
    if l <= 4:
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    20
        return ""
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    21
    d = source.read(l - 4)
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    22
    if len(d) < l - 4:
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    23
        raise util.Abort(_("premature EOF reading chunk"
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    24
                           " (got %d bytes, expected %d)")
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    25
                          % (len(d), l - 4))
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    26
    return d
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    27
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    28
def chunkiter(source):
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    29
    """iterate through the chunks in source"""
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    30
    while 1:
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    31
        c = getchunk(source)
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    32
        if not c:
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    33
            break
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    34
        yield c
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    35
5368
61462e7d62ed changegroup: avoid large copies
Matt Mackall <mpm@selenic.com>
parents: 3932
diff changeset
    36
def chunkheader(length):
61462e7d62ed changegroup: avoid large copies
Matt Mackall <mpm@selenic.com>
parents: 3932
diff changeset
    37
    """build a changegroup chunk header"""
61462e7d62ed changegroup: avoid large copies
Matt Mackall <mpm@selenic.com>
parents: 3932
diff changeset
    38
    return struct.pack(">l", length + 4)
1981
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    39
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    40
def closechunk():
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    41
    return struct.pack(">l", 0)
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    42
3659
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    43
class nocompress(object):
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    44
    def compress(self, x):
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    45
        return x
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    46
    def flush(self):
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    47
        return ""
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    48
3662
f4dc02d7fb71 unduplicate bundle writing code from httprepo
Matt Mackall <mpm@selenic.com>
parents: 3660
diff changeset
    49
bundletypes = {
3704
9c1737a3e254 fix writebundle for bz2 bundles
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 3662
diff changeset
    50
    "": ("", nocompress),
9c1737a3e254 fix writebundle for bz2 bundles
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 3662
diff changeset
    51
    "HG10UN": ("HG10UN", nocompress),
3762
b9d3e12da485 changegroup.py: delay the loading of the bz2 and zlib modules
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 3706
diff changeset
    52
    "HG10BZ": ("HG10", lambda: bz2.BZ2Compressor()),
b9d3e12da485 changegroup.py: delay the loading of the bz2 and zlib modules
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 3706
diff changeset
    53
    "HG10GZ": ("HG10GZ", lambda: zlib.compressobj()),
3662
f4dc02d7fb71 unduplicate bundle writing code from httprepo
Matt Mackall <mpm@selenic.com>
parents: 3660
diff changeset
    54
}
f4dc02d7fb71 unduplicate bundle writing code from httprepo
Matt Mackall <mpm@selenic.com>
parents: 3660
diff changeset
    55
3706
0d810798acb1 Use 'bundletype' instead of 'type' to not shadow built-in function.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 3705
diff changeset
    56
def writebundle(cg, filename, bundletype):
3659
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    57
    """Write a bundle file and return its filename.
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    58
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    59
    Existing files will not be overwritten.
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    60
    If no filename is specified, a temporary file is created.
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    61
    bz2 compression can be turned off.
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    62
    The bundle file will be deleted in case of errors.
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    63
    """
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    64
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    65
    fh = None
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    66
    cleanup = None
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    67
    try:
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    68
        if filename:
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    69
            fh = open(filename, "wb")
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    70
        else:
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    71
            fd, filename = tempfile.mkstemp(prefix="hg-bundle-", suffix=".hg")
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    72
            fh = os.fdopen(fd, "wb")
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    73
        cleanup = filename
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    74
3706
0d810798acb1 Use 'bundletype' instead of 'type' to not shadow built-in function.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 3705
diff changeset
    75
        header, compressor = bundletypes[bundletype]
3704
9c1737a3e254 fix writebundle for bz2 bundles
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 3662
diff changeset
    76
        fh.write(header)
9c1737a3e254 fix writebundle for bz2 bundles
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 3662
diff changeset
    77
        z = compressor()
3662
f4dc02d7fb71 unduplicate bundle writing code from httprepo
Matt Mackall <mpm@selenic.com>
parents: 3660
diff changeset
    78
3659
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    79
        # parse the changegroup data, otherwise we will block
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    80
        # in case of sshrepo because we don't know the end of the stream
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    81
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    82
        # an empty chunkiter is the end of the changegroup
5906
0136d7f58982 allow the creation of bundles with empty changelog/manifest chunks
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 5368
diff changeset
    83
        # a changegroup has at least 2 chunkiters (changelog and manifest).
0136d7f58982 allow the creation of bundles with empty changelog/manifest chunks
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 5368
diff changeset
    84
        # after that, an empty chunkiter is the end of the changegroup
3659
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    85
        empty = False
5906
0136d7f58982 allow the creation of bundles with empty changelog/manifest chunks
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 5368
diff changeset
    86
        count = 0
0136d7f58982 allow the creation of bundles with empty changelog/manifest chunks
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 5368
diff changeset
    87
        while not empty or count <= 2:
3659
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    88
            empty = True
5906
0136d7f58982 allow the creation of bundles with empty changelog/manifest chunks
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 5368
diff changeset
    89
            count += 1
3659
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    90
            for chunk in chunkiter(cg):
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    91
                empty = False
5368
61462e7d62ed changegroup: avoid large copies
Matt Mackall <mpm@selenic.com>
parents: 3932
diff changeset
    92
                fh.write(z.compress(chunkheader(len(chunk))))
61462e7d62ed changegroup: avoid large copies
Matt Mackall <mpm@selenic.com>
parents: 3932
diff changeset
    93
                pos = 0
61462e7d62ed changegroup: avoid large copies
Matt Mackall <mpm@selenic.com>
parents: 3932
diff changeset
    94
                while pos < len(chunk):
61462e7d62ed changegroup: avoid large copies
Matt Mackall <mpm@selenic.com>
parents: 3932
diff changeset
    95
                    next = pos + 2**20
61462e7d62ed changegroup: avoid large copies
Matt Mackall <mpm@selenic.com>
parents: 3932
diff changeset
    96
                    fh.write(z.compress(chunk[pos:next]))
61462e7d62ed changegroup: avoid large copies
Matt Mackall <mpm@selenic.com>
parents: 3932
diff changeset
    97
                    pos = next
3659
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    98
            fh.write(z.compress(closechunk()))
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    99
        fh.write(z.flush())
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
   100
        cleanup = None
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
   101
        return filename
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
   102
    finally:
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
   103
        if fh is not None:
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
   104
            fh.close()
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
   105
        if cleanup is not None:
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
   106
            os.unlink(cleanup)
3660
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3659
diff changeset
   107
3859
8c24b6fd5866 fix errors spotted by pychecker
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 3762
diff changeset
   108
def readbundle(fh, fname):
3660
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3659
diff changeset
   109
    header = fh.read(6)
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3659
diff changeset
   110
    if not header.startswith("HG"):
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3659
diff changeset
   111
        raise util.Abort(_("%s: not a Mercurial bundle file") % fname)
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3659
diff changeset
   112
    elif not header.startswith("HG10"):
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3659
diff changeset
   113
        raise util.Abort(_("%s: unknown bundle version") % fname)
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3659
diff changeset
   114
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3659
diff changeset
   115
    if header == "HG10BZ":
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3659
diff changeset
   116
        def generator(f):
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3659
diff changeset
   117
            zd = bz2.BZ2Decompressor()
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3659
diff changeset
   118
            zd.decompress("BZ")
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3659
diff changeset
   119
            for chunk in util.filechunkiter(f, 4096):
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3659
diff changeset
   120
                yield zd.decompress(chunk)
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3659
diff changeset
   121
        return util.chunkbuffer(generator(fh))
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3659
diff changeset
   122
    elif header == "HG10UN":
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3659
diff changeset
   123
        return fh
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3659
diff changeset
   124
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3659
diff changeset
   125
    raise util.Abort(_("%s: unknown bundle compression type")
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3659
diff changeset
   126
                     % fname)