mercurial/changegroup.py
author Martin von Zweigbergk <martinvonz@google.com>
Wed, 27 Jan 2016 09:07:28 -0800
branchstable
changeset 27953 88609cfa3745
parent 27946 ca8d2b73155d
child 28227 1c36cc8e7870
permissions -rw-r--r--
changegroup: fix pulling to treemanifest repo from flat repo (issue5066) In c0f11347b107 (changegroup: don't support versions 01 and 02 with treemanifests, 2016-01-19), I stopped supporting use of cg1 and cg2 with treemanifest repos. What I had not considered was that it's perfectly safe to pull *to* a treemanifest repo using any changegroup version. As reported in issue5066, I therefore broke pull from old repos into a treemanifest repo. It was not covered by the test case, because that pulled from a local repo while enabling treemanifests, which enabled treemanifests on the source repo as well. After switching to pulling via HTTP, it breaks. Fix by splitting up changegroup.supportedversions() into supportedincomingversions() and supportedoutgoingversions().
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
8226
8b2cd04a6e97 put license and copyright info into comment blocks
Martin Geisler <mg@lazybytes.net>
parents: 8225
diff changeset
     1
# changegroup.py - Mercurial changegroup manipulation functions
8b2cd04a6e97 put license and copyright info into comment blocks
Martin Geisler <mg@lazybytes.net>
parents: 8225
diff changeset
     2
#
8b2cd04a6e97 put license and copyright info into comment blocks
Martin Geisler <mg@lazybytes.net>
parents: 8225
diff changeset
     3
#  Copyright 2006 Matt Mackall <mpm@selenic.com>
8b2cd04a6e97 put license and copyright info into comment blocks
Martin Geisler <mg@lazybytes.net>
parents: 8225
diff changeset
     4
#
8b2cd04a6e97 put license and copyright info into comment blocks
Martin Geisler <mg@lazybytes.net>
parents: 8225
diff changeset
     5
# This software may be used and distributed according to the terms of the
10263
25e572394f5c Update license to GPLv2+
Matt Mackall <mpm@selenic.com>
parents: 9437
diff changeset
     6
# GNU General Public License version 2 or any later version.
3877
abaee83ce0a6 Replace demandload with new demandimport
Matt Mackall <mpm@selenic.com>
parents: 3859
diff changeset
     7
25921
74b303a637bc changegroup: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25831
diff changeset
     8
from __future__ import absolute_import
74b303a637bc changegroup: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25831
diff changeset
     9
74b303a637bc changegroup: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25831
diff changeset
    10
import os
74b303a637bc changegroup: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25831
diff changeset
    11
import struct
74b303a637bc changegroup: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25831
diff changeset
    12
import tempfile
20933
d3775db748a0 localrepo: move the addchangegroup method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20932
diff changeset
    13
import weakref
25921
74b303a637bc changegroup: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25831
diff changeset
    14
74b303a637bc changegroup: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25831
diff changeset
    15
from .i18n import _
74b303a637bc changegroup: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25831
diff changeset
    16
from .node import (
74b303a637bc changegroup: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25831
diff changeset
    17
    hex,
74b303a637bc changegroup: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25831
diff changeset
    18
    nullid,
74b303a637bc changegroup: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25831
diff changeset
    19
    nullrev,
74b303a637bc changegroup: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25831
diff changeset
    20
    short,
74b303a637bc changegroup: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25831
diff changeset
    21
)
74b303a637bc changegroup: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25831
diff changeset
    22
74b303a637bc changegroup: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25831
diff changeset
    23
from . import (
74b303a637bc changegroup: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25831
diff changeset
    24
    branchmap,
74b303a637bc changegroup: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25831
diff changeset
    25
    dagutil,
74b303a637bc changegroup: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25831
diff changeset
    26
    discovery,
74b303a637bc changegroup: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25831
diff changeset
    27
    error,
74b303a637bc changegroup: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25831
diff changeset
    28
    mdiff,
74b303a637bc changegroup: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25831
diff changeset
    29
    phases,
74b303a637bc changegroup: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25831
diff changeset
    30
    util,
74b303a637bc changegroup: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25831
diff changeset
    31
)
1981
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    32
22390
e2806b8613ca changegroup: rename bundle-related functions and classes
Sune Foldager <cryo@cyanite.org>
parents: 22070
diff changeset
    33
_CHANGEGROUPV1_DELTA_HEADER = "20s20s20s20s"
23181
832b7ef275c8 changegroup: introduce cg2packer/unpacker
Sune Foldager <cryo@cyanite.org>
parents: 23178
diff changeset
    34
_CHANGEGROUPV2_DELTA_HEADER = "20s20s20s20s20s"
27433
12f727a5b434 changegroup: add flags field to cg3 delta header
Mike Edgar <adgar@google.com>
parents: 27432
diff changeset
    35
_CHANGEGROUPV3_DELTA_HEADER = ">20s20s20s20s20sH"
14141
bd1cbfe5db5c bundler: make parsechunk return the base revision of the delta
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14060
diff changeset
    36
13457
e74fe15dc7fd changegroup: verify all stream reads
Mads Kiilerich <mads@kiilerich.com>
parents: 13456
diff changeset
    37
def readexactly(stream, n):
e74fe15dc7fd changegroup: verify all stream reads
Mads Kiilerich <mads@kiilerich.com>
parents: 13456
diff changeset
    38
    '''read n bytes from stream.read and abort if less was available'''
e74fe15dc7fd changegroup: verify all stream reads
Mads Kiilerich <mads@kiilerich.com>
parents: 13456
diff changeset
    39
    s = stream.read(n)
e74fe15dc7fd changegroup: verify all stream reads
Mads Kiilerich <mads@kiilerich.com>
parents: 13456
diff changeset
    40
    if len(s) < n:
26587
56b2bcea2529 error: get Abort from 'error' instead of 'util'
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26540
diff changeset
    41
        raise error.Abort(_("stream ended unexpectedly"
13457
e74fe15dc7fd changegroup: verify all stream reads
Mads Kiilerich <mads@kiilerich.com>
parents: 13456
diff changeset
    42
                           " (got %d bytes, expected %d)")
e74fe15dc7fd changegroup: verify all stream reads
Mads Kiilerich <mads@kiilerich.com>
parents: 13456
diff changeset
    43
                          % (len(s), n))
e74fe15dc7fd changegroup: verify all stream reads
Mads Kiilerich <mads@kiilerich.com>
parents: 13456
diff changeset
    44
    return s
e74fe15dc7fd changegroup: verify all stream reads
Mads Kiilerich <mads@kiilerich.com>
parents: 13456
diff changeset
    45
e74fe15dc7fd changegroup: verify all stream reads
Mads Kiilerich <mads@kiilerich.com>
parents: 13456
diff changeset
    46
def getchunk(stream):
e74fe15dc7fd changegroup: verify all stream reads
Mads Kiilerich <mads@kiilerich.com>
parents: 13456
diff changeset
    47
    """return the next chunk from stream as a string"""
e74fe15dc7fd changegroup: verify all stream reads
Mads Kiilerich <mads@kiilerich.com>
parents: 13456
diff changeset
    48
    d = readexactly(stream, 4)
1981
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    49
    l = struct.unpack(">l", d)[0]
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    50
    if l <= 4:
13458
9f2c407caf34 changegroup: don't accept odd chunk headers
Mads Kiilerich <mads@kiilerich.com>
parents: 13457
diff changeset
    51
        if l:
26587
56b2bcea2529 error: get Abort from 'error' instead of 'util'
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26540
diff changeset
    52
            raise error.Abort(_("invalid chunk length %d") % l)
1981
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    53
        return ""
13457
e74fe15dc7fd changegroup: verify all stream reads
Mads Kiilerich <mads@kiilerich.com>
parents: 13456
diff changeset
    54
    return readexactly(stream, l - 4)
1981
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    55
5368
61462e7d62ed changegroup: avoid large copies
Matt Mackall <mpm@selenic.com>
parents: 3932
diff changeset
    56
def chunkheader(length):
9437
1c4e4004f3a6 Improve some docstrings relating to changegroups and prepush().
Greg Ward <greg-hg@gerg.ca>
parents: 9087
diff changeset
    57
    """return a changegroup chunk header (string)"""
5368
61462e7d62ed changegroup: avoid large copies
Matt Mackall <mpm@selenic.com>
parents: 3932
diff changeset
    58
    return struct.pack(">l", length + 4)
1981
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    59
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    60
def closechunk():
9437
1c4e4004f3a6 Improve some docstrings relating to changegroups and prepush().
Greg Ward <greg-hg@gerg.ca>
parents: 9087
diff changeset
    61
    """return a changegroup chunk header (string) for a zero-length chunk"""
1981
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    62
    return struct.pack(">l", 0)
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    63
23890
7817059917d0 pullbundle2: extract addchangegroup result combining into its own function
Eric Sumner <ericsumner@fb.com>
parents: 23748
diff changeset
    64
def combineresults(results):
7817059917d0 pullbundle2: extract addchangegroup result combining into its own function
Eric Sumner <ericsumner@fb.com>
parents: 23748
diff changeset
    65
    """logic to combine 0 or more addchangegroup results into one"""
7817059917d0 pullbundle2: extract addchangegroup result combining into its own function
Eric Sumner <ericsumner@fb.com>
parents: 23748
diff changeset
    66
    changedheads = 0
7817059917d0 pullbundle2: extract addchangegroup result combining into its own function
Eric Sumner <ericsumner@fb.com>
parents: 23748
diff changeset
    67
    result = 1
7817059917d0 pullbundle2: extract addchangegroup result combining into its own function
Eric Sumner <ericsumner@fb.com>
parents: 23748
diff changeset
    68
    for ret in results:
7817059917d0 pullbundle2: extract addchangegroup result combining into its own function
Eric Sumner <ericsumner@fb.com>
parents: 23748
diff changeset
    69
        # If any changegroup result is 0, return 0
7817059917d0 pullbundle2: extract addchangegroup result combining into its own function
Eric Sumner <ericsumner@fb.com>
parents: 23748
diff changeset
    70
        if ret == 0:
7817059917d0 pullbundle2: extract addchangegroup result combining into its own function
Eric Sumner <ericsumner@fb.com>
parents: 23748
diff changeset
    71
            result = 0
7817059917d0 pullbundle2: extract addchangegroup result combining into its own function
Eric Sumner <ericsumner@fb.com>
parents: 23748
diff changeset
    72
            break
7817059917d0 pullbundle2: extract addchangegroup result combining into its own function
Eric Sumner <ericsumner@fb.com>
parents: 23748
diff changeset
    73
        if ret < -1:
7817059917d0 pullbundle2: extract addchangegroup result combining into its own function
Eric Sumner <ericsumner@fb.com>
parents: 23748
diff changeset
    74
            changedheads += ret + 1
7817059917d0 pullbundle2: extract addchangegroup result combining into its own function
Eric Sumner <ericsumner@fb.com>
parents: 23748
diff changeset
    75
        elif ret > 1:
7817059917d0 pullbundle2: extract addchangegroup result combining into its own function
Eric Sumner <ericsumner@fb.com>
parents: 23748
diff changeset
    76
            changedheads += ret - 1
7817059917d0 pullbundle2: extract addchangegroup result combining into its own function
Eric Sumner <ericsumner@fb.com>
parents: 23748
diff changeset
    77
    if changedheads > 0:
7817059917d0 pullbundle2: extract addchangegroup result combining into its own function
Eric Sumner <ericsumner@fb.com>
parents: 23748
diff changeset
    78
        result = 1 + changedheads
7817059917d0 pullbundle2: extract addchangegroup result combining into its own function
Eric Sumner <ericsumner@fb.com>
parents: 23748
diff changeset
    79
    elif changedheads < 0:
7817059917d0 pullbundle2: extract addchangegroup result combining into its own function
Eric Sumner <ericsumner@fb.com>
parents: 23748
diff changeset
    80
        result = -1 + changedheads
7817059917d0 pullbundle2: extract addchangegroup result combining into its own function
Eric Sumner <ericsumner@fb.com>
parents: 23748
diff changeset
    81
    return result
7817059917d0 pullbundle2: extract addchangegroup result combining into its own function
Eric Sumner <ericsumner@fb.com>
parents: 23748
diff changeset
    82
3662
f4dc02d7fb71 unduplicate bundle writing code from httprepo
Matt Mackall <mpm@selenic.com>
parents: 3660
diff changeset
    83
bundletypes = {
26272
59c410db8c68 readbundle: fix typo of None compression
Yuya Nishihara <yuya@tcha.org>
parents: 26271
diff changeset
    84
    "": ("", None),       # only when using unbundle on ssh and old http servers
14060
aaa9a5989405 bundle: more comments about the different header types, remove useless if
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 13831
diff changeset
    85
                          # since the unification ssh accepts a header but there
aaa9a5989405 bundle: more comments about the different header types, remove useless if
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 13831
diff changeset
    86
                          # is no capability signaling it.
24686
e0e28e910fa3 bundle2: rename format, parts and config to final names
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24190
diff changeset
    87
    "HG20": (), # special-cased below
26271
a0eff7ebc2ae readbundle: map 'HG10UN' to None compression
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26270
diff changeset
    88
    "HG10UN": ("HG10UN", None),
26266
1e042e31bd0c changegroup: move all compressions utilities in util
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25921
diff changeset
    89
    "HG10BZ": ("HG10", 'BZ'),
1e042e31bd0c changegroup: move all compressions utilities in util
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25921
diff changeset
    90
    "HG10GZ": ("HG10GZ", 'GZ'),
3662
f4dc02d7fb71 unduplicate bundle writing code from httprepo
Matt Mackall <mpm@selenic.com>
parents: 3660
diff changeset
    91
}
f4dc02d7fb71 unduplicate bundle writing code from httprepo
Matt Mackall <mpm@selenic.com>
parents: 3660
diff changeset
    92
9087
f48454a279b9 typos: "it's" -> "its"
Martin Geisler <mg@lazybytes.net>
parents: 8312
diff changeset
    93
# hgweb uses this list to communicate its preferred type
6152
c050548307a4 hgweb: use bundletypes from mercurial.changegroup
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents: 5906
diff changeset
    94
bundlepriority = ['HG10GZ', 'HG10BZ', 'HG10UN']
c050548307a4 hgweb: use bundletypes from mercurial.changegroup
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents: 5906
diff changeset
    95
26540
7469067de2ba changegroup: extract the file management part in its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26509
diff changeset
    96
def writechunks(ui, chunks, filename, vfs=None):
7469067de2ba changegroup: extract the file management part in its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26509
diff changeset
    97
    """Write chunks to a file and return its filename.
3659
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    98
26540
7469067de2ba changegroup: extract the file management part in its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26509
diff changeset
    99
    The stream is assumed to be a bundle file.
3659
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
   100
    Existing files will not be overwritten.
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
   101
    If no filename is specified, a temporary file is created.
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
   102
    """
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
   103
    fh = None
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
   104
    cleanup = None
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
   105
    try:
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
   106
        if filename:
20976
c20f4898631e changegroup: add "vfs" argument to "writebundle()" for relative access via vfs
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 20966
diff changeset
   107
            if vfs:
c20f4898631e changegroup: add "vfs" argument to "writebundle()" for relative access via vfs
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 20966
diff changeset
   108
                fh = vfs.open(filename, "wb")
c20f4898631e changegroup: add "vfs" argument to "writebundle()" for relative access via vfs
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 20966
diff changeset
   109
            else:
c20f4898631e changegroup: add "vfs" argument to "writebundle()" for relative access via vfs
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 20966
diff changeset
   110
                fh = open(filename, "wb")
3659
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
   111
        else:
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
   112
            fd, filename = tempfile.mkstemp(prefix="hg-bundle-", suffix=".hg")
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
   113
            fh = os.fdopen(fd, "wb")
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
   114
        cleanup = filename
26540
7469067de2ba changegroup: extract the file management part in its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26509
diff changeset
   115
        for c in chunks:
7469067de2ba changegroup: extract the file management part in its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26509
diff changeset
   116
            fh.write(c)
3659
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
   117
        cleanup = None
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
   118
        return filename
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
   119
    finally:
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
   120
        if fh is not None:
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
   121
            fh.close()
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
   122
        if cleanup is not None:
20976
c20f4898631e changegroup: add "vfs" argument to "writebundle()" for relative access via vfs
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 20966
diff changeset
   123
            if filename and vfs:
c20f4898631e changegroup: add "vfs" argument to "writebundle()" for relative access via vfs
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 20966
diff changeset
   124
                vfs.unlink(cleanup)
c20f4898631e changegroup: add "vfs" argument to "writebundle()" for relative access via vfs
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 20966
diff changeset
   125
            else:
c20f4898631e changegroup: add "vfs" argument to "writebundle()" for relative access via vfs
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 20966
diff changeset
   126
                os.unlink(cleanup)
3660
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3659
diff changeset
   127
26540
7469067de2ba changegroup: extract the file management part in its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26509
diff changeset
   128
def writebundle(ui, cg, filename, bundletype, vfs=None, compression=None):
7469067de2ba changegroup: extract the file management part in its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26509
diff changeset
   129
    """Write a bundle file and return its filename.
7469067de2ba changegroup: extract the file management part in its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26509
diff changeset
   130
7469067de2ba changegroup: extract the file management part in its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26509
diff changeset
   131
    Existing files will not be overwritten.
7469067de2ba changegroup: extract the file management part in its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26509
diff changeset
   132
    If no filename is specified, a temporary file is created.
7469067de2ba changegroup: extract the file management part in its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26509
diff changeset
   133
    bz2 compression can be turned off.
7469067de2ba changegroup: extract the file management part in its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26509
diff changeset
   134
    The bundle file will be deleted in case of errors.
7469067de2ba changegroup: extract the file management part in its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26509
diff changeset
   135
    """
7469067de2ba changegroup: extract the file management part in its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26509
diff changeset
   136
7469067de2ba changegroup: extract the file management part in its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26509
diff changeset
   137
    if bundletype == "HG20":
7469067de2ba changegroup: extract the file management part in its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26509
diff changeset
   138
        from . import bundle2
7469067de2ba changegroup: extract the file management part in its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26509
diff changeset
   139
        bundle = bundle2.bundle20(ui)
7469067de2ba changegroup: extract the file management part in its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26509
diff changeset
   140
        bundle.setcompression(compression)
7469067de2ba changegroup: extract the file management part in its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26509
diff changeset
   141
        part = bundle.newpart('changegroup', data=cg.getchunks())
7469067de2ba changegroup: extract the file management part in its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26509
diff changeset
   142
        part.addparam('version', cg.version)
7469067de2ba changegroup: extract the file management part in its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26509
diff changeset
   143
        chunkiter = bundle.getchunks()
7469067de2ba changegroup: extract the file management part in its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26509
diff changeset
   144
    else:
7469067de2ba changegroup: extract the file management part in its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26509
diff changeset
   145
        # compression argument is only for the bundle2 case
7469067de2ba changegroup: extract the file management part in its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26509
diff changeset
   146
        assert compression is None
7469067de2ba changegroup: extract the file management part in its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26509
diff changeset
   147
        if cg.version != '01':
26587
56b2bcea2529 error: get Abort from 'error' instead of 'util'
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26540
diff changeset
   148
            raise error.Abort(_('old bundle types only supports v1 '
56b2bcea2529 error: get Abort from 'error' instead of 'util'
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26540
diff changeset
   149
                                'changegroups'))
26540
7469067de2ba changegroup: extract the file management part in its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26509
diff changeset
   150
        header, comp = bundletypes[bundletype]
7469067de2ba changegroup: extract the file management part in its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26509
diff changeset
   151
        if comp not in util.compressors:
26587
56b2bcea2529 error: get Abort from 'error' instead of 'util'
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26540
diff changeset
   152
            raise error.Abort(_('unknown stream compression type: %s')
56b2bcea2529 error: get Abort from 'error' instead of 'util'
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26540
diff changeset
   153
                              % comp)
26540
7469067de2ba changegroup: extract the file management part in its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26509
diff changeset
   154
        z = util.compressors[comp]()
7469067de2ba changegroup: extract the file management part in its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26509
diff changeset
   155
        subchunkiter = cg.getchunks()
7469067de2ba changegroup: extract the file management part in its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26509
diff changeset
   156
        def chunkiter():
7469067de2ba changegroup: extract the file management part in its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26509
diff changeset
   157
            yield header
7469067de2ba changegroup: extract the file management part in its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26509
diff changeset
   158
            for chunk in subchunkiter:
7469067de2ba changegroup: extract the file management part in its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26509
diff changeset
   159
                yield z.compress(chunk)
7469067de2ba changegroup: extract the file management part in its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26509
diff changeset
   160
            yield z.flush()
7469067de2ba changegroup: extract the file management part in its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26509
diff changeset
   161
        chunkiter = chunkiter()
7469067de2ba changegroup: extract the file management part in its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26509
diff changeset
   162
7469067de2ba changegroup: extract the file management part in its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26509
diff changeset
   163
    # parse the changegroup data, otherwise we will block
7469067de2ba changegroup: extract the file management part in its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26509
diff changeset
   164
    # in case of sshrepo because we don't know the end of the stream
7469067de2ba changegroup: extract the file management part in its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26509
diff changeset
   165
7469067de2ba changegroup: extract the file management part in its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26509
diff changeset
   166
    # an empty chunkgroup is the end of the changegroup
7469067de2ba changegroup: extract the file management part in its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26509
diff changeset
   167
    # a changegroup has at least 2 chunkgroups (changelog and manifest).
7469067de2ba changegroup: extract the file management part in its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26509
diff changeset
   168
    # after that, an empty chunkgroup is the end of the changegroup
7469067de2ba changegroup: extract the file management part in its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26509
diff changeset
   169
    return writechunks(ui, chunkiter, filename, vfs=vfs)
7469067de2ba changegroup: extract the file management part in its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26509
diff changeset
   170
22390
e2806b8613ca changegroup: rename bundle-related functions and classes
Sune Foldager <cryo@cyanite.org>
parents: 22070
diff changeset
   171
class cg1unpacker(object):
26708
749d913f24b8 changegroup: document the public surface area of cg?unpackers
Augie Fackler <augie@google.com>
parents: 26707
diff changeset
   172
    """Unpacker for cg1 changegroup streams.
749d913f24b8 changegroup: document the public surface area of cg?unpackers
Augie Fackler <augie@google.com>
parents: 26707
diff changeset
   173
749d913f24b8 changegroup: document the public surface area of cg?unpackers
Augie Fackler <augie@google.com>
parents: 26707
diff changeset
   174
    A changegroup unpacker handles the framing of the revision data in
749d913f24b8 changegroup: document the public surface area of cg?unpackers
Augie Fackler <augie@google.com>
parents: 26707
diff changeset
   175
    the wire format. Most consumers will want to use the apply()
749d913f24b8 changegroup: document the public surface area of cg?unpackers
Augie Fackler <augie@google.com>
parents: 26707
diff changeset
   176
    method to add the changes from the changegroup to a repository.
749d913f24b8 changegroup: document the public surface area of cg?unpackers
Augie Fackler <augie@google.com>
parents: 26707
diff changeset
   177
749d913f24b8 changegroup: document the public surface area of cg?unpackers
Augie Fackler <augie@google.com>
parents: 26707
diff changeset
   178
    If you're forwarding a changegroup unmodified to another consumer,
749d913f24b8 changegroup: document the public surface area of cg?unpackers
Augie Fackler <augie@google.com>
parents: 26707
diff changeset
   179
    use getchunks(), which returns an iterator of changegroup
749d913f24b8 changegroup: document the public surface area of cg?unpackers
Augie Fackler <augie@google.com>
parents: 26707
diff changeset
   180
    chunks. This is mostly useful for cases where you need to know the
749d913f24b8 changegroup: document the public surface area of cg?unpackers
Augie Fackler <augie@google.com>
parents: 26707
diff changeset
   181
    data stream has ended by observing the end of the changegroup.
749d913f24b8 changegroup: document the public surface area of cg?unpackers
Augie Fackler <augie@google.com>
parents: 26707
diff changeset
   182
749d913f24b8 changegroup: document the public surface area of cg?unpackers
Augie Fackler <augie@google.com>
parents: 26707
diff changeset
   183
    deltachunk() is useful only if you're applying delta data. Most
749d913f24b8 changegroup: document the public surface area of cg?unpackers
Augie Fackler <augie@google.com>
parents: 26707
diff changeset
   184
    consumers should prefer apply() instead.
749d913f24b8 changegroup: document the public surface area of cg?unpackers
Augie Fackler <augie@google.com>
parents: 26707
diff changeset
   185
749d913f24b8 changegroup: document the public surface area of cg?unpackers
Augie Fackler <augie@google.com>
parents: 26707
diff changeset
   186
    A few other public methods exist. Those are used only for
749d913f24b8 changegroup: document the public surface area of cg?unpackers
Augie Fackler <augie@google.com>
parents: 26707
diff changeset
   187
    bundlerepo and some debug commands - their use is discouraged.
749d913f24b8 changegroup: document the public surface area of cg?unpackers
Augie Fackler <augie@google.com>
parents: 26707
diff changeset
   188
    """
22390
e2806b8613ca changegroup: rename bundle-related functions and classes
Sune Foldager <cryo@cyanite.org>
parents: 22070
diff changeset
   189
    deltaheader = _CHANGEGROUPV1_DELTA_HEADER
14141
bd1cbfe5db5c bundler: make parsechunk return the base revision of the delta
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14060
diff changeset
   190
    deltaheadersize = struct.calcsize(deltaheader)
23896
becfecaf9087 changegroup.writebundle: HG2Y support
Eric Sumner <ericsumner@fb.com>
parents: 23895
diff changeset
   191
    version = '01'
27920
da5f23362517 changegroup: cg3 has two empty groups *after* manifests
Martin von Zweigbergk <martinvonz@google.com>
parents: 27867
diff changeset
   192
    _grouplistcount = 1 # One list of files after the manifests
da5f23362517 changegroup: cg3 has two empty groups *after* manifests
Martin von Zweigbergk <martinvonz@google.com>
parents: 27867
diff changeset
   193
12043
bef5effb3db0 bundle: introduce bundle class
Matt Mackall <mpm@selenic.com>
parents: 12042
diff changeset
   194
    def __init__(self, fh, alg):
26267
eca468b8fae4 compression: use 'None' for no-compression
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26266
diff changeset
   195
        if alg == 'UN':
eca468b8fae4 compression: use 'None' for no-compression
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26266
diff changeset
   196
            alg = None # get more modern without breaking too much
26266
1e042e31bd0c changegroup: move all compressions utilities in util
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25921
diff changeset
   197
        if not alg in util.decompressors:
26587
56b2bcea2529 error: get Abort from 'error' instead of 'util'
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26540
diff changeset
   198
            raise error.Abort(_('unknown stream compression type: %s')
26266
1e042e31bd0c changegroup: move all compressions utilities in util
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25921
diff changeset
   199
                             % alg)
26392
127b59787fd5 changegroup: use a different compression key for BZ in HG10
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26272
diff changeset
   200
        if alg == 'BZ':
127b59787fd5 changegroup: use a different compression key for BZ in HG10
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26272
diff changeset
   201
            alg = '_truncatedBZ'
26266
1e042e31bd0c changegroup: move all compressions utilities in util
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25921
diff changeset
   202
        self._stream = util.decompressors[alg](fh)
12044
bcc7139521b7 bundlerepo: remove duplication of bundle decompressors
Matt Mackall <mpm@selenic.com>
parents: 12043
diff changeset
   203
        self._type = alg
12334
50946802593d bundle: refactor progress callback
Matt Mackall <mpm@selenic.com>
parents: 12333
diff changeset
   204
        self.callback = None
26706
8c0c3059f478 changegroup: note why a few methods on cg1unpacker exist
Augie Fackler <augie@google.com>
parents: 26704
diff changeset
   205
8c0c3059f478 changegroup: note why a few methods on cg1unpacker exist
Augie Fackler <augie@google.com>
parents: 26704
diff changeset
   206
    # These methods (compressed, read, seek, tell) all appear to only
8c0c3059f478 changegroup: note why a few methods on cg1unpacker exist
Augie Fackler <augie@google.com>
parents: 26704
diff changeset
   207
    # be used by bundlerepo, but it's a little hard to tell.
12044
bcc7139521b7 bundlerepo: remove duplication of bundle decompressors
Matt Mackall <mpm@selenic.com>
parents: 12043
diff changeset
   208
    def compressed(self):
26267
eca468b8fae4 compression: use 'None' for no-compression
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26266
diff changeset
   209
        return self._type is not None
12043
bef5effb3db0 bundle: introduce bundle class
Matt Mackall <mpm@selenic.com>
parents: 12042
diff changeset
   210
    def read(self, l):
bef5effb3db0 bundle: introduce bundle class
Matt Mackall <mpm@selenic.com>
parents: 12042
diff changeset
   211
        return self._stream.read(l)
12330
e527b8635881 bundle: make unbundle object seekable
Matt Mackall <mpm@selenic.com>
parents: 12329
diff changeset
   212
    def seek(self, pos):
e527b8635881 bundle: make unbundle object seekable
Matt Mackall <mpm@selenic.com>
parents: 12329
diff changeset
   213
        return self._stream.seek(pos)
e527b8635881 bundle: make unbundle object seekable
Matt Mackall <mpm@selenic.com>
parents: 12329
diff changeset
   214
    def tell(self):
12332
680fe77ab5b8 bundlerepo: use bundle objects everywhere
Matt Mackall <mpm@selenic.com>
parents: 12330
diff changeset
   215
        return self._stream.tell()
12347
6277a9469dff bundlerepo: restore close() method
Matt Mackall <mpm@selenic.com>
parents: 12336
diff changeset
   216
    def close(self):
6277a9469dff bundlerepo: restore close() method
Matt Mackall <mpm@selenic.com>
parents: 12336
diff changeset
   217
        return self._stream.close()
12334
50946802593d bundle: refactor progress callback
Matt Mackall <mpm@selenic.com>
parents: 12333
diff changeset
   218
26707
5ee6bd529300 changegroup: mark cg1unpacker.chunklength as private
Augie Fackler <augie@google.com>
parents: 26706
diff changeset
   219
    def _chunklength(self):
13459
acbe171c8fbe changegroup: fix typo introduced in 9f2c407caf34
Jim Hague <jim.hague@acm.org>
parents: 13458
diff changeset
   220
        d = readexactly(self._stream, 4)
13458
9f2c407caf34 changegroup: don't accept odd chunk headers
Mads Kiilerich <mads@kiilerich.com>
parents: 13457
diff changeset
   221
        l = struct.unpack(">l", d)[0]
9f2c407caf34 changegroup: don't accept odd chunk headers
Mads Kiilerich <mads@kiilerich.com>
parents: 13457
diff changeset
   222
        if l <= 4:
9f2c407caf34 changegroup: don't accept odd chunk headers
Mads Kiilerich <mads@kiilerich.com>
parents: 13457
diff changeset
   223
            if l:
26587
56b2bcea2529 error: get Abort from 'error' instead of 'util'
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26540
diff changeset
   224
                raise error.Abort(_("invalid chunk length %d") % l)
13458
9f2c407caf34 changegroup: don't accept odd chunk headers
Mads Kiilerich <mads@kiilerich.com>
parents: 13457
diff changeset
   225
            return 0
9f2c407caf34 changegroup: don't accept odd chunk headers
Mads Kiilerich <mads@kiilerich.com>
parents: 13457
diff changeset
   226
        if self.callback:
12334
50946802593d bundle: refactor progress callback
Matt Mackall <mpm@selenic.com>
parents: 12333
diff changeset
   227
            self.callback()
13458
9f2c407caf34 changegroup: don't accept odd chunk headers
Mads Kiilerich <mads@kiilerich.com>
parents: 13457
diff changeset
   228
        return l - 4
12334
50946802593d bundle: refactor progress callback
Matt Mackall <mpm@selenic.com>
parents: 12333
diff changeset
   229
14144
3c3c53d8343a unbundler: separate delta and header parsing
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14143
diff changeset
   230
    def changelogheader(self):
3c3c53d8343a unbundler: separate delta and header parsing
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14143
diff changeset
   231
        """v10 does not have a changelog header chunk"""
3c3c53d8343a unbundler: separate delta and header parsing
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14143
diff changeset
   232
        return {}
3c3c53d8343a unbundler: separate delta and header parsing
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14143
diff changeset
   233
3c3c53d8343a unbundler: separate delta and header parsing
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14143
diff changeset
   234
    def manifestheader(self):
3c3c53d8343a unbundler: separate delta and header parsing
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14143
diff changeset
   235
        """v10 does not have a manifest header chunk"""
3c3c53d8343a unbundler: separate delta and header parsing
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14143
diff changeset
   236
        return {}
3c3c53d8343a unbundler: separate delta and header parsing
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14143
diff changeset
   237
3c3c53d8343a unbundler: separate delta and header parsing
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14143
diff changeset
   238
    def filelogheader(self):
3c3c53d8343a unbundler: separate delta and header parsing
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14143
diff changeset
   239
        """return the header of the filelogs chunk, v10 only has the filename"""
26707
5ee6bd529300 changegroup: mark cg1unpacker.chunklength as private
Augie Fackler <augie@google.com>
parents: 26706
diff changeset
   240
        l = self._chunklength()
14144
3c3c53d8343a unbundler: separate delta and header parsing
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14143
diff changeset
   241
        if not l:
3c3c53d8343a unbundler: separate delta and header parsing
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14143
diff changeset
   242
            return {}
3c3c53d8343a unbundler: separate delta and header parsing
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14143
diff changeset
   243
        fname = readexactly(self._stream, l)
20675
f8d50add83e1 changegroup: move from dict() construction to {} literals
Augie Fackler <raf@durin42.com>
parents: 19708
diff changeset
   244
        return {'filename': fname}
12334
50946802593d bundle: refactor progress callback
Matt Mackall <mpm@selenic.com>
parents: 12333
diff changeset
   245
14141
bd1cbfe5db5c bundler: make parsechunk return the base revision of the delta
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14060
diff changeset
   246
    def _deltaheader(self, headertuple, prevnode):
bd1cbfe5db5c bundler: make parsechunk return the base revision of the delta
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14060
diff changeset
   247
        node, p1, p2, cs = headertuple
bd1cbfe5db5c bundler: make parsechunk return the base revision of the delta
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14060
diff changeset
   248
        if prevnode is None:
bd1cbfe5db5c bundler: make parsechunk return the base revision of the delta
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14060
diff changeset
   249
            deltabase = p1
bd1cbfe5db5c bundler: make parsechunk return the base revision of the delta
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14060
diff changeset
   250
        else:
bd1cbfe5db5c bundler: make parsechunk return the base revision of the delta
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14060
diff changeset
   251
            deltabase = prevnode
27433
12f727a5b434 changegroup: add flags field to cg3 delta header
Mike Edgar <adgar@google.com>
parents: 27432
diff changeset
   252
        flags = 0
12f727a5b434 changegroup: add flags field to cg3 delta header
Mike Edgar <adgar@google.com>
parents: 27432
diff changeset
   253
        return node, p1, p2, deltabase, cs, flags
14141
bd1cbfe5db5c bundler: make parsechunk return the base revision of the delta
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14060
diff changeset
   254
14144
3c3c53d8343a unbundler: separate delta and header parsing
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14143
diff changeset
   255
    def deltachunk(self, prevnode):
26707
5ee6bd529300 changegroup: mark cg1unpacker.chunklength as private
Augie Fackler <augie@google.com>
parents: 26706
diff changeset
   256
        l = self._chunklength()
12336
9d234f7d8a77 bundle: move chunk parsing into unbundle class
Matt Mackall <mpm@selenic.com>
parents: 12335
diff changeset
   257
        if not l:
9d234f7d8a77 bundle: move chunk parsing into unbundle class
Matt Mackall <mpm@selenic.com>
parents: 12335
diff changeset
   258
            return {}
14141
bd1cbfe5db5c bundler: make parsechunk return the base revision of the delta
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14060
diff changeset
   259
        headerdata = readexactly(self._stream, self.deltaheadersize)
bd1cbfe5db5c bundler: make parsechunk return the base revision of the delta
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14060
diff changeset
   260
        header = struct.unpack(self.deltaheader, headerdata)
bd1cbfe5db5c bundler: make parsechunk return the base revision of the delta
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14060
diff changeset
   261
        delta = readexactly(self._stream, l - self.deltaheadersize)
27433
12f727a5b434 changegroup: add flags field to cg3 delta header
Mike Edgar <adgar@google.com>
parents: 27432
diff changeset
   262
        node, p1, p2, deltabase, cs, flags = self._deltaheader(header, prevnode)
20675
f8d50add83e1 changegroup: move from dict() construction to {} literals
Augie Fackler <raf@durin42.com>
parents: 19708
diff changeset
   263
        return {'node': node, 'p1': p1, 'p2': p2, 'cs': cs,
27433
12f727a5b434 changegroup: add flags field to cg3 delta header
Mike Edgar <adgar@google.com>
parents: 27432
diff changeset
   264
                'deltabase': deltabase, 'delta': delta, 'flags': flags}
12336
9d234f7d8a77 bundle: move chunk parsing into unbundle class
Matt Mackall <mpm@selenic.com>
parents: 12335
diff changeset
   265
20999
1e28ec9744bf changegroup: move chunk extraction into a getchunks method of unbundle10
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20978
diff changeset
   266
    def getchunks(self):
1e28ec9744bf changegroup: move chunk extraction into a getchunks method of unbundle10
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20978
diff changeset
   267
        """returns all the chunks contains in the bundle
1e28ec9744bf changegroup: move chunk extraction into a getchunks method of unbundle10
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20978
diff changeset
   268
1e28ec9744bf changegroup: move chunk extraction into a getchunks method of unbundle10
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20978
diff changeset
   269
        Used when you need to forward the binary stream to a file or another
1e28ec9744bf changegroup: move chunk extraction into a getchunks method of unbundle10
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20978
diff changeset
   270
        network API. To do so, it parse the changegroup data, otherwise it will
1e28ec9744bf changegroup: move chunk extraction into a getchunks method of unbundle10
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20978
diff changeset
   271
        block in case of sshrepo because it don't know the end of the stream.
1e28ec9744bf changegroup: move chunk extraction into a getchunks method of unbundle10
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20978
diff changeset
   272
        """
1e28ec9744bf changegroup: move chunk extraction into a getchunks method of unbundle10
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20978
diff changeset
   273
        # an empty chunkgroup is the end of the changegroup
1e28ec9744bf changegroup: move chunk extraction into a getchunks method of unbundle10
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20978
diff changeset
   274
        # a changegroup has at least 2 chunkgroups (changelog and manifest).
27920
da5f23362517 changegroup: cg3 has two empty groups *after* manifests
Martin von Zweigbergk <martinvonz@google.com>
parents: 27867
diff changeset
   275
        # after that, changegroup versions 1 and 2 have a series of groups
da5f23362517 changegroup: cg3 has two empty groups *after* manifests
Martin von Zweigbergk <martinvonz@google.com>
parents: 27867
diff changeset
   276
        # with one group per file. changegroup 3 has a series of directory
da5f23362517 changegroup: cg3 has two empty groups *after* manifests
Martin von Zweigbergk <martinvonz@google.com>
parents: 27867
diff changeset
   277
        # manifests before the files.
20999
1e28ec9744bf changegroup: move chunk extraction into a getchunks method of unbundle10
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20978
diff changeset
   278
        count = 0
27920
da5f23362517 changegroup: cg3 has two empty groups *after* manifests
Martin von Zweigbergk <martinvonz@google.com>
parents: 27867
diff changeset
   279
        emptycount = 0
da5f23362517 changegroup: cg3 has two empty groups *after* manifests
Martin von Zweigbergk <martinvonz@google.com>
parents: 27867
diff changeset
   280
        while emptycount < self._grouplistcount:
20999
1e28ec9744bf changegroup: move chunk extraction into a getchunks method of unbundle10
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20978
diff changeset
   281
            empty = True
1e28ec9744bf changegroup: move chunk extraction into a getchunks method of unbundle10
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20978
diff changeset
   282
            count += 1
1e28ec9744bf changegroup: move chunk extraction into a getchunks method of unbundle10
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20978
diff changeset
   283
            while True:
1e28ec9744bf changegroup: move chunk extraction into a getchunks method of unbundle10
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20978
diff changeset
   284
                chunk = getchunk(self)
1e28ec9744bf changegroup: move chunk extraction into a getchunks method of unbundle10
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20978
diff changeset
   285
                if not chunk:
27920
da5f23362517 changegroup: cg3 has two empty groups *after* manifests
Martin von Zweigbergk <martinvonz@google.com>
parents: 27867
diff changeset
   286
                    if empty and count > 2:
da5f23362517 changegroup: cg3 has two empty groups *after* manifests
Martin von Zweigbergk <martinvonz@google.com>
parents: 27867
diff changeset
   287
                        emptycount += 1
20999
1e28ec9744bf changegroup: move chunk extraction into a getchunks method of unbundle10
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20978
diff changeset
   288
                    break
1e28ec9744bf changegroup: move chunk extraction into a getchunks method of unbundle10
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20978
diff changeset
   289
                empty = False
1e28ec9744bf changegroup: move chunk extraction into a getchunks method of unbundle10
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20978
diff changeset
   290
                yield chunkheader(len(chunk))
1e28ec9744bf changegroup: move chunk extraction into a getchunks method of unbundle10
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20978
diff changeset
   291
                pos = 0
1e28ec9744bf changegroup: move chunk extraction into a getchunks method of unbundle10
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20978
diff changeset
   292
                while pos < len(chunk):
1e28ec9744bf changegroup: move chunk extraction into a getchunks method of unbundle10
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20978
diff changeset
   293
                    next = pos + 2**20
1e28ec9744bf changegroup: move chunk extraction into a getchunks method of unbundle10
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20978
diff changeset
   294
                    yield chunk[pos:next]
1e28ec9744bf changegroup: move chunk extraction into a getchunks method of unbundle10
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20978
diff changeset
   295
                    pos = next
1e28ec9744bf changegroup: move chunk extraction into a getchunks method of unbundle10
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20978
diff changeset
   296
            yield closechunk()
1e28ec9744bf changegroup: move chunk extraction into a getchunks method of unbundle10
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20978
diff changeset
   297
26712
04176eaf911b changegroup: move manifest unpacking into its own method
Augie Fackler <augie@google.com>
parents: 26711
diff changeset
   298
    def _unpackmanifests(self, repo, revmap, trp, prog, numchanges):
04176eaf911b changegroup: move manifest unpacking into its own method
Augie Fackler <augie@google.com>
parents: 26711
diff changeset
   299
        # We know that we'll never have more manifests than we had
04176eaf911b changegroup: move manifest unpacking into its own method
Augie Fackler <augie@google.com>
parents: 26711
diff changeset
   300
        # changesets.
04176eaf911b changegroup: move manifest unpacking into its own method
Augie Fackler <augie@google.com>
parents: 26711
diff changeset
   301
        self.callback = prog(_('manifests'), numchanges)
04176eaf911b changegroup: move manifest unpacking into its own method
Augie Fackler <augie@google.com>
parents: 26711
diff changeset
   302
        # no need to check for empty manifest group here:
04176eaf911b changegroup: move manifest unpacking into its own method
Augie Fackler <augie@google.com>
parents: 26711
diff changeset
   303
        # if the result of the merge of 1 and 2 is the same in 3 and 4,
04176eaf911b changegroup: move manifest unpacking into its own method
Augie Fackler <augie@google.com>
parents: 26711
diff changeset
   304
        # no new manifest will be created and the manifest group will
04176eaf911b changegroup: move manifest unpacking into its own method
Augie Fackler <augie@google.com>
parents: 26711
diff changeset
   305
        # be empty during the pull
04176eaf911b changegroup: move manifest unpacking into its own method
Augie Fackler <augie@google.com>
parents: 26711
diff changeset
   306
        self.manifestheader()
04176eaf911b changegroup: move manifest unpacking into its own method
Augie Fackler <augie@google.com>
parents: 26711
diff changeset
   307
        repo.manifest.addgroup(self, revmap, trp)
04176eaf911b changegroup: move manifest unpacking into its own method
Augie Fackler <augie@google.com>
parents: 26711
diff changeset
   308
        repo.ui.progress(_('manifests'), None)
04176eaf911b changegroup: move manifest unpacking into its own method
Augie Fackler <augie@google.com>
parents: 26711
diff changeset
   309
26695
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
   310
    def apply(self, repo, srctype, url, emptyok=False,
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
   311
              targetphase=phases.draft, expectedtotal=None):
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
   312
        """Add the changegroup returned by source.read() to this repo.
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
   313
        srctype is a string like 'push', 'pull', or 'unbundle'.  url is
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
   314
        the URL of the repo where this changegroup is coming from.
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
   315
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
   316
        Return an integer summarizing the change to this repo:
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
   317
        - nothing changed or no source: 0
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
   318
        - more heads than before: 1+added heads (2..n)
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
   319
        - fewer heads than before: -1-removed heads (-2..-n)
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
   320
        - number of heads stays the same: 1
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
   321
        """
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
   322
        repo = repo.unfiltered()
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
   323
        def csmap(x):
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
   324
            repo.ui.debug("add changeset %s\n" % short(x))
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
   325
            return len(cl)
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
   326
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
   327
        def revmap(x):
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
   328
            return cl.rev(x)
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
   329
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
   330
        changesets = files = revisions = 0
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
   331
26880
fa7f8b686633 changegroup: fix the scope of a try finally
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26859
diff changeset
   332
        try:
27867
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   333
            with repo.transaction("\n".join([srctype,
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   334
                                             util.hidepassword(url)])) as tr:
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   335
                # The transaction could have been created before and already
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   336
                # carries source information. In this case we use the top
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   337
                # level data. We overwrite the argument because we need to use
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   338
                # the top level value (if they exist) in this function.
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   339
                srctype = tr.hookargs.setdefault('source', srctype)
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   340
                url = tr.hookargs.setdefault('url', url)
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   341
                repo.hook('prechangegroup', throw=True, **tr.hookargs)
26695
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
   342
27867
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   343
                # write changelog data to temp files so concurrent readers
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   344
                # will not see an inconsistent view
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   345
                cl = repo.changelog
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   346
                cl.delayupdate(tr)
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   347
                oldheads = cl.heads()
26695
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
   348
27867
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   349
                trp = weakref.proxy(tr)
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   350
                # pull off the changeset group
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   351
                repo.ui.status(_("adding changesets\n"))
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   352
                clstart = len(cl)
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   353
                class prog(object):
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   354
                    def __init__(self, step, total):
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   355
                        self._step = step
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   356
                        self._total = total
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   357
                        self._count = 1
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   358
                    def __call__(self):
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   359
                        repo.ui.progress(self._step, self._count,
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   360
                                         unit=_('chunks'), total=self._total)
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   361
                        self._count += 1
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   362
                self.callback = prog(_('changesets'), expectedtotal)
26695
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
   363
27867
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   364
                efiles = set()
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   365
                def onchangelog(cl, node):
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   366
                    efiles.update(cl.read(node)[3])
26695
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
   367
27867
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   368
                self.changelogheader()
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   369
                srccontent = cl.addgroup(self, csmap, trp,
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   370
                                         addrevisioncb=onchangelog)
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   371
                efiles = len(efiles)
26695
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
   372
27867
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   373
                if not (srccontent or emptyok):
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   374
                    raise error.Abort(_("received changelog group is empty"))
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   375
                clend = len(cl)
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   376
                changesets = clend - clstart
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   377
                repo.ui.progress(_('changesets'), None)
26695
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
   378
27867
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   379
                # pull off the manifest group
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   380
                repo.ui.status(_("adding manifests\n"))
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   381
                self._unpackmanifests(repo, revmap, trp, prog, changesets)
26695
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
   382
27867
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   383
                needfiles = {}
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   384
                if repo.ui.configbool('server', 'validate', default=False):
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   385
                    # validate incoming csets have their manifests
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   386
                    for cset in xrange(clstart, clend):
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   387
                        mfnode = repo.changelog.read(
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   388
                            repo.changelog.node(cset))[0]
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   389
                        mfest = repo.manifest.readdelta(mfnode)
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   390
                        # store file nodes we must see
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   391
                        for f, n in mfest.iteritems():
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   392
                            needfiles.setdefault(f, set()).add(n)
26695
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
   393
27867
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   394
                # process the files
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   395
                repo.ui.status(_("adding file changes\n"))
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   396
                self.callback = None
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   397
                pr = prog(_('files'), efiles)
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   398
                newrevs, newfiles = _addchangegroupfiles(
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   399
                    repo, self, revmap, trp, pr, needfiles)
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   400
                revisions += newrevs
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   401
                files += newfiles
26695
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
   402
27867
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   403
                dh = 0
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   404
                if oldheads:
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   405
                    heads = cl.heads()
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   406
                    dh = len(heads) - len(oldheads)
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   407
                    for h in heads:
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   408
                        if h not in oldheads and repo[h].closesbranch():
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   409
                            dh -= 1
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   410
                htext = ""
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   411
                if dh:
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   412
                    htext = _(" (%+d heads)") % dh
26695
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
   413
27867
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   414
                repo.ui.status(_("added %d changesets"
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   415
                                 " with %d changes to %d files%s\n")
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   416
                                 % (changesets, revisions, files, htext))
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   417
                repo.invalidatevolatilesets()
26695
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
   418
27867
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   419
                if changesets > 0:
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   420
                    if 'node' not in tr.hookargs:
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   421
                        tr.hookargs['node'] = hex(cl.node(clstart))
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   422
                        tr.hookargs['node_last'] = hex(cl.node(clend - 1))
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   423
                        hookargs = dict(tr.hookargs)
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   424
                    else:
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   425
                        hookargs = dict(tr.hookargs)
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   426
                        hookargs['node'] = hex(cl.node(clstart))
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   427
                        hookargs['node_last'] = hex(cl.node(clend - 1))
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   428
                    repo.hook('pretxnchangegroup', throw=True, **hookargs)
26695
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
   429
27867
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   430
                added = [cl.node(r) for r in xrange(clstart, clend)]
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   431
                publishing = repo.publishing()
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   432
                if srctype in ('push', 'serve'):
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   433
                    # Old servers can not push the boundary themselves.
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   434
                    # New servers won't push the boundary if changeset already
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   435
                    # exists locally as secret
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   436
                    #
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   437
                    # We should not use added here but the list of all change in
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   438
                    # the bundle
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   439
                    if publishing:
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   440
                        phases.advanceboundary(repo, tr, phases.public,
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   441
                                               srccontent)
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   442
                    else:
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   443
                        # Those changesets have been pushed from the
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   444
                        # outside, their phases are going to be pushed
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   445
                        # alongside. Therefor `targetphase` is
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   446
                        # ignored.
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   447
                        phases.advanceboundary(repo, tr, phases.draft,
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   448
                                               srccontent)
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   449
                        phases.retractboundary(repo, tr, phases.draft, added)
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   450
                elif srctype != 'strip':
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   451
                    # publishing only alter behavior during push
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   452
                    #
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   453
                    # strip should not touch boundary at all
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   454
                    phases.retractboundary(repo, tr, targetphase, added)
26695
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
   455
27867
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   456
                if changesets > 0:
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   457
                    if srctype != 'strip':
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   458
                        # During strip, branchcache is invalid but
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   459
                        # coming call to `destroyed` will repair it.
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   460
                        # In other case we can safely update cache on
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   461
                        # disk.
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   462
                        branchmap.updatecache(repo.filtered('served'))
26695
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
   463
27867
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   464
                    def runhooks():
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   465
                        # These hooks run when the lock releases, not when the
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   466
                        # transaction closes. So it's possible for the changelog
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   467
                        # to have changed since we last saw it.
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   468
                        if clstart >= len(repo):
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   469
                            return
26695
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
   470
27867
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   471
                        # forcefully update the on-disk branch cache
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   472
                        repo.ui.debug("updating the branch cache\n")
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   473
                        repo.hook("changegroup", **hookargs)
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   474
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   475
                        for n in added:
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   476
                            args = hookargs.copy()
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   477
                            args['node'] = hex(n)
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   478
                            del args['node_last']
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   479
                            repo.hook("incoming", **args)
26695
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
   480
27867
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   481
                        newheads = [h for h in repo.heads()
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   482
                                    if h not in oldheads]
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   483
                        repo.ui.log("incoming",
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   484
                                    "%s incoming changes - new heads: %s\n",
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   485
                                    len(added),
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   486
                                    ', '.join([hex(c[:6]) for c in newheads]))
26695
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
   487
27867
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   488
                    tr.addpostclose('changegroup-runhooks-%020i' % clstart,
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
   489
                                    lambda tr: repo._afterlock(runhooks))
26695
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
   490
        finally:
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
   491
            repo.ui.flush()
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
   492
        # never return 0 here:
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
   493
        if dh < 0:
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
   494
            return dh - 1
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
   495
        else:
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
   496
            return dh + 1
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
   497
23181
832b7ef275c8 changegroup: introduce cg2packer/unpacker
Sune Foldager <cryo@cyanite.org>
parents: 23178
diff changeset
   498
class cg2unpacker(cg1unpacker):
26708
749d913f24b8 changegroup: document the public surface area of cg?unpackers
Augie Fackler <augie@google.com>
parents: 26707
diff changeset
   499
    """Unpacker for cg2 streams.
749d913f24b8 changegroup: document the public surface area of cg?unpackers
Augie Fackler <augie@google.com>
parents: 26707
diff changeset
   500
749d913f24b8 changegroup: document the public surface area of cg?unpackers
Augie Fackler <augie@google.com>
parents: 26707
diff changeset
   501
    cg2 streams add support for generaldelta, so the delta header
749d913f24b8 changegroup: document the public surface area of cg?unpackers
Augie Fackler <augie@google.com>
parents: 26707
diff changeset
   502
    format is slightly different. All other features about the data
749d913f24b8 changegroup: document the public surface area of cg?unpackers
Augie Fackler <augie@google.com>
parents: 26707
diff changeset
   503
    remain the same.
749d913f24b8 changegroup: document the public surface area of cg?unpackers
Augie Fackler <augie@google.com>
parents: 26707
diff changeset
   504
    """
23181
832b7ef275c8 changegroup: introduce cg2packer/unpacker
Sune Foldager <cryo@cyanite.org>
parents: 23178
diff changeset
   505
    deltaheader = _CHANGEGROUPV2_DELTA_HEADER
832b7ef275c8 changegroup: introduce cg2packer/unpacker
Sune Foldager <cryo@cyanite.org>
parents: 23178
diff changeset
   506
    deltaheadersize = struct.calcsize(deltaheader)
23896
becfecaf9087 changegroup.writebundle: HG2Y support
Eric Sumner <ericsumner@fb.com>
parents: 23895
diff changeset
   507
    version = '02'
23181
832b7ef275c8 changegroup: introduce cg2packer/unpacker
Sune Foldager <cryo@cyanite.org>
parents: 23178
diff changeset
   508
832b7ef275c8 changegroup: introduce cg2packer/unpacker
Sune Foldager <cryo@cyanite.org>
parents: 23178
diff changeset
   509
    def _deltaheader(self, headertuple, prevnode):
832b7ef275c8 changegroup: introduce cg2packer/unpacker
Sune Foldager <cryo@cyanite.org>
parents: 23178
diff changeset
   510
        node, p1, p2, deltabase, cs = headertuple
27433
12f727a5b434 changegroup: add flags field to cg3 delta header
Mike Edgar <adgar@google.com>
parents: 27432
diff changeset
   511
        flags = 0
12f727a5b434 changegroup: add flags field to cg3 delta header
Mike Edgar <adgar@google.com>
parents: 27432
diff changeset
   512
        return node, p1, p2, deltabase, cs, flags
23181
832b7ef275c8 changegroup: introduce cg2packer/unpacker
Sune Foldager <cryo@cyanite.org>
parents: 23178
diff changeset
   513
27432
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   514
class cg3unpacker(cg2unpacker):
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   515
    """Unpacker for cg3 streams.
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   516
27433
12f727a5b434 changegroup: add flags field to cg3 delta header
Mike Edgar <adgar@google.com>
parents: 27432
diff changeset
   517
    cg3 streams add support for exchanging treemanifests and revlog
27753
d4071cc73f46 changegroup3: add empty chunk separating directories and files
Martin von Zweigbergk <martinvonz@google.com>
parents: 27752
diff changeset
   518
    flags. It adds the revlog flags to the delta header and an empty chunk
d4071cc73f46 changegroup3: add empty chunk separating directories and files
Martin von Zweigbergk <martinvonz@google.com>
parents: 27752
diff changeset
   519
    separating manifests and files.
27432
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   520
    """
27433
12f727a5b434 changegroup: add flags field to cg3 delta header
Mike Edgar <adgar@google.com>
parents: 27432
diff changeset
   521
    deltaheader = _CHANGEGROUPV3_DELTA_HEADER
12f727a5b434 changegroup: add flags field to cg3 delta header
Mike Edgar <adgar@google.com>
parents: 27432
diff changeset
   522
    deltaheadersize = struct.calcsize(deltaheader)
27432
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   523
    version = '03'
27920
da5f23362517 changegroup: cg3 has two empty groups *after* manifests
Martin von Zweigbergk <martinvonz@google.com>
parents: 27867
diff changeset
   524
    _grouplistcount = 2 # One list of manifests and one list of files
27432
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   525
27433
12f727a5b434 changegroup: add flags field to cg3 delta header
Mike Edgar <adgar@google.com>
parents: 27432
diff changeset
   526
    def _deltaheader(self, headertuple, prevnode):
12f727a5b434 changegroup: add flags field to cg3 delta header
Mike Edgar <adgar@google.com>
parents: 27432
diff changeset
   527
        node, p1, p2, deltabase, cs, flags = headertuple
12f727a5b434 changegroup: add flags field to cg3 delta header
Mike Edgar <adgar@google.com>
parents: 27432
diff changeset
   528
        return node, p1, p2, deltabase, cs, flags
12f727a5b434 changegroup: add flags field to cg3 delta header
Mike Edgar <adgar@google.com>
parents: 27432
diff changeset
   529
27754
a09f143daaf4 changegroup3: move treemanifest support into _unpackmanifests()
Martin von Zweigbergk <martinvonz@google.com>
parents: 27753
diff changeset
   530
    def _unpackmanifests(self, repo, revmap, trp, prog, numchanges):
a09f143daaf4 changegroup3: move treemanifest support into _unpackmanifests()
Martin von Zweigbergk <martinvonz@google.com>
parents: 27753
diff changeset
   531
        super(cg3unpacker, self)._unpackmanifests(repo, revmap, trp, prog,
a09f143daaf4 changegroup3: move treemanifest support into _unpackmanifests()
Martin von Zweigbergk <martinvonz@google.com>
parents: 27753
diff changeset
   532
                                                  numchanges)
a09f143daaf4 changegroup3: move treemanifest support into _unpackmanifests()
Martin von Zweigbergk <martinvonz@google.com>
parents: 27753
diff changeset
   533
        while True:
a09f143daaf4 changegroup3: move treemanifest support into _unpackmanifests()
Martin von Zweigbergk <martinvonz@google.com>
parents: 27753
diff changeset
   534
            chunkdata = self.filelogheader()
a09f143daaf4 changegroup3: move treemanifest support into _unpackmanifests()
Martin von Zweigbergk <martinvonz@google.com>
parents: 27753
diff changeset
   535
            if not chunkdata:
a09f143daaf4 changegroup3: move treemanifest support into _unpackmanifests()
Martin von Zweigbergk <martinvonz@google.com>
parents: 27753
diff changeset
   536
                break
a09f143daaf4 changegroup3: move treemanifest support into _unpackmanifests()
Martin von Zweigbergk <martinvonz@google.com>
parents: 27753
diff changeset
   537
            # If we get here, there are directory manifests in the changegroup
a09f143daaf4 changegroup3: move treemanifest support into _unpackmanifests()
Martin von Zweigbergk <martinvonz@google.com>
parents: 27753
diff changeset
   538
            d = chunkdata["filename"]
a09f143daaf4 changegroup3: move treemanifest support into _unpackmanifests()
Martin von Zweigbergk <martinvonz@google.com>
parents: 27753
diff changeset
   539
            repo.ui.debug("adding %s revisions\n" % d)
a09f143daaf4 changegroup3: move treemanifest support into _unpackmanifests()
Martin von Zweigbergk <martinvonz@google.com>
parents: 27753
diff changeset
   540
            dirlog = repo.manifest.dirlog(d)
a09f143daaf4 changegroup3: move treemanifest support into _unpackmanifests()
Martin von Zweigbergk <martinvonz@google.com>
parents: 27753
diff changeset
   541
            if not dirlog.addgroup(self, revmap, trp):
a09f143daaf4 changegroup3: move treemanifest support into _unpackmanifests()
Martin von Zweigbergk <martinvonz@google.com>
parents: 27753
diff changeset
   542
                raise error.Abort(_("received dir revlog group is empty"))
a09f143daaf4 changegroup3: move treemanifest support into _unpackmanifests()
Martin von Zweigbergk <martinvonz@google.com>
parents: 27753
diff changeset
   543
12329
7458de933f26 bundle: push chunkbuffer down into decompress
Matt Mackall <mpm@selenic.com>
parents: 12044
diff changeset
   544
class headerlessfixup(object):
7458de933f26 bundle: push chunkbuffer down into decompress
Matt Mackall <mpm@selenic.com>
parents: 12044
diff changeset
   545
    def __init__(self, fh, h):
7458de933f26 bundle: push chunkbuffer down into decompress
Matt Mackall <mpm@selenic.com>
parents: 12044
diff changeset
   546
        self._h = h
7458de933f26 bundle: push chunkbuffer down into decompress
Matt Mackall <mpm@selenic.com>
parents: 12044
diff changeset
   547
        self._fh = fh
7458de933f26 bundle: push chunkbuffer down into decompress
Matt Mackall <mpm@selenic.com>
parents: 12044
diff changeset
   548
    def read(self, n):
7458de933f26 bundle: push chunkbuffer down into decompress
Matt Mackall <mpm@selenic.com>
parents: 12044
diff changeset
   549
        if self._h:
7458de933f26 bundle: push chunkbuffer down into decompress
Matt Mackall <mpm@selenic.com>
parents: 12044
diff changeset
   550
            d, self._h = self._h[:n], self._h[n:]
7458de933f26 bundle: push chunkbuffer down into decompress
Matt Mackall <mpm@selenic.com>
parents: 12044
diff changeset
   551
            if len(d) < n:
13457
e74fe15dc7fd changegroup: verify all stream reads
Mads Kiilerich <mads@kiilerich.com>
parents: 13456
diff changeset
   552
                d += readexactly(self._fh, n - len(d))
12329
7458de933f26 bundle: push chunkbuffer down into decompress
Matt Mackall <mpm@selenic.com>
parents: 12044
diff changeset
   553
            return d
13457
e74fe15dc7fd changegroup: verify all stream reads
Mads Kiilerich <mads@kiilerich.com>
parents: 13456
diff changeset
   554
        return readexactly(self._fh, n)
12329
7458de933f26 bundle: push chunkbuffer down into decompress
Matt Mackall <mpm@selenic.com>
parents: 12044
diff changeset
   555
27432
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   556
def _moddirs(files):
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   557
    """Given a set of modified files, find the list of modified directories.
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   558
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   559
    This returns a list of (path to changed dir, changed dir) tuples,
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   560
    as that's what the one client needs anyway.
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   561
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   562
    >>> _moddirs(['a/b/c.py', 'a/b/c.txt', 'a/d/e/f/g.txt', 'i.txt', ])
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   563
    [('/', 'a/'), ('a/', 'b/'), ('a/', 'd/'), ('a/d/', 'e/'), ('a/d/e/', 'f/')]
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   564
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   565
    """
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   566
    alldirs = set()
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   567
    for f in files:
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   568
        path = f.split('/')[:-1]
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   569
        for i in xrange(len(path) - 1, -1, -1):
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   570
            dn = '/'.join(path[:i])
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   571
            current = dn + '/', path[i] + '/'
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   572
            if current in alldirs:
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   573
                break
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   574
            alldirs.add(current)
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   575
    return sorted(alldirs)
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   576
22390
e2806b8613ca changegroup: rename bundle-related functions and classes
Sune Foldager <cryo@cyanite.org>
parents: 22070
diff changeset
   577
class cg1packer(object):
e2806b8613ca changegroup: rename bundle-related functions and classes
Sune Foldager <cryo@cyanite.org>
parents: 22070
diff changeset
   578
    deltaheader = _CHANGEGROUPV1_DELTA_HEADER
23896
becfecaf9087 changegroup.writebundle: HG2Y support
Eric Sumner <ericsumner@fb.com>
parents: 23895
diff changeset
   579
    version = '01'
19202
0455fc94ae00 bundle-ng: move gengroup into bundler, pass repo object to bundler
Sune Foldager <cryo@cyanite.org>
parents: 19201
diff changeset
   580
    def __init__(self, repo, bundlecaps=None):
0455fc94ae00 bundle-ng: move gengroup into bundler, pass repo object to bundler
Sune Foldager <cryo@cyanite.org>
parents: 19201
diff changeset
   581
        """Given a source repo, construct a bundler.
0455fc94ae00 bundle-ng: move gengroup into bundler, pass repo object to bundler
Sune Foldager <cryo@cyanite.org>
parents: 19201
diff changeset
   582
0455fc94ae00 bundle-ng: move gengroup into bundler, pass repo object to bundler
Sune Foldager <cryo@cyanite.org>
parents: 19201
diff changeset
   583
        bundlecaps is optional and can be used to specify the set of
0455fc94ae00 bundle-ng: move gengroup into bundler, pass repo object to bundler
Sune Foldager <cryo@cyanite.org>
parents: 19201
diff changeset
   584
        capabilities which can be used to build the bundle.
0455fc94ae00 bundle-ng: move gengroup into bundler, pass repo object to bundler
Sune Foldager <cryo@cyanite.org>
parents: 19201
diff changeset
   585
        """
19201
309c439cdbaa bundle-ng: add bundlecaps argument to getbundle() command
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 19200
diff changeset
   586
        # Set of capabilities we can use to build the bundle.
309c439cdbaa bundle-ng: add bundlecaps argument to getbundle() command
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 19200
diff changeset
   587
        if bundlecaps is None:
309c439cdbaa bundle-ng: add bundlecaps argument to getbundle() command
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 19200
diff changeset
   588
            bundlecaps = set()
309c439cdbaa bundle-ng: add bundlecaps argument to getbundle() command
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 19200
diff changeset
   589
        self._bundlecaps = bundlecaps
25831
578fc97904da generaldelta: mark experimental reordering option
Matt Mackall <mpm@selenic.com>
parents: 25823
diff changeset
   590
        # experimental config: bundle.reorder
19202
0455fc94ae00 bundle-ng: move gengroup into bundler, pass repo object to bundler
Sune Foldager <cryo@cyanite.org>
parents: 19201
diff changeset
   591
        reorder = repo.ui.config('bundle', 'reorder', 'auto')
0455fc94ae00 bundle-ng: move gengroup into bundler, pass repo object to bundler
Sune Foldager <cryo@cyanite.org>
parents: 19201
diff changeset
   592
        if reorder == 'auto':
0455fc94ae00 bundle-ng: move gengroup into bundler, pass repo object to bundler
Sune Foldager <cryo@cyanite.org>
parents: 19201
diff changeset
   593
            reorder = None
0455fc94ae00 bundle-ng: move gengroup into bundler, pass repo object to bundler
Sune Foldager <cryo@cyanite.org>
parents: 19201
diff changeset
   594
        else:
0455fc94ae00 bundle-ng: move gengroup into bundler, pass repo object to bundler
Sune Foldager <cryo@cyanite.org>
parents: 19201
diff changeset
   595
            reorder = util.parsebool(reorder)
0455fc94ae00 bundle-ng: move gengroup into bundler, pass repo object to bundler
Sune Foldager <cryo@cyanite.org>
parents: 19201
diff changeset
   596
        self._repo = repo
0455fc94ae00 bundle-ng: move gengroup into bundler, pass repo object to bundler
Sune Foldager <cryo@cyanite.org>
parents: 19201
diff changeset
   597
        self._reorder = reorder
19208
0b564cf359a7 bundle-ng: move progress handling out of the linkrev callback
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 19207
diff changeset
   598
        self._progress = repo.ui.progress
23748
4ab66de46a96 bundle: when verbose, show what takes up the space in the generated bundle
Mads Kiilerich <madski@unity3d.com>
parents: 23382
diff changeset
   599
        if self._repo.ui.verbose and not self._repo.ui.debugflag:
4ab66de46a96 bundle: when verbose, show what takes up the space in the generated bundle
Mads Kiilerich <madski@unity3d.com>
parents: 23382
diff changeset
   600
            self._verbosenote = self._repo.ui.note
4ab66de46a96 bundle: when verbose, show what takes up the space in the generated bundle
Mads Kiilerich <madski@unity3d.com>
parents: 23382
diff changeset
   601
        else:
4ab66de46a96 bundle: when verbose, show what takes up the space in the generated bundle
Mads Kiilerich <madski@unity3d.com>
parents: 23382
diff changeset
   602
            self._verbosenote = lambda s: None
4ab66de46a96 bundle: when verbose, show what takes up the space in the generated bundle
Mads Kiilerich <madski@unity3d.com>
parents: 23382
diff changeset
   603
13831
d69c9510d648 changegroup: introduce bundler objects
Matt Mackall <mpm@selenic.com>
parents: 13786
diff changeset
   604
    def close(self):
d69c9510d648 changegroup: introduce bundler objects
Matt Mackall <mpm@selenic.com>
parents: 13786
diff changeset
   605
        return closechunk()
19200
4cfdec944edf bundle-ng: move group into the bundler
Sune Foldager <cryo@cyanite.org>
parents: 19199
diff changeset
   606
13831
d69c9510d648 changegroup: introduce bundler objects
Matt Mackall <mpm@selenic.com>
parents: 13786
diff changeset
   607
    def fileheader(self, fname):
d69c9510d648 changegroup: introduce bundler objects
Matt Mackall <mpm@selenic.com>
parents: 13786
diff changeset
   608
        return chunkheader(len(fname)) + fname
19200
4cfdec944edf bundle-ng: move group into the bundler
Sune Foldager <cryo@cyanite.org>
parents: 19199
diff changeset
   609
24912
e285b98c65cc changegroup.group: drop 'reorder' parameter
Martin von Zweigbergk <martinvonz@google.com>
parents: 24911
diff changeset
   610
    def group(self, nodelist, revlog, lookup, units=None):
19200
4cfdec944edf bundle-ng: move group into the bundler
Sune Foldager <cryo@cyanite.org>
parents: 19199
diff changeset
   611
        """Calculate a delta group, yielding a sequence of changegroup chunks
4cfdec944edf bundle-ng: move group into the bundler
Sune Foldager <cryo@cyanite.org>
parents: 19199
diff changeset
   612
        (strings).
4cfdec944edf bundle-ng: move group into the bundler
Sune Foldager <cryo@cyanite.org>
parents: 19199
diff changeset
   613
4cfdec944edf bundle-ng: move group into the bundler
Sune Foldager <cryo@cyanite.org>
parents: 19199
diff changeset
   614
        Given a list of changeset revs, return a set of deltas and
4cfdec944edf bundle-ng: move group into the bundler
Sune Foldager <cryo@cyanite.org>
parents: 19199
diff changeset
   615
        metadata corresponding to nodes. The first delta is
4cfdec944edf bundle-ng: move group into the bundler
Sune Foldager <cryo@cyanite.org>
parents: 19199
diff changeset
   616
        first parent(nodelist[0]) -> nodelist[0], the receiver is
4cfdec944edf bundle-ng: move group into the bundler
Sune Foldager <cryo@cyanite.org>
parents: 19199
diff changeset
   617
        guaranteed to have this parent as it has all history before
4cfdec944edf bundle-ng: move group into the bundler
Sune Foldager <cryo@cyanite.org>
parents: 19199
diff changeset
   618
        these changesets. In the case firstparent is nullrev the
4cfdec944edf bundle-ng: move group into the bundler
Sune Foldager <cryo@cyanite.org>
parents: 19199
diff changeset
   619
        changegroup starts with a full revision.
19208
0b564cf359a7 bundle-ng: move progress handling out of the linkrev callback
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 19207
diff changeset
   620
0b564cf359a7 bundle-ng: move progress handling out of the linkrev callback
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 19207
diff changeset
   621
        If units is not None, progress detail will be generated, units specifies
0b564cf359a7 bundle-ng: move progress handling out of the linkrev callback
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 19207
diff changeset
   622
        the type of revlog that is touched (changelog, manifest, etc.).
19200
4cfdec944edf bundle-ng: move group into the bundler
Sune Foldager <cryo@cyanite.org>
parents: 19199
diff changeset
   623
        """
4cfdec944edf bundle-ng: move group into the bundler
Sune Foldager <cryo@cyanite.org>
parents: 19199
diff changeset
   624
        # if we don't have any revisions touched by these changesets, bail
4cfdec944edf bundle-ng: move group into the bundler
Sune Foldager <cryo@cyanite.org>
parents: 19199
diff changeset
   625
        if len(nodelist) == 0:
4cfdec944edf bundle-ng: move group into the bundler
Sune Foldager <cryo@cyanite.org>
parents: 19199
diff changeset
   626
            yield self.close()
4cfdec944edf bundle-ng: move group into the bundler
Sune Foldager <cryo@cyanite.org>
parents: 19199
diff changeset
   627
            return
4cfdec944edf bundle-ng: move group into the bundler
Sune Foldager <cryo@cyanite.org>
parents: 19199
diff changeset
   628
4cfdec944edf bundle-ng: move group into the bundler
Sune Foldager <cryo@cyanite.org>
parents: 19199
diff changeset
   629
        # for generaldelta revlogs, we linearize the revs; this will both be
4cfdec944edf bundle-ng: move group into the bundler
Sune Foldager <cryo@cyanite.org>
parents: 19199
diff changeset
   630
        # much quicker and generate a much smaller bundle
24912
e285b98c65cc changegroup.group: drop 'reorder' parameter
Martin von Zweigbergk <martinvonz@google.com>
parents: 24911
diff changeset
   631
        if (revlog._generaldelta and self._reorder is None) or self._reorder:
19200
4cfdec944edf bundle-ng: move group into the bundler
Sune Foldager <cryo@cyanite.org>
parents: 19199
diff changeset
   632
            dag = dagutil.revlogdag(revlog)
4cfdec944edf bundle-ng: move group into the bundler
Sune Foldager <cryo@cyanite.org>
parents: 19199
diff changeset
   633
            revs = set(revlog.rev(n) for n in nodelist)
4cfdec944edf bundle-ng: move group into the bundler
Sune Foldager <cryo@cyanite.org>
parents: 19199
diff changeset
   634
            revs = dag.linearize(revs)
4cfdec944edf bundle-ng: move group into the bundler
Sune Foldager <cryo@cyanite.org>
parents: 19199
diff changeset
   635
        else:
4cfdec944edf bundle-ng: move group into the bundler
Sune Foldager <cryo@cyanite.org>
parents: 19199
diff changeset
   636
            revs = sorted([revlog.rev(n) for n in nodelist])
4cfdec944edf bundle-ng: move group into the bundler
Sune Foldager <cryo@cyanite.org>
parents: 19199
diff changeset
   637
4cfdec944edf bundle-ng: move group into the bundler
Sune Foldager <cryo@cyanite.org>
parents: 19199
diff changeset
   638
        # add the parent of the first rev
4cfdec944edf bundle-ng: move group into the bundler
Sune Foldager <cryo@cyanite.org>
parents: 19199
diff changeset
   639
        p = revlog.parentrevs(revs[0])[0]
4cfdec944edf bundle-ng: move group into the bundler
Sune Foldager <cryo@cyanite.org>
parents: 19199
diff changeset
   640
        revs.insert(0, p)
4cfdec944edf bundle-ng: move group into the bundler
Sune Foldager <cryo@cyanite.org>
parents: 19199
diff changeset
   641
4cfdec944edf bundle-ng: move group into the bundler
Sune Foldager <cryo@cyanite.org>
parents: 19199
diff changeset
   642
        # build deltas
19208
0b564cf359a7 bundle-ng: move progress handling out of the linkrev callback
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 19207
diff changeset
   643
        total = len(revs) - 1
0b564cf359a7 bundle-ng: move progress handling out of the linkrev callback
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 19207
diff changeset
   644
        msgbundling = _('bundling')
19200
4cfdec944edf bundle-ng: move group into the bundler
Sune Foldager <cryo@cyanite.org>
parents: 19199
diff changeset
   645
        for r in xrange(len(revs) - 1):
19208
0b564cf359a7 bundle-ng: move progress handling out of the linkrev callback
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 19207
diff changeset
   646
            if units is not None:
0b564cf359a7 bundle-ng: move progress handling out of the linkrev callback
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 19207
diff changeset
   647
                self._progress(msgbundling, r + 1, unit=units, total=total)
19200
4cfdec944edf bundle-ng: move group into the bundler
Sune Foldager <cryo@cyanite.org>
parents: 19199
diff changeset
   648
            prev, curr = revs[r], revs[r + 1]
19207
a67e1380dfbd bundle-ng: simplify lookup and state handling
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 19206
diff changeset
   649
            linknode = lookup(revlog.node(curr))
a67e1380dfbd bundle-ng: simplify lookup and state handling
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 19206
diff changeset
   650
            for c in self.revchunk(revlog, curr, prev, linknode):
19200
4cfdec944edf bundle-ng: move group into the bundler
Sune Foldager <cryo@cyanite.org>
parents: 19199
diff changeset
   651
                yield c
4cfdec944edf bundle-ng: move group into the bundler
Sune Foldager <cryo@cyanite.org>
parents: 19199
diff changeset
   652
24901
e9edd53770fb changegroup: close progress in same function as it's started
Martin von Zweigbergk <martinvonz@google.com>
parents: 24900
diff changeset
   653
        if units is not None:
e9edd53770fb changegroup: close progress in same function as it's started
Martin von Zweigbergk <martinvonz@google.com>
parents: 24900
diff changeset
   654
            self._progress(msgbundling, None)
19200
4cfdec944edf bundle-ng: move group into the bundler
Sune Foldager <cryo@cyanite.org>
parents: 19199
diff changeset
   655
        yield self.close()
4cfdec944edf bundle-ng: move group into the bundler
Sune Foldager <cryo@cyanite.org>
parents: 19199
diff changeset
   656
19289
6ea1f858efd9 bundle: refactor changegroup prune to be its own function
Durham Goode <durham@fb.com>
parents: 19208
diff changeset
   657
    # filter any nodes that claim to be part of the known set
24896
9183cb6886ef changegroup: removed unused 'source' parameter from prune()
Martin von Zweigbergk <martinvonz@google.com>
parents: 24717
diff changeset
   658
    def prune(self, revlog, missing, commonrevs):
19289
6ea1f858efd9 bundle: refactor changegroup prune to be its own function
Durham Goode <durham@fb.com>
parents: 19208
diff changeset
   659
        rr, rl = revlog.rev, revlog.linkrev
6ea1f858efd9 bundle: refactor changegroup prune to be its own function
Durham Goode <durham@fb.com>
parents: 19208
diff changeset
   660
        return [n for n in missing if rl(rr(n)) not in commonrevs]
6ea1f858efd9 bundle: refactor changegroup prune to be its own function
Durham Goode <durham@fb.com>
parents: 19208
diff changeset
   661
27432
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   662
    def _packmanifests(self, mfnodes, tmfnodes, lookuplinknode):
26711
0ef0aec56090 changegroup: move manifest packing into a separate function
Augie Fackler <augie@google.com>
parents: 26710
diff changeset
   663
        """Pack flat manifests into a changegroup stream."""
0ef0aec56090 changegroup: move manifest packing into a separate function
Augie Fackler <augie@google.com>
parents: 26710
diff changeset
   664
        ml = self._repo.manifest
0ef0aec56090 changegroup: move manifest packing into a separate function
Augie Fackler <augie@google.com>
parents: 26710
diff changeset
   665
        size = 0
0ef0aec56090 changegroup: move manifest packing into a separate function
Augie Fackler <augie@google.com>
parents: 26710
diff changeset
   666
        for chunk in self.group(
0ef0aec56090 changegroup: move manifest packing into a separate function
Augie Fackler <augie@google.com>
parents: 26710
diff changeset
   667
                mfnodes, ml, lookuplinknode, units=_('manifests')):
0ef0aec56090 changegroup: move manifest packing into a separate function
Augie Fackler <augie@google.com>
parents: 26710
diff changeset
   668
            size += len(chunk)
0ef0aec56090 changegroup: move manifest packing into a separate function
Augie Fackler <augie@google.com>
parents: 26710
diff changeset
   669
            yield chunk
0ef0aec56090 changegroup: move manifest packing into a separate function
Augie Fackler <augie@google.com>
parents: 26710
diff changeset
   670
        self._verbosenote(_('%8.i (manifests)\n') % size)
27432
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   671
        # It looks odd to assert this here, but tmfnodes doesn't get
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   672
        # filled in until after we've called lookuplinknode for
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   673
        # sending root manifests, so the only way to tell the streams
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   674
        # got crossed is to check after we've done all the work.
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   675
        assert not tmfnodes
26711
0ef0aec56090 changegroup: move manifest packing into a separate function
Augie Fackler <augie@google.com>
parents: 26710
diff changeset
   676
19204
e9c5b1c246dc bundle-ng: move bundle generation to changegroup.py
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 19202
diff changeset
   677
    def generate(self, commonrevs, clnodes, fastpathlinkrev, source):
19202
0455fc94ae00 bundle-ng: move gengroup into bundler, pass repo object to bundler
Sune Foldager <cryo@cyanite.org>
parents: 19201
diff changeset
   678
        '''yield a sequence of changegroup chunks (strings)'''
0455fc94ae00 bundle-ng: move gengroup into bundler, pass repo object to bundler
Sune Foldager <cryo@cyanite.org>
parents: 19201
diff changeset
   679
        repo = self._repo
24978
f52560c64953 changegroup: drop _changelog and _manifest properties
Martin von Zweigbergk <martinvonz@google.com>
parents: 24977
diff changeset
   680
        cl = repo.changelog
f52560c64953 changegroup: drop _changelog and _manifest properties
Martin von Zweigbergk <martinvonz@google.com>
parents: 24977
diff changeset
   681
        ml = repo.manifest
19204
e9c5b1c246dc bundle-ng: move bundle generation to changegroup.py
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 19202
diff changeset
   682
23381
cc0ff93d0c0c changegroup: fix file linkrevs during reorders (issue4462)
Durham Goode <durham@fb.com>
parents: 23226
diff changeset
   683
        clrevorder = {}
19204
e9c5b1c246dc bundle-ng: move bundle generation to changegroup.py
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 19202
diff changeset
   684
        mfs = {} # needed manifests
27432
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   685
        tmfnodes = {}
19204
e9c5b1c246dc bundle-ng: move bundle generation to changegroup.py
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 19202
diff changeset
   686
        fnodes = {} # needed file nodes
27237
c08814b48ae5 changegroup: avoid iterating the whole manifest
Augie Fackler <augie@google.com>
parents: 27219
diff changeset
   687
        # maps manifest node id -> set(changed files)
c08814b48ae5 changegroup: avoid iterating the whole manifest
Augie Fackler <augie@google.com>
parents: 27219
diff changeset
   688
        mfchangedfiles = {}
19204
e9c5b1c246dc bundle-ng: move bundle generation to changegroup.py
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 19202
diff changeset
   689
19207
a67e1380dfbd bundle-ng: simplify lookup and state handling
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 19206
diff changeset
   690
        # Callback for the changelog, used to collect changed files and manifest
a67e1380dfbd bundle-ng: simplify lookup and state handling
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 19206
diff changeset
   691
        # nodes.
a67e1380dfbd bundle-ng: simplify lookup and state handling
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 19206
diff changeset
   692
        # Returns the linkrev node (identity in the changelog case).
a67e1380dfbd bundle-ng: simplify lookup and state handling
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 19206
diff changeset
   693
        def lookupcl(x):
a67e1380dfbd bundle-ng: simplify lookup and state handling
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 19206
diff changeset
   694
            c = cl.read(x)
23381
cc0ff93d0c0c changegroup: fix file linkrevs during reorders (issue4462)
Durham Goode <durham@fb.com>
parents: 23226
diff changeset
   695
            clrevorder[x] = len(clrevorder)
27237
c08814b48ae5 changegroup: avoid iterating the whole manifest
Augie Fackler <augie@google.com>
parents: 27219
diff changeset
   696
            n = c[0]
19207
a67e1380dfbd bundle-ng: simplify lookup and state handling
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 19206
diff changeset
   697
            # record the first changeset introducing this manifest version
27237
c08814b48ae5 changegroup: avoid iterating the whole manifest
Augie Fackler <augie@google.com>
parents: 27219
diff changeset
   698
            mfs.setdefault(n, x)
c08814b48ae5 changegroup: avoid iterating the whole manifest
Augie Fackler <augie@google.com>
parents: 27219
diff changeset
   699
            # Record a complete list of potentially-changed files in
c08814b48ae5 changegroup: avoid iterating the whole manifest
Augie Fackler <augie@google.com>
parents: 27219
diff changeset
   700
            # this manifest.
c08814b48ae5 changegroup: avoid iterating the whole manifest
Augie Fackler <augie@google.com>
parents: 27219
diff changeset
   701
            mfchangedfiles.setdefault(n, set()).update(c[3])
19207
a67e1380dfbd bundle-ng: simplify lookup and state handling
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 19206
diff changeset
   702
            return x
19204
e9c5b1c246dc bundle-ng: move bundle generation to changegroup.py
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 19202
diff changeset
   703
23748
4ab66de46a96 bundle: when verbose, show what takes up the space in the generated bundle
Mads Kiilerich <madski@unity3d.com>
parents: 23382
diff changeset
   704
        self._verbosenote(_('uncompressed size of bundle content:\n'))
4ab66de46a96 bundle: when verbose, show what takes up the space in the generated bundle
Mads Kiilerich <madski@unity3d.com>
parents: 23382
diff changeset
   705
        size = 0
24912
e285b98c65cc changegroup.group: drop 'reorder' parameter
Martin von Zweigbergk <martinvonz@google.com>
parents: 24911
diff changeset
   706
        for chunk in self.group(clnodes, cl, lookupcl, units=_('changesets')):
23748
4ab66de46a96 bundle: when verbose, show what takes up the space in the generated bundle
Mads Kiilerich <madski@unity3d.com>
parents: 23382
diff changeset
   707
            size += len(chunk)
23224
f4ab47ccefde changegroup: don't define lookupmf() until it is needed
Gregory Szorc <gregory.szorc@gmail.com>
parents: 22971
diff changeset
   708
            yield chunk
23748
4ab66de46a96 bundle: when verbose, show what takes up the space in the generated bundle
Mads Kiilerich <madski@unity3d.com>
parents: 23382
diff changeset
   709
        self._verbosenote(_('%8.i (changelog)\n') % size)
23224
f4ab47ccefde changegroup: don't define lookupmf() until it is needed
Gregory Szorc <gregory.szorc@gmail.com>
parents: 22971
diff changeset
   710
24977
4289383cb9d2 changegroup: document the cases where reordering complicates linkrevs
Martin von Zweigbergk <martinvonz@google.com>
parents: 24976
diff changeset
   711
        # We need to make sure that the linkrev in the changegroup refers to
4289383cb9d2 changegroup: document the cases where reordering complicates linkrevs
Martin von Zweigbergk <martinvonz@google.com>
parents: 24976
diff changeset
   712
        # the first changeset that introduced the manifest or file revision.
4289383cb9d2 changegroup: document the cases where reordering complicates linkrevs
Martin von Zweigbergk <martinvonz@google.com>
parents: 24976
diff changeset
   713
        # The fastpath is usually safer than the slowpath, because the filelogs
4289383cb9d2 changegroup: document the cases where reordering complicates linkrevs
Martin von Zweigbergk <martinvonz@google.com>
parents: 24976
diff changeset
   714
        # are walked in revlog order.
4289383cb9d2 changegroup: document the cases where reordering complicates linkrevs
Martin von Zweigbergk <martinvonz@google.com>
parents: 24976
diff changeset
   715
        #
4289383cb9d2 changegroup: document the cases where reordering complicates linkrevs
Martin von Zweigbergk <martinvonz@google.com>
parents: 24976
diff changeset
   716
        # When taking the slowpath with reorder=None and the manifest revlog
4289383cb9d2 changegroup: document the cases where reordering complicates linkrevs
Martin von Zweigbergk <martinvonz@google.com>
parents: 24976
diff changeset
   717
        # uses generaldelta, the manifest may be walked in the "wrong" order.
4289383cb9d2 changegroup: document the cases where reordering complicates linkrevs
Martin von Zweigbergk <martinvonz@google.com>
parents: 24976
diff changeset
   718
        # Without 'clrevorder', we would get an incorrect linkrev (see fix in
4289383cb9d2 changegroup: document the cases where reordering complicates linkrevs
Martin von Zweigbergk <martinvonz@google.com>
parents: 24976
diff changeset
   719
        # cc0ff93d0c0c).
4289383cb9d2 changegroup: document the cases where reordering complicates linkrevs
Martin von Zweigbergk <martinvonz@google.com>
parents: 24976
diff changeset
   720
        #
4289383cb9d2 changegroup: document the cases where reordering complicates linkrevs
Martin von Zweigbergk <martinvonz@google.com>
parents: 24976
diff changeset
   721
        # When taking the fastpath, we are only vulnerable to reordering
4289383cb9d2 changegroup: document the cases where reordering complicates linkrevs
Martin von Zweigbergk <martinvonz@google.com>
parents: 24976
diff changeset
   722
        # of the changelog itself. The changelog never uses generaldelta, so
4289383cb9d2 changegroup: document the cases where reordering complicates linkrevs
Martin von Zweigbergk <martinvonz@google.com>
parents: 24976
diff changeset
   723
        # it is only reordered when reorder=True. To handle this case, we
4289383cb9d2 changegroup: document the cases where reordering complicates linkrevs
Martin von Zweigbergk <martinvonz@google.com>
parents: 24976
diff changeset
   724
        # simply take the slowpath, which already has the 'clrevorder' logic.
4289383cb9d2 changegroup: document the cases where reordering complicates linkrevs
Martin von Zweigbergk <martinvonz@google.com>
parents: 24976
diff changeset
   725
        # This was also fixed in cc0ff93d0c0c.
24976
147d8892fc4b changegroup: extract condition for linkrev fastpath
Martin von Zweigbergk <martinvonz@google.com>
parents: 24912
diff changeset
   726
        fastpathlinkrev = fastpathlinkrev and not self._reorder
27432
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   727
        # Treemanifests don't work correctly with fastpathlinkrev
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   728
        # either, because we don't discover which directory nodes to
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   729
        # send along with files. This could probably be fixed.
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   730
        fastpathlinkrev = fastpathlinkrev and (
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   731
            'treemanifest' not in repo.requirements)
19207
a67e1380dfbd bundle-ng: simplify lookup and state handling
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 19206
diff changeset
   732
        # Callback for the manifest, used to collect linkrevs for filelog
a67e1380dfbd bundle-ng: simplify lookup and state handling
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 19206
diff changeset
   733
        # revisions.
a67e1380dfbd bundle-ng: simplify lookup and state handling
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 19206
diff changeset
   734
        # Returns the linkrev node (collected in lookupcl).
27239
65c47779bcb5 changegroup: remove one special case from lookupmflinknode
Augie Fackler <augie@google.com>
parents: 27238
diff changeset
   735
        if fastpathlinkrev:
65c47779bcb5 changegroup: remove one special case from lookupmflinknode
Augie Fackler <augie@google.com>
parents: 27238
diff changeset
   736
            lookupmflinknode = mfs.__getitem__
65c47779bcb5 changegroup: remove one special case from lookupmflinknode
Augie Fackler <augie@google.com>
parents: 27238
diff changeset
   737
        else:
65c47779bcb5 changegroup: remove one special case from lookupmflinknode
Augie Fackler <augie@google.com>
parents: 27238
diff changeset
   738
            def lookupmflinknode(x):
65c47779bcb5 changegroup: remove one special case from lookupmflinknode
Augie Fackler <augie@google.com>
parents: 27238
diff changeset
   739
                """Callback for looking up the linknode for manifests.
27219
beb60a898dd0 changegroup: document manifest linkrev callback some more
Augie Fackler <augie@google.com>
parents: 27218
diff changeset
   740
27239
65c47779bcb5 changegroup: remove one special case from lookupmflinknode
Augie Fackler <augie@google.com>
parents: 27238
diff changeset
   741
                Returns the linkrev node for the specified manifest.
27219
beb60a898dd0 changegroup: document manifest linkrev callback some more
Augie Fackler <augie@google.com>
parents: 27218
diff changeset
   742
27239
65c47779bcb5 changegroup: remove one special case from lookupmflinknode
Augie Fackler <augie@google.com>
parents: 27238
diff changeset
   743
                SIDE EFFECT:
65c47779bcb5 changegroup: remove one special case from lookupmflinknode
Augie Fackler <augie@google.com>
parents: 27238
diff changeset
   744
27432
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   745
                1) fclnodes gets populated with the list of relevant
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   746
                   file nodes if we're not using fastpathlinkrev
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   747
                2) When treemanifests are in use, collects treemanifest nodes
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   748
                   to send
27219
beb60a898dd0 changegroup: document manifest linkrev callback some more
Augie Fackler <augie@google.com>
parents: 27218
diff changeset
   749
27432
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   750
                Note that this means manifests must be completely sent to
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   751
                the client before you can trust the list of files and
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   752
                treemanifests to send.
27239
65c47779bcb5 changegroup: remove one special case from lookupmflinknode
Augie Fackler <augie@google.com>
parents: 27238
diff changeset
   753
                """
65c47779bcb5 changegroup: remove one special case from lookupmflinknode
Augie Fackler <augie@google.com>
parents: 27238
diff changeset
   754
                clnode = mfs[x]
27432
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   755
                # We no longer actually care about reading deltas of
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   756
                # the manifest here, because we already know the list
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   757
                # of changed files, so for treemanifests (which
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   758
                # lazily-load anyway to *generate* a readdelta) we can
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   759
                # just load them with read() and then we'll actually
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   760
                # be able to correctly load node IDs from the
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   761
                # submanifest entries.
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   762
                if 'treemanifest' in repo.requirements:
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   763
                    mdata = ml.read(x)
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   764
                else:
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   765
                    mdata = ml.readfast(x)
27237
c08814b48ae5 changegroup: avoid iterating the whole manifest
Augie Fackler <augie@google.com>
parents: 27219
diff changeset
   766
                for f in mfchangedfiles[x]:
27238
c3dc03109401 changegroup: drop 'if True' that made the previous change clearer
Augie Fackler <augie@google.com>
parents: 27237
diff changeset
   767
                    try:
c3dc03109401 changegroup: drop 'if True' that made the previous change clearer
Augie Fackler <augie@google.com>
parents: 27237
diff changeset
   768
                        n = mdata[f]
c3dc03109401 changegroup: drop 'if True' that made the previous change clearer
Augie Fackler <augie@google.com>
parents: 27237
diff changeset
   769
                    except KeyError:
c3dc03109401 changegroup: drop 'if True' that made the previous change clearer
Augie Fackler <augie@google.com>
parents: 27237
diff changeset
   770
                        continue
c3dc03109401 changegroup: drop 'if True' that made the previous change clearer
Augie Fackler <augie@google.com>
parents: 27237
diff changeset
   771
                    # record the first changeset introducing this filelog
c3dc03109401 changegroup: drop 'if True' that made the previous change clearer
Augie Fackler <augie@google.com>
parents: 27237
diff changeset
   772
                    # version
c3dc03109401 changegroup: drop 'if True' that made the previous change clearer
Augie Fackler <augie@google.com>
parents: 27237
diff changeset
   773
                    fclnodes = fnodes.setdefault(f, {})
c3dc03109401 changegroup: drop 'if True' that made the previous change clearer
Augie Fackler <augie@google.com>
parents: 27237
diff changeset
   774
                    fclnode = fclnodes.setdefault(n, clnode)
c3dc03109401 changegroup: drop 'if True' that made the previous change clearer
Augie Fackler <augie@google.com>
parents: 27237
diff changeset
   775
                    if clrevorder[clnode] < clrevorder[fclnode]:
c3dc03109401 changegroup: drop 'if True' that made the previous change clearer
Augie Fackler <augie@google.com>
parents: 27237
diff changeset
   776
                        fclnodes[n] = clnode
27432
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   777
                # gather list of changed treemanifest nodes
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   778
                if 'treemanifest' in repo.requirements:
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   779
                    submfs = {'/': mdata}
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   780
                    for dn, bn in _moddirs(mfchangedfiles[x]):
27946
ca8d2b73155d changegroup: fix treemanifest exchange code (issue5061)
Augie Fackler <augie@google.com>
parents: 27931
diff changeset
   781
                        try:
ca8d2b73155d changegroup: fix treemanifest exchange code (issue5061)
Augie Fackler <augie@google.com>
parents: 27931
diff changeset
   782
                            submf = submfs[dn]
ca8d2b73155d changegroup: fix treemanifest exchange code (issue5061)
Augie Fackler <augie@google.com>
parents: 27931
diff changeset
   783
                            submf = submf._dirs[bn]
ca8d2b73155d changegroup: fix treemanifest exchange code (issue5061)
Augie Fackler <augie@google.com>
parents: 27931
diff changeset
   784
                        except KeyError:
ca8d2b73155d changegroup: fix treemanifest exchange code (issue5061)
Augie Fackler <augie@google.com>
parents: 27931
diff changeset
   785
                            continue # deleted directory, so nothing to send
27432
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   786
                        submfs[submf.dir()] = submf
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   787
                        tmfclnodes = tmfnodes.setdefault(submf.dir(), {})
27946
ca8d2b73155d changegroup: fix treemanifest exchange code (issue5061)
Augie Fackler <augie@google.com>
parents: 27931
diff changeset
   788
                        tmfclnode = tmfclnodes.setdefault(submf._node, clnode)
ca8d2b73155d changegroup: fix treemanifest exchange code (issue5061)
Augie Fackler <augie@google.com>
parents: 27931
diff changeset
   789
                        if clrevorder[clnode] < clrevorder[tmfclnode]:
27432
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   790
                            tmfclnodes[n] = clnode
27239
65c47779bcb5 changegroup: remove one special case from lookupmflinknode
Augie Fackler <augie@google.com>
parents: 27238
diff changeset
   791
                return clnode
19206
6308896b1d4a bundle-ng: simplify bundle10.generate
Sune Foldager <cryo@cyanite.org>
parents: 19204
diff changeset
   792
24899
cf1ea7566b20 changegroup: rename 'mf' to 'ml' to match 'cl', since it's a revlog
Martin von Zweigbergk <martinvonz@google.com>
parents: 24898
diff changeset
   793
        mfnodes = self.prune(ml, mfs, commonrevs)
27432
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   794
        for x in self._packmanifests(
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   795
            mfnodes, tmfnodes, lookupmflinknode):
26711
0ef0aec56090 changegroup: move manifest packing into a separate function
Augie Fackler <augie@google.com>
parents: 26710
diff changeset
   796
            yield x
19206
6308896b1d4a bundle-ng: simplify bundle10.generate
Sune Foldager <cryo@cyanite.org>
parents: 19204
diff changeset
   797
6308896b1d4a bundle-ng: simplify bundle10.generate
Sune Foldager <cryo@cyanite.org>
parents: 19204
diff changeset
   798
        mfs.clear()
24898
0bbf061564cf changegroup: rename 'needed' to 'clrevs' to match 'clnodes'
Martin von Zweigbergk <martinvonz@google.com>
parents: 24897
diff changeset
   799
        clrevs = set(cl.rev(x) for x in clnodes)
19206
6308896b1d4a bundle-ng: simplify bundle10.generate
Sune Foldager <cryo@cyanite.org>
parents: 19204
diff changeset
   800
27240
94a3d6fdc315 changegroup: clean up file lookup function
Augie Fackler <augie@google.com>
parents: 27239
diff changeset
   801
        if not fastpathlinkrev:
94a3d6fdc315 changegroup: clean up file lookup function
Augie Fackler <augie@google.com>
parents: 27239
diff changeset
   802
            def linknodes(unused, fname):
94a3d6fdc315 changegroup: clean up file lookup function
Augie Fackler <augie@google.com>
parents: 27239
diff changeset
   803
                return fnodes.get(fname, {})
94a3d6fdc315 changegroup: clean up file lookup function
Augie Fackler <augie@google.com>
parents: 27239
diff changeset
   804
        else:
27241
ead8e7069998 changegroup: restate file linknode callback using generator expressions
Augie Fackler <augie@google.com>
parents: 27240
diff changeset
   805
            cln = cl.node
27240
94a3d6fdc315 changegroup: clean up file lookup function
Augie Fackler <augie@google.com>
parents: 27239
diff changeset
   806
            def linknodes(filerevlog, fname):
20936
bfb40168391c changegroup: remove unused variable caught by pyflakes
Sean Farley <sean.michael.farley@gmail.com>
parents: 19708
diff changeset
   807
                llr = filerevlog.linkrev
27241
ead8e7069998 changegroup: restate file linknode callback using generator expressions
Augie Fackler <augie@google.com>
parents: 27240
diff changeset
   808
                fln = filerevlog.node
ead8e7069998 changegroup: restate file linknode callback using generator expressions
Augie Fackler <augie@google.com>
parents: 27240
diff changeset
   809
                revs = ((r, llr(r)) for r in filerevlog)
ead8e7069998 changegroup: restate file linknode callback using generator expressions
Augie Fackler <augie@google.com>
parents: 27240
diff changeset
   810
                return dict((fln(r), cln(lr)) for r, lr in revs if lr in clrevs)
19207
a67e1380dfbd bundle-ng: simplify lookup and state handling
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 19206
diff changeset
   811
27237
c08814b48ae5 changegroup: avoid iterating the whole manifest
Augie Fackler <augie@google.com>
parents: 27219
diff changeset
   812
        changedfiles = set()
c08814b48ae5 changegroup: avoid iterating the whole manifest
Augie Fackler <augie@google.com>
parents: 27219
diff changeset
   813
        for x in mfchangedfiles.itervalues():
c08814b48ae5 changegroup: avoid iterating the whole manifest
Augie Fackler <augie@google.com>
parents: 27219
diff changeset
   814
            changedfiles.update(x)
19334
95a49112e7ab bundle: move file chunk generation to it's own function
Durham Goode <durham@fb.com>
parents: 19325
diff changeset
   815
        for chunk in self.generatefiles(changedfiles, linknodes, commonrevs,
95a49112e7ab bundle: move file chunk generation to it's own function
Durham Goode <durham@fb.com>
parents: 19325
diff changeset
   816
                                        source):
95a49112e7ab bundle: move file chunk generation to it's own function
Durham Goode <durham@fb.com>
parents: 19325
diff changeset
   817
            yield chunk
95a49112e7ab bundle: move file chunk generation to it's own function
Durham Goode <durham@fb.com>
parents: 19325
diff changeset
   818
95a49112e7ab bundle: move file chunk generation to it's own function
Durham Goode <durham@fb.com>
parents: 19325
diff changeset
   819
        yield self.close()
95a49112e7ab bundle: move file chunk generation to it's own function
Durham Goode <durham@fb.com>
parents: 19325
diff changeset
   820
95a49112e7ab bundle: move file chunk generation to it's own function
Durham Goode <durham@fb.com>
parents: 19325
diff changeset
   821
        if clnodes:
95a49112e7ab bundle: move file chunk generation to it's own function
Durham Goode <durham@fb.com>
parents: 19325
diff changeset
   822
            repo.hook('outgoing', node=hex(clnodes[0]), source=source)
95a49112e7ab bundle: move file chunk generation to it's own function
Durham Goode <durham@fb.com>
parents: 19325
diff changeset
   823
24897
5c35a6040352 changegroup: document that 'source' parameter exists for extensions
Martin von Zweigbergk <martinvonz@google.com>
parents: 24896
diff changeset
   824
    # The 'source' parameter is useful for extensions
19334
95a49112e7ab bundle: move file chunk generation to it's own function
Durham Goode <durham@fb.com>
parents: 19325
diff changeset
   825
    def generatefiles(self, changedfiles, linknodes, commonrevs, source):
95a49112e7ab bundle: move file chunk generation to it's own function
Durham Goode <durham@fb.com>
parents: 19325
diff changeset
   826
        repo = self._repo
95a49112e7ab bundle: move file chunk generation to it's own function
Durham Goode <durham@fb.com>
parents: 19325
diff changeset
   827
        progress = self._progress
95a49112e7ab bundle: move file chunk generation to it's own function
Durham Goode <durham@fb.com>
parents: 19325
diff changeset
   828
        msgbundling = _('bundling')
95a49112e7ab bundle: move file chunk generation to it's own function
Durham Goode <durham@fb.com>
parents: 19325
diff changeset
   829
95a49112e7ab bundle: move file chunk generation to it's own function
Durham Goode <durham@fb.com>
parents: 19325
diff changeset
   830
        total = len(changedfiles)
95a49112e7ab bundle: move file chunk generation to it's own function
Durham Goode <durham@fb.com>
parents: 19325
diff changeset
   831
        # for progress output
95a49112e7ab bundle: move file chunk generation to it's own function
Durham Goode <durham@fb.com>
parents: 19325
diff changeset
   832
        msgfiles = _('files')
95a49112e7ab bundle: move file chunk generation to it's own function
Durham Goode <durham@fb.com>
parents: 19325
diff changeset
   833
        for i, fname in enumerate(sorted(changedfiles)):
95a49112e7ab bundle: move file chunk generation to it's own function
Durham Goode <durham@fb.com>
parents: 19325
diff changeset
   834
            filerevlog = repo.file(fname)
95a49112e7ab bundle: move file chunk generation to it's own function
Durham Goode <durham@fb.com>
parents: 19325
diff changeset
   835
            if not filerevlog:
26587
56b2bcea2529 error: get Abort from 'error' instead of 'util'
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26540
diff changeset
   836
                raise error.Abort(_("empty or missing revlog for %s") % fname)
19334
95a49112e7ab bundle: move file chunk generation to it's own function
Durham Goode <durham@fb.com>
parents: 19325
diff changeset
   837
95a49112e7ab bundle: move file chunk generation to it's own function
Durham Goode <durham@fb.com>
parents: 19325
diff changeset
   838
            linkrevnodes = linknodes(filerevlog, fname)
19207
a67e1380dfbd bundle-ng: simplify lookup and state handling
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 19206
diff changeset
   839
            # Lookup for filenodes, we collected the linkrev nodes above in the
a67e1380dfbd bundle-ng: simplify lookup and state handling
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 19206
diff changeset
   840
            # fastpath case and with lookupmf in the slowpath case.
a67e1380dfbd bundle-ng: simplify lookup and state handling
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 19206
diff changeset
   841
            def lookupfilelog(x):
a67e1380dfbd bundle-ng: simplify lookup and state handling
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 19206
diff changeset
   842
                return linkrevnodes[x]
a67e1380dfbd bundle-ng: simplify lookup and state handling
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 19206
diff changeset
   843
24896
9183cb6886ef changegroup: removed unused 'source' parameter from prune()
Martin von Zweigbergk <martinvonz@google.com>
parents: 24717
diff changeset
   844
            filenodes = self.prune(filerevlog, linkrevnodes, commonrevs)
19206
6308896b1d4a bundle-ng: simplify bundle10.generate
Sune Foldager <cryo@cyanite.org>
parents: 19204
diff changeset
   845
            if filenodes:
19208
0b564cf359a7 bundle-ng: move progress handling out of the linkrev callback
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 19207
diff changeset
   846
                progress(msgbundling, i + 1, item=fname, unit=msgfiles,
0b564cf359a7 bundle-ng: move progress handling out of the linkrev callback
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 19207
diff changeset
   847
                         total=total)
23748
4ab66de46a96 bundle: when verbose, show what takes up the space in the generated bundle
Mads Kiilerich <madski@unity3d.com>
parents: 23382
diff changeset
   848
                h = self.fileheader(fname)
4ab66de46a96 bundle: when verbose, show what takes up the space in the generated bundle
Mads Kiilerich <madski@unity3d.com>
parents: 23382
diff changeset
   849
                size = len(h)
4ab66de46a96 bundle: when verbose, show what takes up the space in the generated bundle
Mads Kiilerich <madski@unity3d.com>
parents: 23382
diff changeset
   850
                yield h
24912
e285b98c65cc changegroup.group: drop 'reorder' parameter
Martin von Zweigbergk <martinvonz@google.com>
parents: 24911
diff changeset
   851
                for chunk in self.group(filenodes, filerevlog, lookupfilelog):
23748
4ab66de46a96 bundle: when verbose, show what takes up the space in the generated bundle
Mads Kiilerich <madski@unity3d.com>
parents: 23382
diff changeset
   852
                    size += len(chunk)
19202
0455fc94ae00 bundle-ng: move gengroup into bundler, pass repo object to bundler
Sune Foldager <cryo@cyanite.org>
parents: 19201
diff changeset
   853
                    yield chunk
23748
4ab66de46a96 bundle: when verbose, show what takes up the space in the generated bundle
Mads Kiilerich <madski@unity3d.com>
parents: 23382
diff changeset
   854
                self._verbosenote(_('%8.i  %s\n') % (size, fname))
24901
e9edd53770fb changegroup: close progress in same function as it's started
Martin von Zweigbergk <martinvonz@google.com>
parents: 24900
diff changeset
   855
        progress(msgbundling, None)
19200
4cfdec944edf bundle-ng: move group into the bundler
Sune Foldager <cryo@cyanite.org>
parents: 19199
diff changeset
   856
23181
832b7ef275c8 changegroup: introduce cg2packer/unpacker
Sune Foldager <cryo@cyanite.org>
parents: 23178
diff changeset
   857
    def deltaparent(self, revlog, rev, p1, p2, prev):
832b7ef275c8 changegroup: introduce cg2packer/unpacker
Sune Foldager <cryo@cyanite.org>
parents: 23178
diff changeset
   858
        return prev
832b7ef275c8 changegroup: introduce cg2packer/unpacker
Sune Foldager <cryo@cyanite.org>
parents: 23178
diff changeset
   859
19207
a67e1380dfbd bundle-ng: simplify lookup and state handling
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 19206
diff changeset
   860
    def revchunk(self, revlog, rev, prev, linknode):
14143
da635d3c5620 changegroup: new bundler API
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14141
diff changeset
   861
        node = revlog.node(rev)
da635d3c5620 changegroup: new bundler API
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14141
diff changeset
   862
        p1, p2 = revlog.parentrevs(rev)
23181
832b7ef275c8 changegroup: introduce cg2packer/unpacker
Sune Foldager <cryo@cyanite.org>
parents: 23178
diff changeset
   863
        base = self.deltaparent(revlog, rev, p1, p2, prev)
14143
da635d3c5620 changegroup: new bundler API
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14141
diff changeset
   864
da635d3c5620 changegroup: new bundler API
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14141
diff changeset
   865
        prefix = ''
24190
903c7e8c97ad changegroup: emit full-replacement deltas if either revision is censored
Mike Edgar <adgar@google.com>
parents: 24180
diff changeset
   866
        if revlog.iscensored(base) or revlog.iscensored(rev):
903c7e8c97ad changegroup: emit full-replacement deltas if either revision is censored
Mike Edgar <adgar@google.com>
parents: 24180
diff changeset
   867
            try:
903c7e8c97ad changegroup: emit full-replacement deltas if either revision is censored
Mike Edgar <adgar@google.com>
parents: 24180
diff changeset
   868
                delta = revlog.revision(node)
25660
328739ea70c3 global: mass rewrite to use modern exception syntax
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25624
diff changeset
   869
            except error.CensoredNodeError as e:
24190
903c7e8c97ad changegroup: emit full-replacement deltas if either revision is censored
Mike Edgar <adgar@google.com>
parents: 24180
diff changeset
   870
                delta = e.tombstone
903c7e8c97ad changegroup: emit full-replacement deltas if either revision is censored
Mike Edgar <adgar@google.com>
parents: 24180
diff changeset
   871
            if base == nullrev:
903c7e8c97ad changegroup: emit full-replacement deltas if either revision is censored
Mike Edgar <adgar@google.com>
parents: 24180
diff changeset
   872
                prefix = mdiff.trivialdiffheader(len(delta))
903c7e8c97ad changegroup: emit full-replacement deltas if either revision is censored
Mike Edgar <adgar@google.com>
parents: 24180
diff changeset
   873
            else:
903c7e8c97ad changegroup: emit full-replacement deltas if either revision is censored
Mike Edgar <adgar@google.com>
parents: 24180
diff changeset
   874
                baselen = revlog.rawsize(base)
903c7e8c97ad changegroup: emit full-replacement deltas if either revision is censored
Mike Edgar <adgar@google.com>
parents: 24180
diff changeset
   875
                prefix = mdiff.replacediffheader(baselen, len(delta))
903c7e8c97ad changegroup: emit full-replacement deltas if either revision is censored
Mike Edgar <adgar@google.com>
parents: 24180
diff changeset
   876
        elif base == nullrev:
14143
da635d3c5620 changegroup: new bundler API
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14141
diff changeset
   877
            delta = revlog.revision(node)
da635d3c5620 changegroup: new bundler API
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14141
diff changeset
   878
            prefix = mdiff.trivialdiffheader(len(delta))
da635d3c5620 changegroup: new bundler API
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14141
diff changeset
   879
        else:
da635d3c5620 changegroup: new bundler API
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14141
diff changeset
   880
            delta = revlog.revdiff(base, rev)
da635d3c5620 changegroup: new bundler API
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14141
diff changeset
   881
        p1n, p2n = revlog.parents(node)
da635d3c5620 changegroup: new bundler API
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14141
diff changeset
   882
        basenode = revlog.node(base)
27433
12f727a5b434 changegroup: add flags field to cg3 delta header
Mike Edgar <adgar@google.com>
parents: 27432
diff changeset
   883
        flags = revlog.flags(rev)
12f727a5b434 changegroup: add flags field to cg3 delta header
Mike Edgar <adgar@google.com>
parents: 27432
diff changeset
   884
        meta = self.builddeltaheader(node, p1n, p2n, basenode, linknode, flags)
14143
da635d3c5620 changegroup: new bundler API
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14141
diff changeset
   885
        meta += prefix
da635d3c5620 changegroup: new bundler API
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14141
diff changeset
   886
        l = len(meta) + len(delta)
13831
d69c9510d648 changegroup: introduce bundler objects
Matt Mackall <mpm@selenic.com>
parents: 13786
diff changeset
   887
        yield chunkheader(l)
d69c9510d648 changegroup: introduce bundler objects
Matt Mackall <mpm@selenic.com>
parents: 13786
diff changeset
   888
        yield meta
14143
da635d3c5620 changegroup: new bundler API
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14141
diff changeset
   889
        yield delta
27433
12f727a5b434 changegroup: add flags field to cg3 delta header
Mike Edgar <adgar@google.com>
parents: 27432
diff changeset
   890
    def builddeltaheader(self, node, p1n, p2n, basenode, linknode, flags):
14143
da635d3c5620 changegroup: new bundler API
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14141
diff changeset
   891
        # do nothing with basenode, it is implicitly the previous one in HG10
27433
12f727a5b434 changegroup: add flags field to cg3 delta header
Mike Edgar <adgar@google.com>
parents: 27432
diff changeset
   892
        # do nothing with flags, it is implicitly 0 for cg1 and cg2
14143
da635d3c5620 changegroup: new bundler API
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14141
diff changeset
   893
        return struct.pack(self.deltaheader, node, p1n, p2n, linknode)
20925
5174c48ed8d8 localrepo: move the _changegroupsubset method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20675
diff changeset
   894
23181
832b7ef275c8 changegroup: introduce cg2packer/unpacker
Sune Foldager <cryo@cyanite.org>
parents: 23178
diff changeset
   895
class cg2packer(cg1packer):
23896
becfecaf9087 changegroup.writebundle: HG2Y support
Eric Sumner <ericsumner@fb.com>
parents: 23895
diff changeset
   896
    version = '02'
23181
832b7ef275c8 changegroup: introduce cg2packer/unpacker
Sune Foldager <cryo@cyanite.org>
parents: 23178
diff changeset
   897
    deltaheader = _CHANGEGROUPV2_DELTA_HEADER
832b7ef275c8 changegroup: introduce cg2packer/unpacker
Sune Foldager <cryo@cyanite.org>
parents: 23178
diff changeset
   898
24911
5447b8523fef cg2packer: set reorder=False in __init__ instead of in group()
Martin von Zweigbergk <martinvonz@google.com>
parents: 24910
diff changeset
   899
    def __init__(self, repo, bundlecaps=None):
5447b8523fef cg2packer: set reorder=False in __init__ instead of in group()
Martin von Zweigbergk <martinvonz@google.com>
parents: 24910
diff changeset
   900
        super(cg2packer, self).__init__(repo, bundlecaps)
5447b8523fef cg2packer: set reorder=False in __init__ instead of in group()
Martin von Zweigbergk <martinvonz@google.com>
parents: 24910
diff changeset
   901
        if self._reorder is None:
5447b8523fef cg2packer: set reorder=False in __init__ instead of in group()
Martin von Zweigbergk <martinvonz@google.com>
parents: 24910
diff changeset
   902
            # Since generaldelta is directly supported by cg2, reordering
5447b8523fef cg2packer: set reorder=False in __init__ instead of in group()
Martin von Zweigbergk <martinvonz@google.com>
parents: 24910
diff changeset
   903
            # generally doesn't help, so we disable it by default (treating
5447b8523fef cg2packer: set reorder=False in __init__ instead of in group()
Martin von Zweigbergk <martinvonz@google.com>
parents: 24910
diff changeset
   904
            # bundle.reorder=auto just like bundle.reorder=False).
5447b8523fef cg2packer: set reorder=False in __init__ instead of in group()
Martin von Zweigbergk <martinvonz@google.com>
parents: 24910
diff changeset
   905
            self._reorder = False
23181
832b7ef275c8 changegroup: introduce cg2packer/unpacker
Sune Foldager <cryo@cyanite.org>
parents: 23178
diff changeset
   906
832b7ef275c8 changegroup: introduce cg2packer/unpacker
Sune Foldager <cryo@cyanite.org>
parents: 23178
diff changeset
   907
    def deltaparent(self, revlog, rev, p1, p2, prev):
832b7ef275c8 changegroup: introduce cg2packer/unpacker
Sune Foldager <cryo@cyanite.org>
parents: 23178
diff changeset
   908
        dp = revlog.deltaparent(rev)
832b7ef275c8 changegroup: introduce cg2packer/unpacker
Sune Foldager <cryo@cyanite.org>
parents: 23178
diff changeset
   909
        # avoid storing full revisions; pick prev in those cases
832b7ef275c8 changegroup: introduce cg2packer/unpacker
Sune Foldager <cryo@cyanite.org>
parents: 23178
diff changeset
   910
        # also pick prev when we can't be sure remote has dp
832b7ef275c8 changegroup: introduce cg2packer/unpacker
Sune Foldager <cryo@cyanite.org>
parents: 23178
diff changeset
   911
        if dp == nullrev or (dp != p1 and dp != p2 and dp != prev):
832b7ef275c8 changegroup: introduce cg2packer/unpacker
Sune Foldager <cryo@cyanite.org>
parents: 23178
diff changeset
   912
            return prev
832b7ef275c8 changegroup: introduce cg2packer/unpacker
Sune Foldager <cryo@cyanite.org>
parents: 23178
diff changeset
   913
        return dp
832b7ef275c8 changegroup: introduce cg2packer/unpacker
Sune Foldager <cryo@cyanite.org>
parents: 23178
diff changeset
   914
27433
12f727a5b434 changegroup: add flags field to cg3 delta header
Mike Edgar <adgar@google.com>
parents: 27432
diff changeset
   915
    def builddeltaheader(self, node, p1n, p2n, basenode, linknode, flags):
12f727a5b434 changegroup: add flags field to cg3 delta header
Mike Edgar <adgar@google.com>
parents: 27432
diff changeset
   916
        # Do nothing with flags, it is implicitly 0 in cg1 and cg2
23181
832b7ef275c8 changegroup: introduce cg2packer/unpacker
Sune Foldager <cryo@cyanite.org>
parents: 23178
diff changeset
   917
        return struct.pack(self.deltaheader, node, p1n, p2n, basenode, linknode)
832b7ef275c8 changegroup: introduce cg2packer/unpacker
Sune Foldager <cryo@cyanite.org>
parents: 23178
diff changeset
   918
27432
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   919
class cg3packer(cg2packer):
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   920
    version = '03'
27433
12f727a5b434 changegroup: add flags field to cg3 delta header
Mike Edgar <adgar@google.com>
parents: 27432
diff changeset
   921
    deltaheader = _CHANGEGROUPV3_DELTA_HEADER
27432
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   922
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   923
    def _packmanifests(self, mfnodes, tmfnodes, lookuplinknode):
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   924
        # Note that debug prints are super confusing in this code, as
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   925
        # tmfnodes gets populated by the calls to lookuplinknode in
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   926
        # the superclass's manifest packer. In the future we should
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   927
        # probably see if we can refactor this somehow to be less
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   928
        # confusing.
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   929
        for x in super(cg3packer, self)._packmanifests(
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   930
            mfnodes, {}, lookuplinknode):
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   931
            yield x
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   932
        dirlog = self._repo.manifest.dirlog
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   933
        for name, nodes in tmfnodes.iteritems():
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   934
            # For now, directory headers are simply file headers with
27691
b0d23a55f91e changegroup: don't add a second trailing '/' in dir name
Martin von Zweigbergk <martinvonz@google.com>
parents: 27690
diff changeset
   935
            # a trailing '/' on the path (already in the name).
b0d23a55f91e changegroup: don't add a second trailing '/' in dir name
Martin von Zweigbergk <martinvonz@google.com>
parents: 27690
diff changeset
   936
            yield self.fileheader(name)
27432
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   937
            for chunk in self.group(nodes, dirlog(name), nodes.get):
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   938
                yield chunk
27753
d4071cc73f46 changegroup3: add empty chunk separating directories and files
Martin von Zweigbergk <martinvonz@google.com>
parents: 27752
diff changeset
   939
        yield self.close()
27432
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   940
27433
12f727a5b434 changegroup: add flags field to cg3 delta header
Mike Edgar <adgar@google.com>
parents: 27432
diff changeset
   941
    def builddeltaheader(self, node, p1n, p2n, basenode, linknode, flags):
12f727a5b434 changegroup: add flags field to cg3 delta header
Mike Edgar <adgar@google.com>
parents: 27432
diff changeset
   942
        return struct.pack(
12f727a5b434 changegroup: add flags field to cg3 delta header
Mike Edgar <adgar@google.com>
parents: 27432
diff changeset
   943
            self.deltaheader, node, p1n, p2n, basenode, linknode, flags)
27432
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   944
27751
a40e2f7fe49d changegroup: hide packermap behind methods
Martin von Zweigbergk <martinvonz@google.com>
parents: 27739
diff changeset
   945
_packermap = {'01': (cg1packer, cg1unpacker),
26709
42733e956887 changegroup: reformat packermap and add comment
Augie Fackler <augie@google.com>
parents: 26708
diff changeset
   946
             # cg2 adds support for exchanging generaldelta
42733e956887 changegroup: reformat packermap and add comment
Augie Fackler <augie@google.com>
parents: 26708
diff changeset
   947
             '02': (cg2packer, cg2unpacker),
27753
d4071cc73f46 changegroup3: add empty chunk separating directories and files
Martin von Zweigbergk <martinvonz@google.com>
parents: 27752
diff changeset
   948
             # cg3 adds support for exchanging revlog flags and treemanifests
27432
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
   949
             '03': (cg3packer, cg3unpacker),
26709
42733e956887 changegroup: reformat packermap and add comment
Augie Fackler <augie@google.com>
parents: 26708
diff changeset
   950
}
23168
a92ba36a1a9d changegroup: add a "packermap" dictionary to track different packer versions
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 22971
diff changeset
   951
27953
88609cfa3745 changegroup: fix pulling to treemanifest repo from flat repo (issue5066)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27946
diff changeset
   952
def allsupportedversions(ui):
27928
c0f11347b107 changegroup: don't support versions 01 and 02 with treemanifests
Martin von Zweigbergk <martinvonz@google.com>
parents: 27920
diff changeset
   953
    versions = set(_packermap.keys())
27953
88609cfa3745 changegroup: fix pulling to treemanifest repo from flat repo (issue5066)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27946
diff changeset
   954
    versions.discard('03')
88609cfa3745 changegroup: fix pulling to treemanifest repo from flat repo (issue5066)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27946
diff changeset
   955
    if (ui.configbool('experimental', 'changegroup3') or
88609cfa3745 changegroup: fix pulling to treemanifest repo from flat repo (issue5066)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27946
diff changeset
   956
        ui.configbool('experimental', 'treemanifest')):
88609cfa3745 changegroup: fix pulling to treemanifest repo from flat repo (issue5066)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27946
diff changeset
   957
        versions.add('03')
88609cfa3745 changegroup: fix pulling to treemanifest repo from flat repo (issue5066)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27946
diff changeset
   958
    return versions
88609cfa3745 changegroup: fix pulling to treemanifest repo from flat repo (issue5066)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27946
diff changeset
   959
88609cfa3745 changegroup: fix pulling to treemanifest repo from flat repo (issue5066)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27946
diff changeset
   960
# Changegroup versions that can be applied to the repo
88609cfa3745 changegroup: fix pulling to treemanifest repo from flat repo (issue5066)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27946
diff changeset
   961
def supportedincomingversions(repo):
88609cfa3745 changegroup: fix pulling to treemanifest repo from flat repo (issue5066)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27946
diff changeset
   962
    versions = allsupportedversions(repo.ui)
88609cfa3745 changegroup: fix pulling to treemanifest repo from flat repo (issue5066)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27946
diff changeset
   963
    if 'treemanifest' in repo.requirements:
88609cfa3745 changegroup: fix pulling to treemanifest repo from flat repo (issue5066)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27946
diff changeset
   964
        versions.add('03')
88609cfa3745 changegroup: fix pulling to treemanifest repo from flat repo (issue5066)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27946
diff changeset
   965
    return versions
88609cfa3745 changegroup: fix pulling to treemanifest repo from flat repo (issue5066)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27946
diff changeset
   966
88609cfa3745 changegroup: fix pulling to treemanifest repo from flat repo (issue5066)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27946
diff changeset
   967
# Changegroup versions that can be created from the repo
88609cfa3745 changegroup: fix pulling to treemanifest repo from flat repo (issue5066)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27946
diff changeset
   968
def supportedoutgoingversions(repo):
88609cfa3745 changegroup: fix pulling to treemanifest repo from flat repo (issue5066)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27946
diff changeset
   969
    versions = allsupportedversions(repo.ui)
88609cfa3745 changegroup: fix pulling to treemanifest repo from flat repo (issue5066)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27946
diff changeset
   970
    if 'treemanifest' in repo.requirements:
27928
c0f11347b107 changegroup: don't support versions 01 and 02 with treemanifests
Martin von Zweigbergk <martinvonz@google.com>
parents: 27920
diff changeset
   971
        # Versions 01 and 02 support only flat manifests and it's just too
c0f11347b107 changegroup: don't support versions 01 and 02 with treemanifests
Martin von Zweigbergk <martinvonz@google.com>
parents: 27920
diff changeset
   972
        # expensive to convert between the flat manifest and tree manifest on
c0f11347b107 changegroup: don't support versions 01 and 02 with treemanifests
Martin von Zweigbergk <martinvonz@google.com>
parents: 27920
diff changeset
   973
        # the fly. Since tree manifests are hashed differently, all of history
c0f11347b107 changegroup: don't support versions 01 and 02 with treemanifests
Martin von Zweigbergk <martinvonz@google.com>
parents: 27920
diff changeset
   974
        # would have to be converted. Instead, we simply don't even pretend to
c0f11347b107 changegroup: don't support versions 01 and 02 with treemanifests
Martin von Zweigbergk <martinvonz@google.com>
parents: 27920
diff changeset
   975
        # support versions 01 and 02.
c0f11347b107 changegroup: don't support versions 01 and 02 with treemanifests
Martin von Zweigbergk <martinvonz@google.com>
parents: 27920
diff changeset
   976
        versions.discard('01')
c0f11347b107 changegroup: don't support versions 01 and 02 with treemanifests
Martin von Zweigbergk <martinvonz@google.com>
parents: 27920
diff changeset
   977
        versions.discard('02')
27953
88609cfa3745 changegroup: fix pulling to treemanifest repo from flat repo (issue5066)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27946
diff changeset
   978
        versions.add('03')
27752
29cfc474c5fd changegroup3: introduce experimental.changegroup3 boolean config
Martin von Zweigbergk <martinvonz@google.com>
parents: 27751
diff changeset
   979
    return versions
27751
a40e2f7fe49d changegroup: hide packermap behind methods
Martin von Zweigbergk <martinvonz@google.com>
parents: 27739
diff changeset
   980
27929
3b2ac2115464 changegroup: introduce safeversion()
Martin von Zweigbergk <martinvonz@google.com>
parents: 27928
diff changeset
   981
def safeversion(repo):
3b2ac2115464 changegroup: introduce safeversion()
Martin von Zweigbergk <martinvonz@google.com>
parents: 27928
diff changeset
   982
    # Finds the smallest version that it's safe to assume clients of the repo
27931
1289a122cf3f shelve: use cg3 for treemanifests
Martin von Zweigbergk <martinvonz@google.com>
parents: 27929
diff changeset
   983
    # will support. For example, all hg versions that support generaldelta also
1289a122cf3f shelve: use cg3 for treemanifests
Martin von Zweigbergk <martinvonz@google.com>
parents: 27929
diff changeset
   984
    # support changegroup 02.
27953
88609cfa3745 changegroup: fix pulling to treemanifest repo from flat repo (issue5066)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27946
diff changeset
   985
    versions = supportedoutgoingversions(repo)
27929
3b2ac2115464 changegroup: introduce safeversion()
Martin von Zweigbergk <martinvonz@google.com>
parents: 27928
diff changeset
   986
    if 'generaldelta' in repo.requirements:
3b2ac2115464 changegroup: introduce safeversion()
Martin von Zweigbergk <martinvonz@google.com>
parents: 27928
diff changeset
   987
        versions.discard('01')
3b2ac2115464 changegroup: introduce safeversion()
Martin von Zweigbergk <martinvonz@google.com>
parents: 27928
diff changeset
   988
    assert versions
3b2ac2115464 changegroup: introduce safeversion()
Martin von Zweigbergk <martinvonz@google.com>
parents: 27928
diff changeset
   989
    return min(versions)
3b2ac2115464 changegroup: introduce safeversion()
Martin von Zweigbergk <martinvonz@google.com>
parents: 27928
diff changeset
   990
27751
a40e2f7fe49d changegroup: hide packermap behind methods
Martin von Zweigbergk <martinvonz@google.com>
parents: 27739
diff changeset
   991
def getbundler(version, repo, bundlecaps=None):
27953
88609cfa3745 changegroup: fix pulling to treemanifest repo from flat repo (issue5066)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27946
diff changeset
   992
    assert version in supportedoutgoingversions(repo)
27751
a40e2f7fe49d changegroup: hide packermap behind methods
Martin von Zweigbergk <martinvonz@google.com>
parents: 27739
diff changeset
   993
    return _packermap[version][0](repo, bundlecaps)
a40e2f7fe49d changegroup: hide packermap behind methods
Martin von Zweigbergk <martinvonz@google.com>
parents: 27739
diff changeset
   994
a40e2f7fe49d changegroup: hide packermap behind methods
Martin von Zweigbergk <martinvonz@google.com>
parents: 27739
diff changeset
   995
def getunbundler(version, fh, alg):
a40e2f7fe49d changegroup: hide packermap behind methods
Martin von Zweigbergk <martinvonz@google.com>
parents: 27739
diff changeset
   996
    return _packermap[version][1](fh, alg)
a40e2f7fe49d changegroup: hide packermap behind methods
Martin von Zweigbergk <martinvonz@google.com>
parents: 27739
diff changeset
   997
20926
7c1ed40e3325 localrepo: move the changegroupinfo method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20925
diff changeset
   998
def _changegroupinfo(repo, nodes, source):
7c1ed40e3325 localrepo: move the changegroupinfo method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20925
diff changeset
   999
    if repo.ui.verbose or source == 'bundle':
7c1ed40e3325 localrepo: move the changegroupinfo method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20925
diff changeset
  1000
        repo.ui.status(_("%d changesets found\n") % len(nodes))
7c1ed40e3325 localrepo: move the changegroupinfo method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20925
diff changeset
  1001
    if repo.ui.debugflag:
7c1ed40e3325 localrepo: move the changegroupinfo method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20925
diff changeset
  1002
        repo.ui.debug("list of changesets:\n")
7c1ed40e3325 localrepo: move the changegroupinfo method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20925
diff changeset
  1003
        for node in nodes:
7c1ed40e3325 localrepo: move the changegroupinfo method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20925
diff changeset
  1004
            repo.ui.debug("%s\n" % hex(node))
7c1ed40e3325 localrepo: move the changegroupinfo method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20925
diff changeset
  1005
23177
706547a14b8b changegroup: introduce "raw" versions of some commands
Sune Foldager <cryo@cyanite.org>
parents: 23168
diff changeset
  1006
def getsubsetraw(repo, outgoing, bundler, source, fastpath=False):
20925
5174c48ed8d8 localrepo: move the _changegroupsubset method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20675
diff changeset
  1007
    repo = repo.unfiltered()
5174c48ed8d8 localrepo: move the _changegroupsubset method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20675
diff changeset
  1008
    commonrevs = outgoing.common
5174c48ed8d8 localrepo: move the _changegroupsubset method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20675
diff changeset
  1009
    csets = outgoing.missing
5174c48ed8d8 localrepo: move the _changegroupsubset method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20675
diff changeset
  1010
    heads = outgoing.missingheads
5174c48ed8d8 localrepo: move the _changegroupsubset method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20675
diff changeset
  1011
    # We go through the fast path if we get told to, or if all (unfiltered
5174c48ed8d8 localrepo: move the _changegroupsubset method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20675
diff changeset
  1012
    # heads have been requested (since we then know there all linkrevs will
5174c48ed8d8 localrepo: move the _changegroupsubset method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20675
diff changeset
  1013
    # be pulled by the client).
5174c48ed8d8 localrepo: move the _changegroupsubset method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20675
diff changeset
  1014
    heads.sort()
5174c48ed8d8 localrepo: move the _changegroupsubset method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20675
diff changeset
  1015
    fastpathlinkrev = fastpath or (
5174c48ed8d8 localrepo: move the _changegroupsubset method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20675
diff changeset
  1016
            repo.filtername is None and heads == sorted(repo.heads()))
5174c48ed8d8 localrepo: move the _changegroupsubset method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20675
diff changeset
  1017
5174c48ed8d8 localrepo: move the _changegroupsubset method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20675
diff changeset
  1018
    repo.hook('preoutgoing', throw=True, source=source)
20926
7c1ed40e3325 localrepo: move the changegroupinfo method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20925
diff changeset
  1019
    _changegroupinfo(repo, csets, source)
23177
706547a14b8b changegroup: introduce "raw" versions of some commands
Sune Foldager <cryo@cyanite.org>
parents: 23168
diff changeset
  1020
    return bundler.generate(commonrevs, csets, fastpathlinkrev, source)
706547a14b8b changegroup: introduce "raw" versions of some commands
Sune Foldager <cryo@cyanite.org>
parents: 23168
diff changeset
  1021
26595
be0489770925 getsubset: get the unpacker version from the bundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26587
diff changeset
  1022
def getsubset(repo, outgoing, bundler, source, fastpath=False):
23177
706547a14b8b changegroup: introduce "raw" versions of some commands
Sune Foldager <cryo@cyanite.org>
parents: 23168
diff changeset
  1023
    gengroup = getsubsetraw(repo, outgoing, bundler, source, fastpath)
27751
a40e2f7fe49d changegroup: hide packermap behind methods
Martin von Zweigbergk <martinvonz@google.com>
parents: 27739
diff changeset
  1024
    return getunbundler(bundler.version, util.chunkbuffer(gengroup), None)
20927
24a443948627 localrepo: move the changegroupsubset method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20926
diff changeset
  1025
23897
f99a6e1865e5 changegroup.getsubset: support multiple versions
Eric Sumner <ericsumner@fb.com>
parents: 23896
diff changeset
  1026
def changegroupsubset(repo, roots, heads, source, version='01'):
20927
24a443948627 localrepo: move the changegroupsubset method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20926
diff changeset
  1027
    """Compute a changegroup consisting of all the nodes that are
24a443948627 localrepo: move the changegroupsubset method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20926
diff changeset
  1028
    descendants of any of the roots and ancestors of any of the heads.
24a443948627 localrepo: move the changegroupsubset method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20926
diff changeset
  1029
    Return a chunkbuffer object whose read() method will return
24a443948627 localrepo: move the changegroupsubset method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20926
diff changeset
  1030
    successive changegroup chunks.
24a443948627 localrepo: move the changegroupsubset method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20926
diff changeset
  1031
24a443948627 localrepo: move the changegroupsubset method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20926
diff changeset
  1032
    It is fairly complex as determining which filenodes and which
24a443948627 localrepo: move the changegroupsubset method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20926
diff changeset
  1033
    manifest nodes need to be included for the changeset to be complete
24a443948627 localrepo: move the changegroupsubset method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20926
diff changeset
  1034
    is non-trivial.
24a443948627 localrepo: move the changegroupsubset method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20926
diff changeset
  1035
24a443948627 localrepo: move the changegroupsubset method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20926
diff changeset
  1036
    Another wrinkle is doing the reverse, figuring out which changeset in
24a443948627 localrepo: move the changegroupsubset method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20926
diff changeset
  1037
    the changegroup a particular filenode or manifestnode belongs to.
24a443948627 localrepo: move the changegroupsubset method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20926
diff changeset
  1038
    """
24a443948627 localrepo: move the changegroupsubset method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20926
diff changeset
  1039
    cl = repo.changelog
24a443948627 localrepo: move the changegroupsubset method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20926
diff changeset
  1040
    if not roots:
24a443948627 localrepo: move the changegroupsubset method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20926
diff changeset
  1041
        roots = [nullid]
24a443948627 localrepo: move the changegroupsubset method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20926
diff changeset
  1042
    discbases = []
24a443948627 localrepo: move the changegroupsubset method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20926
diff changeset
  1043
    for n in roots:
24a443948627 localrepo: move the changegroupsubset method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20926
diff changeset
  1044
        discbases.extend([p for p in cl.parents(n) if p != nullid])
25677
af5b2f4ed594 changegroup: properly compute common base in changeggroupsubset (issue4736)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24717
diff changeset
  1045
    # TODO: remove call to nodesbetween.
af5b2f4ed594 changegroup: properly compute common base in changeggroupsubset (issue4736)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24717
diff changeset
  1046
    csets, roots, heads = cl.nodesbetween(roots, heads)
af5b2f4ed594 changegroup: properly compute common base in changeggroupsubset (issue4736)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24717
diff changeset
  1047
    included = set(csets)
af5b2f4ed594 changegroup: properly compute common base in changeggroupsubset (issue4736)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24717
diff changeset
  1048
    discbases = [n for n in discbases if n not in included]
20927
24a443948627 localrepo: move the changegroupsubset method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20926
diff changeset
  1049
    outgoing = discovery.outgoing(cl, discbases, heads)
27751
a40e2f7fe49d changegroup: hide packermap behind methods
Martin von Zweigbergk <martinvonz@google.com>
parents: 27739
diff changeset
  1050
    bundler = getbundler(version, repo)
26595
be0489770925 getsubset: get the unpacker version from the bundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26587
diff changeset
  1051
    return getsubset(repo, outgoing, bundler, source)
20927
24a443948627 localrepo: move the changegroupsubset method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20926
diff changeset
  1052
23178
5e895ed5e955 changegroup: allow use of different cg#packer in getchangegroupraw
Sune Foldager <cryo@cyanite.org>
parents: 23177
diff changeset
  1053
def getlocalchangegroupraw(repo, source, outgoing, bundlecaps=None,
5e895ed5e955 changegroup: allow use of different cg#packer in getchangegroupraw
Sune Foldager <cryo@cyanite.org>
parents: 23177
diff changeset
  1054
                           version='01'):
23177
706547a14b8b changegroup: introduce "raw" versions of some commands
Sune Foldager <cryo@cyanite.org>
parents: 23168
diff changeset
  1055
    """Like getbundle, but taking a discovery.outgoing as an argument.
706547a14b8b changegroup: introduce "raw" versions of some commands
Sune Foldager <cryo@cyanite.org>
parents: 23168
diff changeset
  1056
706547a14b8b changegroup: introduce "raw" versions of some commands
Sune Foldager <cryo@cyanite.org>
parents: 23168
diff changeset
  1057
    This is only implemented for local repos and reuses potentially
706547a14b8b changegroup: introduce "raw" versions of some commands
Sune Foldager <cryo@cyanite.org>
parents: 23168
diff changeset
  1058
    precomputed sets in outgoing. Returns a raw changegroup generator."""
706547a14b8b changegroup: introduce "raw" versions of some commands
Sune Foldager <cryo@cyanite.org>
parents: 23168
diff changeset
  1059
    if not outgoing.missing:
706547a14b8b changegroup: introduce "raw" versions of some commands
Sune Foldager <cryo@cyanite.org>
parents: 23168
diff changeset
  1060
        return None
27751
a40e2f7fe49d changegroup: hide packermap behind methods
Martin von Zweigbergk <martinvonz@google.com>
parents: 27739
diff changeset
  1061
    bundler = getbundler(version, repo, bundlecaps)
23177
706547a14b8b changegroup: introduce "raw" versions of some commands
Sune Foldager <cryo@cyanite.org>
parents: 23168
diff changeset
  1062
    return getsubsetraw(repo, outgoing, bundler, source)
706547a14b8b changegroup: introduce "raw" versions of some commands
Sune Foldager <cryo@cyanite.org>
parents: 23168
diff changeset
  1063
26508
47a12de9ac3f changegroup: add version argument to getlocalchangegroup
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26424
diff changeset
  1064
def getlocalchangegroup(repo, source, outgoing, bundlecaps=None,
47a12de9ac3f changegroup: add version argument to getlocalchangegroup
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26424
diff changeset
  1065
                        version='01'):
20928
91b47139d0cb localrepo: move the getlocalbundle method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20927
diff changeset
  1066
    """Like getbundle, but taking a discovery.outgoing as an argument.
91b47139d0cb localrepo: move the getlocalbundle method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20927
diff changeset
  1067
91b47139d0cb localrepo: move the getlocalbundle method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20927
diff changeset
  1068
    This is only implemented for local repos and reuses potentially
91b47139d0cb localrepo: move the getlocalbundle method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20927
diff changeset
  1069
    precomputed sets in outgoing."""
91b47139d0cb localrepo: move the getlocalbundle method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20927
diff changeset
  1070
    if not outgoing.missing:
91b47139d0cb localrepo: move the getlocalbundle method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20927
diff changeset
  1071
        return None
27751
a40e2f7fe49d changegroup: hide packermap behind methods
Martin von Zweigbergk <martinvonz@google.com>
parents: 27739
diff changeset
  1072
    bundler = getbundler(version, repo, bundlecaps)
20928
91b47139d0cb localrepo: move the getlocalbundle method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20927
diff changeset
  1073
    return getsubset(repo, outgoing, bundler, source)
91b47139d0cb localrepo: move the getlocalbundle method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20927
diff changeset
  1074
25400
7759dc97c5c7 changegroup: rename _computeoutgoing to computeoutgoing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 24978
diff changeset
  1075
def computeoutgoing(repo, heads, common):
21260
aa3e56607675 changegroup: refactor outgoing logic into a function
Durham Goode <durham@fb.com>
parents: 21153
diff changeset
  1076
    """Computes which revs are outgoing given a set of common
aa3e56607675 changegroup: refactor outgoing logic into a function
Durham Goode <durham@fb.com>
parents: 21153
diff changeset
  1077
    and a set of heads.
aa3e56607675 changegroup: refactor outgoing logic into a function
Durham Goode <durham@fb.com>
parents: 21153
diff changeset
  1078
aa3e56607675 changegroup: refactor outgoing logic into a function
Durham Goode <durham@fb.com>
parents: 21153
diff changeset
  1079
    This is a separate function so extensions can have access to
aa3e56607675 changegroup: refactor outgoing logic into a function
Durham Goode <durham@fb.com>
parents: 21153
diff changeset
  1080
    the logic.
aa3e56607675 changegroup: refactor outgoing logic into a function
Durham Goode <durham@fb.com>
parents: 21153
diff changeset
  1081
aa3e56607675 changegroup: refactor outgoing logic into a function
Durham Goode <durham@fb.com>
parents: 21153
diff changeset
  1082
    Returns a discovery.outgoing object.
aa3e56607675 changegroup: refactor outgoing logic into a function
Durham Goode <durham@fb.com>
parents: 21153
diff changeset
  1083
    """
aa3e56607675 changegroup: refactor outgoing logic into a function
Durham Goode <durham@fb.com>
parents: 21153
diff changeset
  1084
    cl = repo.changelog
aa3e56607675 changegroup: refactor outgoing logic into a function
Durham Goode <durham@fb.com>
parents: 21153
diff changeset
  1085
    if common:
aa3e56607675 changegroup: refactor outgoing logic into a function
Durham Goode <durham@fb.com>
parents: 21153
diff changeset
  1086
        hasnode = cl.hasnode
aa3e56607675 changegroup: refactor outgoing logic into a function
Durham Goode <durham@fb.com>
parents: 21153
diff changeset
  1087
        common = [n for n in common if hasnode(n)]
aa3e56607675 changegroup: refactor outgoing logic into a function
Durham Goode <durham@fb.com>
parents: 21153
diff changeset
  1088
    else:
aa3e56607675 changegroup: refactor outgoing logic into a function
Durham Goode <durham@fb.com>
parents: 21153
diff changeset
  1089
        common = [nullid]
aa3e56607675 changegroup: refactor outgoing logic into a function
Durham Goode <durham@fb.com>
parents: 21153
diff changeset
  1090
    if not heads:
aa3e56607675 changegroup: refactor outgoing logic into a function
Durham Goode <durham@fb.com>
parents: 21153
diff changeset
  1091
        heads = cl.heads()
aa3e56607675 changegroup: refactor outgoing logic into a function
Durham Goode <durham@fb.com>
parents: 21153
diff changeset
  1092
    return discovery.outgoing(cl, common, heads)
aa3e56607675 changegroup: refactor outgoing logic into a function
Durham Goode <durham@fb.com>
parents: 21153
diff changeset
  1093
26509
83d82fbefccb changegroup: add version argument to getchangegroup
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26508
diff changeset
  1094
def getchangegroup(repo, source, heads=None, common=None, bundlecaps=None,
83d82fbefccb changegroup: add version argument to getchangegroup
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26508
diff changeset
  1095
                   version='01'):
20930
4a987060d97e localrepo: move the getbundle method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20928
diff changeset
  1096
    """Like changegroupsubset, but returns the set difference between the
4a987060d97e localrepo: move the getbundle method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20928
diff changeset
  1097
    ancestors of heads and the ancestors common.
4a987060d97e localrepo: move the getbundle method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20928
diff changeset
  1098
4a987060d97e localrepo: move the getbundle method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20928
diff changeset
  1099
    If heads is None, use the local heads. If common is None, use [nullid].
4a987060d97e localrepo: move the getbundle method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20928
diff changeset
  1100
4a987060d97e localrepo: move the getbundle method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20928
diff changeset
  1101
    The nodes in common might not all be known locally due to the way the
4a987060d97e localrepo: move the getbundle method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20928
diff changeset
  1102
    current discovery protocol works.
4a987060d97e localrepo: move the getbundle method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20928
diff changeset
  1103
    """
25400
7759dc97c5c7 changegroup: rename _computeoutgoing to computeoutgoing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 24978
diff changeset
  1104
    outgoing = computeoutgoing(repo, heads, common)
26509
83d82fbefccb changegroup: add version argument to getchangegroup
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26508
diff changeset
  1105
    return getlocalchangegroup(repo, source, outgoing, bundlecaps=bundlecaps,
83d82fbefccb changegroup: add version argument to getchangegroup
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26508
diff changeset
  1106
                               version=version)
20930
4a987060d97e localrepo: move the getbundle method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20928
diff changeset
  1107
20931
de60ca3a390e localrepo: move the changegroup method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20930
diff changeset
  1108
def changegroup(repo, basenodes, source):
de60ca3a390e localrepo: move the changegroup method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20930
diff changeset
  1109
    # to avoid a race we use changegroupsubset() (issue1320)
de60ca3a390e localrepo: move the changegroup method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20930
diff changeset
  1110
    return changegroupsubset(repo, basenodes, repo.heads(), source)
de60ca3a390e localrepo: move the changegroup method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20930
diff changeset
  1111
27735
bd37212c20ed changegroup: remove now-unused 'wasempty' variable and parameter
Martin von Zweigbergk <martinvonz@google.com>
parents: 27734
diff changeset
  1112
def _addchangegroupfiles(repo, source, revmap, trp, pr, needfiles):
20932
0ac83e4e4f7c localrepo: move the addchangegroupfiles method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20931
diff changeset
  1113
    revisions = 0
0ac83e4e4f7c localrepo: move the addchangegroupfiles method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20931
diff changeset
  1114
    files = 0
0ac83e4e4f7c localrepo: move the addchangegroupfiles method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20931
diff changeset
  1115
    while True:
0ac83e4e4f7c localrepo: move the addchangegroupfiles method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20931
diff changeset
  1116
        chunkdata = source.filelogheader()
0ac83e4e4f7c localrepo: move the addchangegroupfiles method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20931
diff changeset
  1117
        if not chunkdata:
0ac83e4e4f7c localrepo: move the addchangegroupfiles method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20931
diff changeset
  1118
            break
0ac83e4e4f7c localrepo: move the addchangegroupfiles method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20931
diff changeset
  1119
        f = chunkdata["filename"]
0ac83e4e4f7c localrepo: move the addchangegroupfiles method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20931
diff changeset
  1120
        repo.ui.debug("adding %s revisions\n" % f)
0ac83e4e4f7c localrepo: move the addchangegroupfiles method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20931
diff changeset
  1121
        pr()
27754
a09f143daaf4 changegroup3: move treemanifest support into _unpackmanifests()
Martin von Zweigbergk <martinvonz@google.com>
parents: 27753
diff changeset
  1122
        fl = repo.file(f)
20932
0ac83e4e4f7c localrepo: move the addchangegroupfiles method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20931
diff changeset
  1123
        o = len(fl)
24120
a450e0a2ba0a revlog: in addgroup, reject ill-formed deltas based on censored nodes
Mike Edgar <adgar@google.com>
parents: 23897
diff changeset
  1124
        try:
a450e0a2ba0a revlog: in addgroup, reject ill-formed deltas based on censored nodes
Mike Edgar <adgar@google.com>
parents: 23897
diff changeset
  1125
            if not fl.addgroup(source, revmap, trp):
26587
56b2bcea2529 error: get Abort from 'error' instead of 'util'
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26540
diff changeset
  1126
                raise error.Abort(_("received file revlog group is empty"))
25660
328739ea70c3 global: mass rewrite to use modern exception syntax
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25624
diff changeset
  1127
        except error.CensoredBaseError as e:
26587
56b2bcea2529 error: get Abort from 'error' instead of 'util'
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26540
diff changeset
  1128
            raise error.Abort(_("received delta base is censored: %s") % e)
27754
a09f143daaf4 changegroup3: move treemanifest support into _unpackmanifests()
Martin von Zweigbergk <martinvonz@google.com>
parents: 27753
diff changeset
  1129
        revisions += len(fl) - o
a09f143daaf4 changegroup3: move treemanifest support into _unpackmanifests()
Martin von Zweigbergk <martinvonz@google.com>
parents: 27753
diff changeset
  1130
        files += 1
20932
0ac83e4e4f7c localrepo: move the addchangegroupfiles method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20931
diff changeset
  1131
        if f in needfiles:
0ac83e4e4f7c localrepo: move the addchangegroupfiles method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20931
diff changeset
  1132
            needs = needfiles[f]
0ac83e4e4f7c localrepo: move the addchangegroupfiles method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20931
diff changeset
  1133
            for new in xrange(o, len(fl)):
0ac83e4e4f7c localrepo: move the addchangegroupfiles method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20931
diff changeset
  1134
                n = fl.node(new)
0ac83e4e4f7c localrepo: move the addchangegroupfiles method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20931
diff changeset
  1135
                if n in needs:
0ac83e4e4f7c localrepo: move the addchangegroupfiles method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20931
diff changeset
  1136
                    needs.remove(n)
0ac83e4e4f7c localrepo: move the addchangegroupfiles method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20931
diff changeset
  1137
                else:
26587
56b2bcea2529 error: get Abort from 'error' instead of 'util'
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26540
diff changeset
  1138
                    raise error.Abort(
20932
0ac83e4e4f7c localrepo: move the addchangegroupfiles method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20931
diff changeset
  1139
                        _("received spurious file revlog entry"))
0ac83e4e4f7c localrepo: move the addchangegroupfiles method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20931
diff changeset
  1140
            if not needs:
0ac83e4e4f7c localrepo: move the addchangegroupfiles method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20931
diff changeset
  1141
                del needfiles[f]
0ac83e4e4f7c localrepo: move the addchangegroupfiles method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20931
diff changeset
  1142
    repo.ui.progress(_('files'), None)
0ac83e4e4f7c localrepo: move the addchangegroupfiles method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20931
diff changeset
  1143
0ac83e4e4f7c localrepo: move the addchangegroupfiles method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20931
diff changeset
  1144
    for f, needs in needfiles.iteritems():
0ac83e4e4f7c localrepo: move the addchangegroupfiles method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20931
diff changeset
  1145
        fl = repo.file(f)
0ac83e4e4f7c localrepo: move the addchangegroupfiles method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20931
diff changeset
  1146
        for n in needs:
0ac83e4e4f7c localrepo: move the addchangegroupfiles method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20931
diff changeset
  1147
            try:
0ac83e4e4f7c localrepo: move the addchangegroupfiles method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20931
diff changeset
  1148
                fl.rev(n)
0ac83e4e4f7c localrepo: move the addchangegroupfiles method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20931
diff changeset
  1149
            except error.LookupError:
26587
56b2bcea2529 error: get Abort from 'error' instead of 'util'
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26540
diff changeset
  1150
                raise error.Abort(
20932
0ac83e4e4f7c localrepo: move the addchangegroupfiles method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20931
diff changeset
  1151
                    _('missing file data for %s:%s - run hg verify') %
0ac83e4e4f7c localrepo: move the addchangegroupfiles method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20931
diff changeset
  1152
                    (f, hex(n)))
0ac83e4e4f7c localrepo: move the addchangegroupfiles method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20931
diff changeset
  1153
0ac83e4e4f7c localrepo: move the addchangegroupfiles method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20931
diff changeset
  1154
    return revisions, files