mercurial/bundle2.py
author Boris Feld <boris.feld@octobus.net>
Wed, 21 Feb 2018 17:26:22 +0100
changeset 36970 79b73be4dda5
parent 36969 9988fc10f49e
child 36971 b89a7ef29013
permissions -rw-r--r--
rev-branch-cache: add a function to generate a part The function is able to produce a rbc part consumed by the function introduced into previous changesets. More details on usage and impact in the next changesets.
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
20801
9c5183cb9bca bundle2: very first version of a bundle2 bundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
diff changeset
     1
# bundle2.py - generic container format to transmit arbitrary data.
9c5183cb9bca bundle2: very first version of a bundle2 bundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
diff changeset
     2
#
9c5183cb9bca bundle2: very first version of a bundle2 bundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
diff changeset
     3
# Copyright 2013 Facebook, Inc.
9c5183cb9bca bundle2: very first version of a bundle2 bundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
diff changeset
     4
#
9c5183cb9bca bundle2: very first version of a bundle2 bundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
diff changeset
     5
# This software may be used and distributed according to the terms of the
9c5183cb9bca bundle2: very first version of a bundle2 bundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
diff changeset
     6
# GNU General Public License version 2 or any later version.
9c5183cb9bca bundle2: very first version of a bundle2 bundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
diff changeset
     7
"""Handling of the new bundle2 format
9c5183cb9bca bundle2: very first version of a bundle2 bundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
diff changeset
     8
9c5183cb9bca bundle2: very first version of a bundle2 bundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
diff changeset
     9
The goal of bundle2 is to act as an atomically packet to transmit a set of
9c5183cb9bca bundle2: very first version of a bundle2 bundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
diff changeset
    10
payloads in an application agnostic way. It consist in a sequence of "parts"
9c5183cb9bca bundle2: very first version of a bundle2 bundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
diff changeset
    11
that will be handed to and processed by the application layer.
9c5183cb9bca bundle2: very first version of a bundle2 bundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
diff changeset
    12
9c5183cb9bca bundle2: very first version of a bundle2 bundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
diff changeset
    13
9c5183cb9bca bundle2: very first version of a bundle2 bundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
diff changeset
    14
General format architecture
9c5183cb9bca bundle2: very first version of a bundle2 bundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
diff changeset
    15
===========================
9c5183cb9bca bundle2: very first version of a bundle2 bundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
diff changeset
    16
9c5183cb9bca bundle2: very first version of a bundle2 bundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
diff changeset
    17
The format is architectured as follow
9c5183cb9bca bundle2: very first version of a bundle2 bundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
diff changeset
    18
9c5183cb9bca bundle2: very first version of a bundle2 bundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
diff changeset
    19
 - magic string
9c5183cb9bca bundle2: very first version of a bundle2 bundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
diff changeset
    20
 - stream level parameters
9c5183cb9bca bundle2: very first version of a bundle2 bundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
diff changeset
    21
 - payload parts (any number)
9c5183cb9bca bundle2: very first version of a bundle2 bundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
diff changeset
    22
 - end of stream marker.
9c5183cb9bca bundle2: very first version of a bundle2 bundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
diff changeset
    23
20856
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20844
diff changeset
    24
the Binary format
20801
9c5183cb9bca bundle2: very first version of a bundle2 bundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
diff changeset
    25
============================
9c5183cb9bca bundle2: very first version of a bundle2 bundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
diff changeset
    26
21024
7731a2281cf0 spelling: fixes from spell checker
Mads Kiilerich <madski@unity3d.com>
parents: 21020
diff changeset
    27
All numbers are unsigned and big-endian.
20801
9c5183cb9bca bundle2: very first version of a bundle2 bundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
diff changeset
    28
9c5183cb9bca bundle2: very first version of a bundle2 bundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
diff changeset
    29
stream level parameters
9c5183cb9bca bundle2: very first version of a bundle2 bundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
diff changeset
    30
------------------------
9c5183cb9bca bundle2: very first version of a bundle2 bundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
diff changeset
    31
9c5183cb9bca bundle2: very first version of a bundle2 bundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
diff changeset
    32
Binary format is as follow
9c5183cb9bca bundle2: very first version of a bundle2 bundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
diff changeset
    33
23009
90f86ad3d4ff bundle2: change header size and make them signed (new format)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23008
diff changeset
    34
:params size: int32
20801
9c5183cb9bca bundle2: very first version of a bundle2 bundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
diff changeset
    35
9c5183cb9bca bundle2: very first version of a bundle2 bundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
diff changeset
    36
  The total number of Bytes used by the parameters
9c5183cb9bca bundle2: very first version of a bundle2 bundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
diff changeset
    37
9c5183cb9bca bundle2: very first version of a bundle2 bundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
diff changeset
    38
:params value: arbitrary number of Bytes
9c5183cb9bca bundle2: very first version of a bundle2 bundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
diff changeset
    39
9c5183cb9bca bundle2: very first version of a bundle2 bundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
diff changeset
    40
  A blob of `params size` containing the serialized version of all stream level
9c5183cb9bca bundle2: very first version of a bundle2 bundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
diff changeset
    41
  parameters.
9c5183cb9bca bundle2: very first version of a bundle2 bundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
diff changeset
    42
21024
7731a2281cf0 spelling: fixes from spell checker
Mads Kiilerich <madski@unity3d.com>
parents: 21020
diff changeset
    43
  The blob contains a space separated list of parameters. Parameters with value
20811
9785c3f8f598 bundle2: urlquote stream parameter name and value
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20810
diff changeset
    44
  are stored in the form `<name>=<value>`. Both name and value are urlquoted.
20804
db9d3991d2c6 bundle2: support bundling simple parameter
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20803
diff changeset
    45
20813
8c74b3ce5b70 bundle2: refuse empty parameter name
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20812
diff changeset
    46
  Empty name are obviously forbidden.
8c74b3ce5b70 bundle2: refuse empty parameter name
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20812
diff changeset
    47
20844
2631204d7305 bundle2: implement the mandatory/advisory logic for parameter
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20843
diff changeset
    48
  Name MUST start with a letter. If this first letter is lower case, the
21024
7731a2281cf0 spelling: fixes from spell checker
Mads Kiilerich <madski@unity3d.com>
parents: 21020
diff changeset
    49
  parameter is advisory and can be safely ignored. However when the first
20844
2631204d7305 bundle2: implement the mandatory/advisory logic for parameter
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20843
diff changeset
    50
  letter is capital, the parameter is mandatory and the bundling process MUST
2631204d7305 bundle2: implement the mandatory/advisory logic for parameter
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20843
diff changeset
    51
  stop if he is not able to proceed it.
20814
8532f5e1b9df bundle2: force the first char of parameter to be an letter.
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20813
diff changeset
    52
20808
4c9130c7a29f bundle2: clarify stream parameter design in the documentation
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20805
diff changeset
    53
  Stream parameters use a simple textual format for two main reasons:
20804
db9d3991d2c6 bundle2: support bundling simple parameter
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20803
diff changeset
    54
21024
7731a2281cf0 spelling: fixes from spell checker
Mads Kiilerich <madski@unity3d.com>
parents: 21020
diff changeset
    55
  - Stream level parameters should remain simple and we want to discourage any
20808
4c9130c7a29f bundle2: clarify stream parameter design in the documentation
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20805
diff changeset
    56
    crazy usage.
21024
7731a2281cf0 spelling: fixes from spell checker
Mads Kiilerich <madski@unity3d.com>
parents: 21020
diff changeset
    57
  - Textual data allow easy human inspection of a bundle2 header in case of
20808
4c9130c7a29f bundle2: clarify stream parameter design in the documentation
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20805
diff changeset
    58
    troubles.
4c9130c7a29f bundle2: clarify stream parameter design in the documentation
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20805
diff changeset
    59
4c9130c7a29f bundle2: clarify stream parameter design in the documentation
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20805
diff changeset
    60
  Any Applicative level options MUST go into a bundle2 part instead.
20801
9c5183cb9bca bundle2: very first version of a bundle2 bundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
diff changeset
    61
9c5183cb9bca bundle2: very first version of a bundle2 bundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
diff changeset
    62
Payload part
9c5183cb9bca bundle2: very first version of a bundle2 bundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
diff changeset
    63
------------------------
9c5183cb9bca bundle2: very first version of a bundle2 bundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
diff changeset
    64
9c5183cb9bca bundle2: very first version of a bundle2 bundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
diff changeset
    65
Binary format is as follow
9c5183cb9bca bundle2: very first version of a bundle2 bundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
diff changeset
    66
23009
90f86ad3d4ff bundle2: change header size and make them signed (new format)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23008
diff changeset
    67
:header size: int32
20801
9c5183cb9bca bundle2: very first version of a bundle2 bundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
diff changeset
    68
25507
5bee483705e8 bundle2: clarify in docstring that header size is for a single header
Martin von Zweigbergk <martinvonz@google.com>
parents: 25506
diff changeset
    69
  The total number of Bytes used by the part header. When the header is empty
20801
9c5183cb9bca bundle2: very first version of a bundle2 bundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
diff changeset
    70
  (size = 0) this is interpreted as the end of stream marker.
9c5183cb9bca bundle2: very first version of a bundle2 bundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
diff changeset
    71
20856
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20844
diff changeset
    72
:header:
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20844
diff changeset
    73
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20844
diff changeset
    74
    The header defines how to interpret the part. It contains two piece of
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20844
diff changeset
    75
    data: the part type, and the part parameters.
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20844
diff changeset
    76
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20844
diff changeset
    77
    The part type is used to route an application level handler, that can
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20844
diff changeset
    78
    interpret payload.
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20844
diff changeset
    79
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20844
diff changeset
    80
    Part parameters are passed to the application level handler.  They are
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20844
diff changeset
    81
    meant to convey information that will help the application level object to
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20844
diff changeset
    82
    interpret the part payload.
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20844
diff changeset
    83
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20844
diff changeset
    84
    The binary format of the header is has follow
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20844
diff changeset
    85
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20844
diff changeset
    86
    :typesize: (one byte)
20877
9e9e3a4e9261 bundle2: part params
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20876
diff changeset
    87
23916
a3f7c781786b bundle2: fix parttype enforcement
Matt Mackall <mpm@selenic.com>
parents: 23868
diff changeset
    88
    :parttype: alphanumerical part name (restricted to [a-zA-Z0-9_:-]*)
20877
9e9e3a4e9261 bundle2: part params
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20876
diff changeset
    89
20995
e995d104c87f bundle2: add an integer id to part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20952
diff changeset
    90
    :partid: A 32bits integer (unique in the bundle) that can be used to refer
e995d104c87f bundle2: add an integer id to part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20952
diff changeset
    91
             to this part.
e995d104c87f bundle2: add an integer id to part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20952
diff changeset
    92
20877
9e9e3a4e9261 bundle2: part params
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20876
diff changeset
    93
    :parameters:
9e9e3a4e9261 bundle2: part params
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20876
diff changeset
    94
21024
7731a2281cf0 spelling: fixes from spell checker
Mads Kiilerich <madski@unity3d.com>
parents: 21020
diff changeset
    95
        Part's parameter may have arbitrary content, the binary structure is::
20877
9e9e3a4e9261 bundle2: part params
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20876
diff changeset
    96
9e9e3a4e9261 bundle2: part params
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20876
diff changeset
    97
            <mandatory-count><advisory-count><param-sizes><param-data>
9e9e3a4e9261 bundle2: part params
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20876
diff changeset
    98
9e9e3a4e9261 bundle2: part params
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20876
diff changeset
    99
        :mandatory-count: 1 byte, number of mandatory parameters
9e9e3a4e9261 bundle2: part params
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20876
diff changeset
   100
9e9e3a4e9261 bundle2: part params
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20876
diff changeset
   101
        :advisory-count:  1 byte, number of advisory parameters
9e9e3a4e9261 bundle2: part params
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20876
diff changeset
   102
9e9e3a4e9261 bundle2: part params
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20876
diff changeset
   103
        :param-sizes:
9e9e3a4e9261 bundle2: part params
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20876
diff changeset
   104
9e9e3a4e9261 bundle2: part params
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20876
diff changeset
   105
            N couple of bytes, where N is the total number of parameters. Each
9e9e3a4e9261 bundle2: part params
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20876
diff changeset
   106
            couple contains (<size-of-key>, <size-of-value) for one parameter.
9e9e3a4e9261 bundle2: part params
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20876
diff changeset
   107
9e9e3a4e9261 bundle2: part params
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20876
diff changeset
   108
        :param-data:
9e9e3a4e9261 bundle2: part params
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20876
diff changeset
   109
9e9e3a4e9261 bundle2: part params
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20876
diff changeset
   110
            A blob of bytes from which each parameter key and value can be
9e9e3a4e9261 bundle2: part params
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20876
diff changeset
   111
            retrieved using the list of size couples stored in the previous
9e9e3a4e9261 bundle2: part params
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20876
diff changeset
   112
            field.
9e9e3a4e9261 bundle2: part params
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20876
diff changeset
   113
9e9e3a4e9261 bundle2: part params
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20876
diff changeset
   114
            Mandatory parameters comes first, then the advisory ones.
20856
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20844
diff changeset
   115
21607
054fa5176fa7 bundle2: forbid duplicate parameter keys
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21606
diff changeset
   116
            Each parameter's key MUST be unique within the part.
054fa5176fa7 bundle2: forbid duplicate parameter keys
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21606
diff changeset
   117
20856
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20844
diff changeset
   118
:payload:
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20844
diff changeset
   119
20876
ddd56f3eb786 bundle2: support for bundling and unbundling payload
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20864
diff changeset
   120
    payload is a series of `<chunksize><chunkdata>`.
ddd56f3eb786 bundle2: support for bundling and unbundling payload
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20864
diff changeset
   121
23009
90f86ad3d4ff bundle2: change header size and make them signed (new format)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23008
diff changeset
   122
    `chunksize` is an int32, `chunkdata` are plain bytes (as much as
20876
ddd56f3eb786 bundle2: support for bundling and unbundling payload
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20864
diff changeset
   123
    `chunksize` says)` The payload part is concluded by a zero size chunk.
ddd56f3eb786 bundle2: support for bundling and unbundling payload
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20864
diff changeset
   124
ddd56f3eb786 bundle2: support for bundling and unbundling payload
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20864
diff changeset
   125
    The current implementation always produces either zero or one chunk.
21024
7731a2281cf0 spelling: fixes from spell checker
Mads Kiilerich <madski@unity3d.com>
parents: 21020
diff changeset
   126
    This is an implementation limitation that will ultimately be lifted.
20891
1c6cd23fc221 bundle2: add some distinction between mandatory and advisory part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20890
diff changeset
   127
23009
90f86ad3d4ff bundle2: change header size and make them signed (new format)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23008
diff changeset
   128
    `chunksize` can be negative to trigger special case processing. No such
90f86ad3d4ff bundle2: change header size and make them signed (new format)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23008
diff changeset
   129
    processing is in place yet.
90f86ad3d4ff bundle2: change header size and make them signed (new format)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23008
diff changeset
   130
20891
1c6cd23fc221 bundle2: add some distinction between mandatory and advisory part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20890
diff changeset
   131
Bundle processing
1c6cd23fc221 bundle2: add some distinction between mandatory and advisory part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20890
diff changeset
   132
============================
1c6cd23fc221 bundle2: add some distinction between mandatory and advisory part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20890
diff changeset
   133
1c6cd23fc221 bundle2: add some distinction between mandatory and advisory part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20890
diff changeset
   134
Each part is processed in order using a "part handler". Handler are registered
1c6cd23fc221 bundle2: add some distinction between mandatory and advisory part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20890
diff changeset
   135
for a certain part type.
1c6cd23fc221 bundle2: add some distinction between mandatory and advisory part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20890
diff changeset
   136
1c6cd23fc221 bundle2: add some distinction between mandatory and advisory part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20890
diff changeset
   137
The matching of a part to its handler is case insensitive. The case of the
1c6cd23fc221 bundle2: add some distinction between mandatory and advisory part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20890
diff changeset
   138
part type is used to know if a part is mandatory or advisory. If the Part type
1c6cd23fc221 bundle2: add some distinction between mandatory and advisory part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20890
diff changeset
   139
contains any uppercase char it is considered mandatory. When no handler is
1c6cd23fc221 bundle2: add some distinction between mandatory and advisory part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20890
diff changeset
   140
known for a Mandatory part, the process is aborted and an exception is raised.
20892
6fe95448596d bundle2: read the whole bundle from stream on abort
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20891
diff changeset
   141
If the part is advisory and no handler is known, the part is ignored. When the
6fe95448596d bundle2: read the whole bundle from stream on abort
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20891
diff changeset
   142
process is aborted, the full bundle is still read from the stream to keep the
6fe95448596d bundle2: read the whole bundle from stream on abort
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20891
diff changeset
   143
channel usable. But none of the part read from an abort are processed. In the
6fe95448596d bundle2: read the whole bundle from stream on abort
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20891
diff changeset
   144
future, dropping the stream may become an option for channel we do not care to
6fe95448596d bundle2: read the whole bundle from stream on abort
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20891
diff changeset
   145
preserve.
20801
9c5183cb9bca bundle2: very first version of a bundle2 bundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
diff changeset
   146
"""
9c5183cb9bca bundle2: very first version of a bundle2 bundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
diff changeset
   147
33698
da7c285ec6da bundle2: use modern Python division
Augie Fackler <augie@google.com>
parents: 33693
diff changeset
   148
from __future__ import absolute_import, division
25919
8221fefaea08 bundle2: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25660
diff changeset
   149
36970
79b73be4dda5 rev-branch-cache: add a function to generate a part
Boris Feld <boris.feld@octobus.net>
parents: 36969
diff changeset
   150
import collections
24026
3daef83a1873 bundle2.unpackermixin: control for underlying file descriptor
Eric Sumner <ericsumner@fb.com>
parents: 23916
diff changeset
   151
import errno
35069
241d9caca11e bundle2: use os.SEEK_* constants
Gregory Szorc <gregory.szorc@gmail.com>
parents: 34820
diff changeset
   152
import os
25919
8221fefaea08 bundle2: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25660
diff changeset
   153
import re
20814
8532f5e1b9df bundle2: force the first char of parameter to be an letter.
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20813
diff changeset
   154
import string
25919
8221fefaea08 bundle2: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25660
diff changeset
   155
import struct
8221fefaea08 bundle2: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25660
diff changeset
   156
import sys
20804
db9d3991d2c6 bundle2: support bundling simple parameter
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20803
diff changeset
   157
25919
8221fefaea08 bundle2: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25660
diff changeset
   158
from .i18n import _
8221fefaea08 bundle2: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25660
diff changeset
   159
from . import (
35267
dbf868623daf bookmark: add a 'check:bookmarks' bundle2 part
Boris Feld <boris.feld@octobus.net>
parents: 35141
diff changeset
   160
    bookmarks,
25919
8221fefaea08 bundle2: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25660
diff changeset
   161
    changegroup,
36969
9988fc10f49e revbranchcache: add a bundle2 handler for a rbc part
Boris Feld <boris.feld@octobus.net>
parents: 36458
diff changeset
   162
    encoding,
25919
8221fefaea08 bundle2: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25660
diff changeset
   163
    error,
34820
a95067b1dca6 phase: introduce a new 'check:phases' part
Boris Feld <boris.feld@octobus.net>
parents: 34637
diff changeset
   164
    node as nodemod,
25919
8221fefaea08 bundle2: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25660
diff changeset
   165
    obsolete,
33043
e8c8d81eb864 bundle: add config option to include phases
Martin von Zweigbergk <martinvonz@google.com>
parents: 33042
diff changeset
   166
    phases,
25919
8221fefaea08 bundle2: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25660
diff changeset
   167
    pushkey,
30031
0f6d6fdd3c2a pycompat: provide 'ispy3' constant
Yuya Nishihara <yuya@tcha.org>
parents: 29941
diff changeset
   168
    pycompat,
35758
b996ddf5963d bundle2: add a 'stream' part handler for stream cloning
Boris Feld <boris.feld@octobus.net>
parents: 35655
diff changeset
   169
    streamclone,
25919
8221fefaea08 bundle2: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25660
diff changeset
   170
    tags,
8221fefaea08 bundle2: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25660
diff changeset
   171
    url,
8221fefaea08 bundle2: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25660
diff changeset
   172
    util,
8221fefaea08 bundle2: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25660
diff changeset
   173
)
20802
520df53ad26a bundle2: a very first version of bundle2 unbundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20801
diff changeset
   174
28883
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents: 28672
diff changeset
   175
urlerr = util.urlerr
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents: 28672
diff changeset
   176
urlreq = util.urlreq
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents: 28672
diff changeset
   177
20804
db9d3991d2c6 bundle2: support bundling simple parameter
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20803
diff changeset
   178
_pack = struct.pack
db9d3991d2c6 bundle2: support bundling simple parameter
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20803
diff changeset
   179
_unpack = struct.unpack
db9d3991d2c6 bundle2: support bundling simple parameter
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20803
diff changeset
   180
23009
90f86ad3d4ff bundle2: change header size and make them signed (new format)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23008
diff changeset
   181
_fstreamparamsize = '>i'
90f86ad3d4ff bundle2: change header size and make them signed (new format)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23008
diff changeset
   182
_fpartheadersize = '>i'
20856
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20844
diff changeset
   183
_fparttypesize = '>B'
20995
e995d104c87f bundle2: add an integer id to part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20952
diff changeset
   184
_fpartid = '>I'
23009
90f86ad3d4ff bundle2: change header size and make them signed (new format)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23008
diff changeset
   185
_fpayloadsize = '>i'
20877
9e9e3a4e9261 bundle2: part params
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20876
diff changeset
   186
_fpartparamcount = '>BB'
9e9e3a4e9261 bundle2: part params
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20876
diff changeset
   187
35793
4fb2bb61597c bundle2: increase payload part chunk size to 32kb
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35792
diff changeset
   188
preferedchunksize = 32768
21001
c93bb6a08fa1 bundle2: support chunk iterator as part data
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21000
diff changeset
   189
23868
405eacbecc52 bundle2: enforce parttype as alphanumerical
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23590
diff changeset
   190
_parttypeforbidden = re.compile('[^a-zA-Z0-9_:-]')
405eacbecc52 bundle2: enforce parttype as alphanumerical
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23590
diff changeset
   191
25313
8f2c362bcb58 bundle2: introduce a specific function for bundling debug message
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25181
diff changeset
   192
def outdebug(ui, message):
8f2c362bcb58 bundle2: introduce a specific function for bundling debug message
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25181
diff changeset
   193
    """debug regarding output stream (bundling)"""
33165
0224820688ac config: register the 'devel.bundle2.debug' config
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 33135
diff changeset
   194
    if ui.configbool('devel', 'bundle2.debug'):
25336
7c60a42265fb bundle2: hide bundle2 stream debug under a config flag
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25335
diff changeset
   195
        ui.debug('bundle2-output: %s\n' % message)
25313
8f2c362bcb58 bundle2: introduce a specific function for bundling debug message
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25181
diff changeset
   196
25318
64dd8f25fcf4 bundle2: introduce a specific function for debug messages while unbundling
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25317
diff changeset
   197
def indebug(ui, message):
64dd8f25fcf4 bundle2: introduce a specific function for debug messages while unbundling
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25317
diff changeset
   198
    """debug on input stream (unbundling)"""
33165
0224820688ac config: register the 'devel.bundle2.debug' config
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 33135
diff changeset
   199
    if ui.configbool('devel', 'bundle2.debug'):
25336
7c60a42265fb bundle2: hide bundle2 stream debug under a config flag
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25335
diff changeset
   200
        ui.debug('bundle2-input: %s\n' % message)
25318
64dd8f25fcf4 bundle2: introduce a specific function for debug messages while unbundling
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25317
diff changeset
   201
23868
405eacbecc52 bundle2: enforce parttype as alphanumerical
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23590
diff changeset
   202
def validateparttype(parttype):
405eacbecc52 bundle2: enforce parttype as alphanumerical
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23590
diff changeset
   203
    """raise ValueError if a parttype contains invalid character"""
23916
a3f7c781786b bundle2: fix parttype enforcement
Matt Mackall <mpm@selenic.com>
parents: 23868
diff changeset
   204
    if _parttypeforbidden.search(parttype):
23868
405eacbecc52 bundle2: enforce parttype as alphanumerical
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23590
diff changeset
   205
        raise ValueError(parttype)
405eacbecc52 bundle2: enforce parttype as alphanumerical
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23590
diff changeset
   206
20877
9e9e3a4e9261 bundle2: part params
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20876
diff changeset
   207
def _makefpartparamsizes(nbparams):
9e9e3a4e9261 bundle2: part params
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20876
diff changeset
   208
    """return a struct format to read part parameter sizes
9e9e3a4e9261 bundle2: part params
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20876
diff changeset
   209
9e9e3a4e9261 bundle2: part params
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20876
diff changeset
   210
    The number parameters is variable so we need to build that format
9e9e3a4e9261 bundle2: part params
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20876
diff changeset
   211
    dynamically.
9e9e3a4e9261 bundle2: part params
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20876
diff changeset
   212
    """
9e9e3a4e9261 bundle2: part params
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20876
diff changeset
   213
    return '>'+('BB'*nbparams)
20804
db9d3991d2c6 bundle2: support bundling simple parameter
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20803
diff changeset
   214
20890
ec7fc110faee bundle2: introduce a `parthandler` decorator
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20889
diff changeset
   215
parthandlermapping = {}
20889
deed5edb72de bundle2: first version of a bundle processing
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20887
diff changeset
   216
21623
5b26d82e4e2a bundle2: make it possible to declare params handled by a part handler
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21622
diff changeset
   217
def parthandler(parttype, params=()):
20890
ec7fc110faee bundle2: introduce a `parthandler` decorator
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20889
diff changeset
   218
    """decorator that register a function as a bundle2 part handler
ec7fc110faee bundle2: introduce a `parthandler` decorator
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20889
diff changeset
   219
ec7fc110faee bundle2: introduce a `parthandler` decorator
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20889
diff changeset
   220
    eg::
ec7fc110faee bundle2: introduce a `parthandler` decorator
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20889
diff changeset
   221
21624
d61066d787c8 bundle2: declare supported parameters for all handlers
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21623
diff changeset
   222
        @parthandler('myparttype', ('mandatory', 'param', 'handled'))
20890
ec7fc110faee bundle2: introduce a `parthandler` decorator
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20889
diff changeset
   223
        def myparttypehandler(...):
ec7fc110faee bundle2: introduce a `parthandler` decorator
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20889
diff changeset
   224
            '''process a part of type "my part".'''
ec7fc110faee bundle2: introduce a `parthandler` decorator
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20889
diff changeset
   225
            ...
ec7fc110faee bundle2: introduce a `parthandler` decorator
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20889
diff changeset
   226
    """
23868
405eacbecc52 bundle2: enforce parttype as alphanumerical
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23590
diff changeset
   227
    validateparttype(parttype)
20890
ec7fc110faee bundle2: introduce a `parthandler` decorator
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20889
diff changeset
   228
    def _decorator(func):
20891
1c6cd23fc221 bundle2: add some distinction between mandatory and advisory part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20890
diff changeset
   229
        lparttype = parttype.lower() # enforce lower case matching.
1c6cd23fc221 bundle2: add some distinction between mandatory and advisory part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20890
diff changeset
   230
        assert lparttype not in parthandlermapping
1c6cd23fc221 bundle2: add some distinction between mandatory and advisory part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20890
diff changeset
   231
        parthandlermapping[lparttype] = func
21623
5b26d82e4e2a bundle2: make it possible to declare params handled by a part handler
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21622
diff changeset
   232
        func.params = frozenset(params)
20890
ec7fc110faee bundle2: introduce a `parthandler` decorator
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20889
diff changeset
   233
        return func
ec7fc110faee bundle2: introduce a `parthandler` decorator
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20889
diff changeset
   234
    return _decorator
20889
deed5edb72de bundle2: first version of a bundle processing
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20887
diff changeset
   235
20949
571f2903ff1e bundle2: record processing results in the bundleoperation object
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20948
diff changeset
   236
class unbundlerecords(object):
571f2903ff1e bundle2: record processing results in the bundleoperation object
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20948
diff changeset
   237
    """keep record of what happens during and unbundle
571f2903ff1e bundle2: record processing results in the bundleoperation object
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20948
diff changeset
   238
571f2903ff1e bundle2: record processing results in the bundleoperation object
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20948
diff changeset
   239
    New records are added using `records.add('cat', obj)`. Where 'cat' is a
21024
7731a2281cf0 spelling: fixes from spell checker
Mads Kiilerich <madski@unity3d.com>
parents: 21020
diff changeset
   240
    category of record and obj is an arbitrary object.
20949
571f2903ff1e bundle2: record processing results in the bundleoperation object
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20948
diff changeset
   241
571f2903ff1e bundle2: record processing results in the bundleoperation object
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20948
diff changeset
   242
    `records['cat']` will return all entries of this category 'cat'.
571f2903ff1e bundle2: record processing results in the bundleoperation object
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20948
diff changeset
   243
571f2903ff1e bundle2: record processing results in the bundleoperation object
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20948
diff changeset
   244
    Iterating on the object itself will yield `('category', obj)` tuples
571f2903ff1e bundle2: record processing results in the bundleoperation object
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20948
diff changeset
   245
    for all entries.
571f2903ff1e bundle2: record processing results in the bundleoperation object
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20948
diff changeset
   246
571f2903ff1e bundle2: record processing results in the bundleoperation object
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20948
diff changeset
   247
    All iterations happens in chronological order.
571f2903ff1e bundle2: record processing results in the bundleoperation object
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20948
diff changeset
   248
    """
571f2903ff1e bundle2: record processing results in the bundleoperation object
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20948
diff changeset
   249
571f2903ff1e bundle2: record processing results in the bundleoperation object
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20948
diff changeset
   250
    def __init__(self):
571f2903ff1e bundle2: record processing results in the bundleoperation object
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20948
diff changeset
   251
        self._categories = {}
571f2903ff1e bundle2: record processing results in the bundleoperation object
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20948
diff changeset
   252
        self._sequences = []
20996
ed3c5e18a047 bundle2: add reply awareness to unbundlerecords
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20995
diff changeset
   253
        self._replies = {}
20949
571f2903ff1e bundle2: record processing results in the bundleoperation object
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20948
diff changeset
   254
20996
ed3c5e18a047 bundle2: add reply awareness to unbundlerecords
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20995
diff changeset
   255
    def add(self, category, entry, inreplyto=None):
20949
571f2903ff1e bundle2: record processing results in the bundleoperation object
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20948
diff changeset
   256
        """add a new record of a given category.
571f2903ff1e bundle2: record processing results in the bundleoperation object
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20948
diff changeset
   257
571f2903ff1e bundle2: record processing results in the bundleoperation object
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20948
diff changeset
   258
        The entry can then be retrieved in the list returned by
571f2903ff1e bundle2: record processing results in the bundleoperation object
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20948
diff changeset
   259
        self['category']."""
571f2903ff1e bundle2: record processing results in the bundleoperation object
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20948
diff changeset
   260
        self._categories.setdefault(category, []).append(entry)
571f2903ff1e bundle2: record processing results in the bundleoperation object
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20948
diff changeset
   261
        self._sequences.append((category, entry))
20996
ed3c5e18a047 bundle2: add reply awareness to unbundlerecords
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20995
diff changeset
   262
        if inreplyto is not None:
ed3c5e18a047 bundle2: add reply awareness to unbundlerecords
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20995
diff changeset
   263
            self.getreplies(inreplyto).add(category, entry)
ed3c5e18a047 bundle2: add reply awareness to unbundlerecords
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20995
diff changeset
   264
ed3c5e18a047 bundle2: add reply awareness to unbundlerecords
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20995
diff changeset
   265
    def getreplies(self, partid):
23139
e53f6b72a0e4 spelling: fixes from proofreading of spell checker issues
Mads Kiilerich <madski@unity3d.com>
parents: 23067
diff changeset
   266
        """get the records that are replies to a specific part"""
20996
ed3c5e18a047 bundle2: add reply awareness to unbundlerecords
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20995
diff changeset
   267
        return self._replies.setdefault(partid, unbundlerecords())
20949
571f2903ff1e bundle2: record processing results in the bundleoperation object
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20948
diff changeset
   268
571f2903ff1e bundle2: record processing results in the bundleoperation object
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20948
diff changeset
   269
    def __getitem__(self, cat):
571f2903ff1e bundle2: record processing results in the bundleoperation object
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20948
diff changeset
   270
        return tuple(self._categories.get(cat, ()))
571f2903ff1e bundle2: record processing results in the bundleoperation object
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20948
diff changeset
   271
571f2903ff1e bundle2: record processing results in the bundleoperation object
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20948
diff changeset
   272
    def __iter__(self):
571f2903ff1e bundle2: record processing results in the bundleoperation object
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20948
diff changeset
   273
        return iter(self._sequences)
571f2903ff1e bundle2: record processing results in the bundleoperation object
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20948
diff changeset
   274
571f2903ff1e bundle2: record processing results in the bundleoperation object
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20948
diff changeset
   275
    def __len__(self):
571f2903ff1e bundle2: record processing results in the bundleoperation object
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20948
diff changeset
   276
        return len(self._sequences)
571f2903ff1e bundle2: record processing results in the bundleoperation object
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20948
diff changeset
   277
571f2903ff1e bundle2: record processing results in the bundleoperation object
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20948
diff changeset
   278
    def __nonzero__(self):
571f2903ff1e bundle2: record processing results in the bundleoperation object
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20948
diff changeset
   279
        return bool(self._sequences)
571f2903ff1e bundle2: record processing results in the bundleoperation object
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20948
diff changeset
   280
31483
413b44003462 py3: add __bool__ to every class defining __nonzero__
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30985
diff changeset
   281
    __bool__ = __nonzero__
413b44003462 py3: add __bool__ to every class defining __nonzero__
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30985
diff changeset
   282
20948
329cd74b52bd bundle2: introduce a bundleoperation object
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20947
diff changeset
   283
class bundleoperation(object):
329cd74b52bd bundle2: introduce a bundleoperation object
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20947
diff changeset
   284
    """an object that represents a single bundling process
329cd74b52bd bundle2: introduce a bundleoperation object
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20947
diff changeset
   285
329cd74b52bd bundle2: introduce a bundleoperation object
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20947
diff changeset
   286
    Its purpose is to carry unbundle-related objects and states.
329cd74b52bd bundle2: introduce a bundleoperation object
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20947
diff changeset
   287
329cd74b52bd bundle2: introduce a bundleoperation object
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20947
diff changeset
   288
    A new object should be created at the beginning of each bundle processing.
329cd74b52bd bundle2: introduce a bundleoperation object
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20947
diff changeset
   289
    The object is to be returned by the processing function.
329cd74b52bd bundle2: introduce a bundleoperation object
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20947
diff changeset
   290
329cd74b52bd bundle2: introduce a bundleoperation object
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20947
diff changeset
   291
    The object has very little content now it will ultimately contain:
329cd74b52bd bundle2: introduce a bundleoperation object
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20947
diff changeset
   292
    * an access to the repo the bundle is applied to,
329cd74b52bd bundle2: introduce a bundleoperation object
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20947
diff changeset
   293
    * a ui object,
329cd74b52bd bundle2: introduce a bundleoperation object
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20947
diff changeset
   294
    * a way to retrieve a transaction to add changes to the repo,
329cd74b52bd bundle2: introduce a bundleoperation object
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20947
diff changeset
   295
    * a way to record the result of processing each part,
329cd74b52bd bundle2: introduce a bundleoperation object
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20947
diff changeset
   296
    * a way to construct a bundle response when applicable.
329cd74b52bd bundle2: introduce a bundleoperation object
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20947
diff changeset
   297
    """
329cd74b52bd bundle2: introduce a bundleoperation object
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20947
diff changeset
   298
24878
e530cde6d115 bundle2: disable ouput capture unless we use http (issue4613 issue4615)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24851
diff changeset
   299
    def __init__(self, repo, transactiongetter, captureoutput=True):
20948
329cd74b52bd bundle2: introduce a bundleoperation object
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20947
diff changeset
   300
        self.repo = repo
329cd74b52bd bundle2: introduce a bundleoperation object
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20947
diff changeset
   301
        self.ui = repo.ui
20949
571f2903ff1e bundle2: record processing results in the bundleoperation object
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20948
diff changeset
   302
        self.records = unbundlerecords()
20997
d7df4b7378ae bundle2: produce a bundle2 reply
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20996
diff changeset
   303
        self.reply = None
24878
e530cde6d115 bundle2: disable ouput capture unless we use http (issue4613 issue4615)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24851
diff changeset
   304
        self.captureoutput = captureoutput
33692
f3407d56a6e8 bundle2: add the capability to store hookargs on bundle operation object
Pulkit Goyal <7895pulkit@gmail.com>
parents: 33541
diff changeset
   305
        self.hookargs = {}
33693
5fc4ddfbe626 bundle2: load hookargs from bundleoperation into transaction when started one
Pulkit Goyal <7895pulkit@gmail.com>
parents: 33692
diff changeset
   306
        self._gettransaction = transactiongetter
35274
1f30cbac34b6 bundle2: add a 'modes' dictionary to the bundle operation
Boris Feld <boris.feld@octobus.net>
parents: 35270
diff changeset
   307
        # carries value that can modify part behavior
1f30cbac34b6 bundle2: add a 'modes' dictionary to the bundle operation
Boris Feld <boris.feld@octobus.net>
parents: 35270
diff changeset
   308
        self.modes = {}
33693
5fc4ddfbe626 bundle2: load hookargs from bundleoperation into transaction when started one
Pulkit Goyal <7895pulkit@gmail.com>
parents: 33692
diff changeset
   309
5fc4ddfbe626 bundle2: load hookargs from bundleoperation into transaction when started one
Pulkit Goyal <7895pulkit@gmail.com>
parents: 33692
diff changeset
   310
    def gettransaction(self):
5fc4ddfbe626 bundle2: load hookargs from bundleoperation into transaction when started one
Pulkit Goyal <7895pulkit@gmail.com>
parents: 33692
diff changeset
   311
        transaction = self._gettransaction()
5fc4ddfbe626 bundle2: load hookargs from bundleoperation into transaction when started one
Pulkit Goyal <7895pulkit@gmail.com>
parents: 33692
diff changeset
   312
33831
b3a41f4d837b bundle2: relax the condition to update transaction.hookargs
Yuya Nishihara <yuya@tcha.org>
parents: 33830
diff changeset
   313
        if self.hookargs:
33693
5fc4ddfbe626 bundle2: load hookargs from bundleoperation into transaction when started one
Pulkit Goyal <7895pulkit@gmail.com>
parents: 33692
diff changeset
   314
            # the ones added to the transaction supercede those added
5fc4ddfbe626 bundle2: load hookargs from bundleoperation into transaction when started one
Pulkit Goyal <7895pulkit@gmail.com>
parents: 33692
diff changeset
   315
            # to the operation.
5fc4ddfbe626 bundle2: load hookargs from bundleoperation into transaction when started one
Pulkit Goyal <7895pulkit@gmail.com>
parents: 33692
diff changeset
   316
            self.hookargs.update(transaction.hookargs)
5fc4ddfbe626 bundle2: load hookargs from bundleoperation into transaction when started one
Pulkit Goyal <7895pulkit@gmail.com>
parents: 33692
diff changeset
   317
            transaction.hookargs = self.hookargs
5fc4ddfbe626 bundle2: load hookargs from bundleoperation into transaction when started one
Pulkit Goyal <7895pulkit@gmail.com>
parents: 33692
diff changeset
   318
33831
b3a41f4d837b bundle2: relax the condition to update transaction.hookargs
Yuya Nishihara <yuya@tcha.org>
parents: 33830
diff changeset
   319
        # mark the hookargs as flushed.  further attempts to add to
b3a41f4d837b bundle2: relax the condition to update transaction.hookargs
Yuya Nishihara <yuya@tcha.org>
parents: 33830
diff changeset
   320
        # hookargs will result in an abort.
b3a41f4d837b bundle2: relax the condition to update transaction.hookargs
Yuya Nishihara <yuya@tcha.org>
parents: 33830
diff changeset
   321
        self.hookargs = None
33693
5fc4ddfbe626 bundle2: load hookargs from bundleoperation into transaction when started one
Pulkit Goyal <7895pulkit@gmail.com>
parents: 33692
diff changeset
   322
5fc4ddfbe626 bundle2: load hookargs from bundleoperation into transaction when started one
Pulkit Goyal <7895pulkit@gmail.com>
parents: 33692
diff changeset
   323
        return transaction
33692
f3407d56a6e8 bundle2: add the capability to store hookargs on bundle operation object
Pulkit Goyal <7895pulkit@gmail.com>
parents: 33541
diff changeset
   324
f3407d56a6e8 bundle2: add the capability to store hookargs on bundle operation object
Pulkit Goyal <7895pulkit@gmail.com>
parents: 33541
diff changeset
   325
    def addhookargs(self, hookargs):
33693
5fc4ddfbe626 bundle2: load hookargs from bundleoperation into transaction when started one
Pulkit Goyal <7895pulkit@gmail.com>
parents: 33692
diff changeset
   326
        if self.hookargs is None:
33830
1bf5c5507614 bundle2: raise ProgrammingError for invalid call of addhookargs()
Yuya Nishihara <yuya@tcha.org>
parents: 33791
diff changeset
   327
            raise error.ProgrammingError('attempted to add hookargs to '
1bf5c5507614 bundle2: raise ProgrammingError for invalid call of addhookargs()
Yuya Nishihara <yuya@tcha.org>
parents: 33791
diff changeset
   328
                                         'operation after transaction started')
33692
f3407d56a6e8 bundle2: add the capability to store hookargs on bundle operation object
Pulkit Goyal <7895pulkit@gmail.com>
parents: 33541
diff changeset
   329
        self.hookargs.update(hookargs)
20948
329cd74b52bd bundle2: introduce a bundleoperation object
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20947
diff changeset
   330
20952
b24ee5076b94 bundle2: make it possible have a global transaction for the unbundling
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20950
diff changeset
   331
class TransactionUnavailable(RuntimeError):
b24ee5076b94 bundle2: make it possible have a global transaction for the unbundling
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20950
diff changeset
   332
    pass
b24ee5076b94 bundle2: make it possible have a global transaction for the unbundling
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20950
diff changeset
   333
b24ee5076b94 bundle2: make it possible have a global transaction for the unbundling
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20950
diff changeset
   334
def _notransaction():
b24ee5076b94 bundle2: make it possible have a global transaction for the unbundling
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20950
diff changeset
   335
    """default method to get a transaction while processing a bundle
b24ee5076b94 bundle2: make it possible have a global transaction for the unbundling
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20950
diff changeset
   336
b24ee5076b94 bundle2: make it possible have a global transaction for the unbundling
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20950
diff changeset
   337
    Raise an exception to highlight the fact that no transaction was expected
b24ee5076b94 bundle2: make it possible have a global transaction for the unbundling
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20950
diff changeset
   338
    to be created"""
b24ee5076b94 bundle2: make it possible have a global transaction for the unbundling
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20950
diff changeset
   339
    raise TransactionUnavailable()
b24ee5076b94 bundle2: make it possible have a global transaction for the unbundling
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20950
diff changeset
   340
33055
18c2489ac96d bundle: make applybundle() delegate v1 bundles to applybundle1()
Martin von Zweigbergk <martinvonz@google.com>
parents: 33052
diff changeset
   341
def applybundle(repo, unbundler, tr, source=None, url=None, **kwargs):
26790
28a6c2d72097 bundle2: introduce an "applybundle" function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26698
diff changeset
   342
    # transform me into unbundler.apply() as soon as the freeze is lifted
33055
18c2489ac96d bundle: make applybundle() delegate v1 bundles to applybundle1()
Martin von Zweigbergk <martinvonz@google.com>
parents: 33052
diff changeset
   343
    if isinstance(unbundler, unbundle20):
18c2489ac96d bundle: make applybundle() delegate v1 bundles to applybundle1()
Martin von Zweigbergk <martinvonz@google.com>
parents: 33052
diff changeset
   344
        tr.hookargs['bundle2'] = '1'
18c2489ac96d bundle: make applybundle() delegate v1 bundles to applybundle1()
Martin von Zweigbergk <martinvonz@google.com>
parents: 33052
diff changeset
   345
        if source is not None and 'source' not in tr.hookargs:
18c2489ac96d bundle: make applybundle() delegate v1 bundles to applybundle1()
Martin von Zweigbergk <martinvonz@google.com>
parents: 33052
diff changeset
   346
            tr.hookargs['source'] = source
18c2489ac96d bundle: make applybundle() delegate v1 bundles to applybundle1()
Martin von Zweigbergk <martinvonz@google.com>
parents: 33052
diff changeset
   347
        if url is not None and 'url' not in tr.hookargs:
18c2489ac96d bundle: make applybundle() delegate v1 bundles to applybundle1()
Martin von Zweigbergk <martinvonz@google.com>
parents: 33052
diff changeset
   348
            tr.hookargs['url'] = url
18c2489ac96d bundle: make applybundle() delegate v1 bundles to applybundle1()
Martin von Zweigbergk <martinvonz@google.com>
parents: 33052
diff changeset
   349
        return processbundle(repo, unbundler, lambda: tr)
18c2489ac96d bundle: make applybundle() delegate v1 bundles to applybundle1()
Martin von Zweigbergk <martinvonz@google.com>
parents: 33052
diff changeset
   350
    else:
33056
1d2b68952950 bundle: inline applybundle1()
Martin von Zweigbergk <martinvonz@google.com>
parents: 33055
diff changeset
   351
        # the transactiongetter won't be used, but we might as well set it
1d2b68952950 bundle: inline applybundle1()
Martin von Zweigbergk <martinvonz@google.com>
parents: 33055
diff changeset
   352
        op = bundleoperation(repo, lambda: tr)
1d2b68952950 bundle: inline applybundle1()
Martin von Zweigbergk <martinvonz@google.com>
parents: 33055
diff changeset
   353
        _processchangegroup(op, unbundler, tr, source, url, **kwargs)
1d2b68952950 bundle: inline applybundle1()
Martin von Zweigbergk <martinvonz@google.com>
parents: 33055
diff changeset
   354
        return op
26790
28a6c2d72097 bundle2: introduce an "applybundle" function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26698
diff changeset
   355
34162
e9e0e1143fc5 bundle2: move part iterator a separate class
Durham Goode <durham@fb.com>
parents: 34116
diff changeset
   356
class partiterator(object):
34164
21c2df59a1da bundle2: move exception handling into part iterator
Durham Goode <durham@fb.com>
parents: 34163
diff changeset
   357
    def __init__(self, repo, op, unbundler):
34163
550343626bb2 bundle2: move part counter to partiterator
Durham Goode <durham@fb.com>
parents: 34162
diff changeset
   358
        self.repo = repo
34164
21c2df59a1da bundle2: move exception handling into part iterator
Durham Goode <durham@fb.com>
parents: 34163
diff changeset
   359
        self.op = op
34162
e9e0e1143fc5 bundle2: move part iterator a separate class
Durham Goode <durham@fb.com>
parents: 34116
diff changeset
   360
        self.unbundler = unbundler
34163
550343626bb2 bundle2: move part counter to partiterator
Durham Goode <durham@fb.com>
parents: 34162
diff changeset
   361
        self.iterator = None
550343626bb2 bundle2: move part counter to partiterator
Durham Goode <durham@fb.com>
parents: 34162
diff changeset
   362
        self.count = 0
34265
e71890f27767 bundle2: move processpart stream maintenance into part iterator
Durham Goode <durham@fb.com>
parents: 34256
diff changeset
   363
        self.current = None
34162
e9e0e1143fc5 bundle2: move part iterator a separate class
Durham Goode <durham@fb.com>
parents: 34116
diff changeset
   364
e9e0e1143fc5 bundle2: move part iterator a separate class
Durham Goode <durham@fb.com>
parents: 34116
diff changeset
   365
    def __enter__(self):
34163
550343626bb2 bundle2: move part counter to partiterator
Durham Goode <durham@fb.com>
parents: 34162
diff changeset
   366
        def func():
550343626bb2 bundle2: move part counter to partiterator
Durham Goode <durham@fb.com>
parents: 34162
diff changeset
   367
            itr = enumerate(self.unbundler.iterparts())
550343626bb2 bundle2: move part counter to partiterator
Durham Goode <durham@fb.com>
parents: 34162
diff changeset
   368
            for count, p in itr:
550343626bb2 bundle2: move part counter to partiterator
Durham Goode <durham@fb.com>
parents: 34162
diff changeset
   369
                self.count = count
34265
e71890f27767 bundle2: move processpart stream maintenance into part iterator
Durham Goode <durham@fb.com>
parents: 34256
diff changeset
   370
                self.current = p
34163
550343626bb2 bundle2: move part counter to partiterator
Durham Goode <durham@fb.com>
parents: 34162
diff changeset
   371
                yield p
35135
db5038525718 bundle2: implement consume() API on unbundlepart
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35134
diff changeset
   372
                p.consume()
34265
e71890f27767 bundle2: move processpart stream maintenance into part iterator
Durham Goode <durham@fb.com>
parents: 34256
diff changeset
   373
                self.current = None
34163
550343626bb2 bundle2: move part counter to partiterator
Durham Goode <durham@fb.com>
parents: 34162
diff changeset
   374
        self.iterator = func()
550343626bb2 bundle2: move part counter to partiterator
Durham Goode <durham@fb.com>
parents: 34162
diff changeset
   375
        return self.iterator
34162
e9e0e1143fc5 bundle2: move part iterator a separate class
Durham Goode <durham@fb.com>
parents: 34116
diff changeset
   376
34164
21c2df59a1da bundle2: move exception handling into part iterator
Durham Goode <durham@fb.com>
parents: 34163
diff changeset
   377
    def __exit__(self, type, exc, tb):
34163
550343626bb2 bundle2: move part counter to partiterator
Durham Goode <durham@fb.com>
parents: 34162
diff changeset
   378
        if not self.iterator:
550343626bb2 bundle2: move part counter to partiterator
Durham Goode <durham@fb.com>
parents: 34162
diff changeset
   379
            return
550343626bb2 bundle2: move part counter to partiterator
Durham Goode <durham@fb.com>
parents: 34162
diff changeset
   380
34637
5f79f5f8487a bundle2: immediate exit for ctrl+c (issue5692)
Durham Goode <durham@fb.com>
parents: 34330
diff changeset
   381
        # Only gracefully abort in a normal exception situation. User aborts
5f79f5f8487a bundle2: immediate exit for ctrl+c (issue5692)
Durham Goode <durham@fb.com>
parents: 34330
diff changeset
   382
        # like Ctrl+C throw a KeyboardInterrupt which is not a base Exception,
5f79f5f8487a bundle2: immediate exit for ctrl+c (issue5692)
Durham Goode <durham@fb.com>
parents: 34330
diff changeset
   383
        # and should not gracefully cleanup.
5f79f5f8487a bundle2: immediate exit for ctrl+c (issue5692)
Durham Goode <durham@fb.com>
parents: 34330
diff changeset
   384
        if isinstance(exc, Exception):
34164
21c2df59a1da bundle2: move exception handling into part iterator
Durham Goode <durham@fb.com>
parents: 34163
diff changeset
   385
            # Any exceptions seeking to the end of the bundle at this point are
21c2df59a1da bundle2: move exception handling into part iterator
Durham Goode <durham@fb.com>
parents: 34163
diff changeset
   386
            # almost certainly related to the underlying stream being bad.
21c2df59a1da bundle2: move exception handling into part iterator
Durham Goode <durham@fb.com>
parents: 34163
diff changeset
   387
            # And, chances are that the exception we're handling is related to
21c2df59a1da bundle2: move exception handling into part iterator
Durham Goode <durham@fb.com>
parents: 34163
diff changeset
   388
            # getting in that bad state. So, we swallow the seeking error and
21c2df59a1da bundle2: move exception handling into part iterator
Durham Goode <durham@fb.com>
parents: 34163
diff changeset
   389
            # re-raise the original error.
21c2df59a1da bundle2: move exception handling into part iterator
Durham Goode <durham@fb.com>
parents: 34163
diff changeset
   390
            seekerror = False
21c2df59a1da bundle2: move exception handling into part iterator
Durham Goode <durham@fb.com>
parents: 34163
diff changeset
   391
            try:
34637
5f79f5f8487a bundle2: immediate exit for ctrl+c (issue5692)
Durham Goode <durham@fb.com>
parents: 34330
diff changeset
   392
                if self.current:
5f79f5f8487a bundle2: immediate exit for ctrl+c (issue5692)
Durham Goode <durham@fb.com>
parents: 34330
diff changeset
   393
                    # consume the part content to not corrupt the stream.
35135
db5038525718 bundle2: implement consume() API on unbundlepart
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35134
diff changeset
   394
                    self.current.consume()
34637
5f79f5f8487a bundle2: immediate exit for ctrl+c (issue5692)
Durham Goode <durham@fb.com>
parents: 34330
diff changeset
   395
34164
21c2df59a1da bundle2: move exception handling into part iterator
Durham Goode <durham@fb.com>
parents: 34163
diff changeset
   396
                for part in self.iterator:
21c2df59a1da bundle2: move exception handling into part iterator
Durham Goode <durham@fb.com>
parents: 34163
diff changeset
   397
                    # consume the bundle content
35135
db5038525718 bundle2: implement consume() API on unbundlepart
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35134
diff changeset
   398
                    part.consume()
34164
21c2df59a1da bundle2: move exception handling into part iterator
Durham Goode <durham@fb.com>
parents: 34163
diff changeset
   399
            except Exception:
21c2df59a1da bundle2: move exception handling into part iterator
Durham Goode <durham@fb.com>
parents: 34163
diff changeset
   400
                seekerror = True
21c2df59a1da bundle2: move exception handling into part iterator
Durham Goode <durham@fb.com>
parents: 34163
diff changeset
   401
21c2df59a1da bundle2: move exception handling into part iterator
Durham Goode <durham@fb.com>
parents: 34163
diff changeset
   402
            # Small hack to let caller code distinguish exceptions from bundle2
21c2df59a1da bundle2: move exception handling into part iterator
Durham Goode <durham@fb.com>
parents: 34163
diff changeset
   403
            # processing from processing the old format. This is mostly needed
21c2df59a1da bundle2: move exception handling into part iterator
Durham Goode <durham@fb.com>
parents: 34163
diff changeset
   404
            # to handle different return codes to unbundle according to the type
21c2df59a1da bundle2: move exception handling into part iterator
Durham Goode <durham@fb.com>
parents: 34163
diff changeset
   405
            # of bundle. We should probably clean up or drop this return code
21c2df59a1da bundle2: move exception handling into part iterator
Durham Goode <durham@fb.com>
parents: 34163
diff changeset
   406
            # craziness in a future version.
21c2df59a1da bundle2: move exception handling into part iterator
Durham Goode <durham@fb.com>
parents: 34163
diff changeset
   407
            exc.duringunbundle2 = True
21c2df59a1da bundle2: move exception handling into part iterator
Durham Goode <durham@fb.com>
parents: 34163
diff changeset
   408
            salvaged = []
21c2df59a1da bundle2: move exception handling into part iterator
Durham Goode <durham@fb.com>
parents: 34163
diff changeset
   409
            replycaps = None
21c2df59a1da bundle2: move exception handling into part iterator
Durham Goode <durham@fb.com>
parents: 34163
diff changeset
   410
            if self.op.reply is not None:
21c2df59a1da bundle2: move exception handling into part iterator
Durham Goode <durham@fb.com>
parents: 34163
diff changeset
   411
                salvaged = self.op.reply.salvageoutput()
21c2df59a1da bundle2: move exception handling into part iterator
Durham Goode <durham@fb.com>
parents: 34163
diff changeset
   412
                replycaps = self.op.reply.capabilities
21c2df59a1da bundle2: move exception handling into part iterator
Durham Goode <durham@fb.com>
parents: 34163
diff changeset
   413
            exc._replycaps = replycaps
21c2df59a1da bundle2: move exception handling into part iterator
Durham Goode <durham@fb.com>
parents: 34163
diff changeset
   414
            exc._bundle2salvagedoutput = salvaged
21c2df59a1da bundle2: move exception handling into part iterator
Durham Goode <durham@fb.com>
parents: 34163
diff changeset
   415
21c2df59a1da bundle2: move exception handling into part iterator
Durham Goode <durham@fb.com>
parents: 34163
diff changeset
   416
            # Re-raising from a variable loses the original stack. So only use
21c2df59a1da bundle2: move exception handling into part iterator
Durham Goode <durham@fb.com>
parents: 34163
diff changeset
   417
            # that form if we need to.
21c2df59a1da bundle2: move exception handling into part iterator
Durham Goode <durham@fb.com>
parents: 34163
diff changeset
   418
            if seekerror:
21c2df59a1da bundle2: move exception handling into part iterator
Durham Goode <durham@fb.com>
parents: 34163
diff changeset
   419
                raise exc
21c2df59a1da bundle2: move exception handling into part iterator
Durham Goode <durham@fb.com>
parents: 34163
diff changeset
   420
34163
550343626bb2 bundle2: move part counter to partiterator
Durham Goode <durham@fb.com>
parents: 34162
diff changeset
   421
        self.repo.ui.debug('bundle2-input-bundle: %i parts total\n' %
550343626bb2 bundle2: move part counter to partiterator
Durham Goode <durham@fb.com>
parents: 34162
diff changeset
   422
                           self.count)
34162
e9e0e1143fc5 bundle2: move part iterator a separate class
Durham Goode <durham@fb.com>
parents: 34116
diff changeset
   423
24851
df0ce98c882f bundle2: also save output when error happens during part processing
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24849
diff changeset
   424
def processbundle(repo, unbundler, transactiongetter=None, op=None):
20889
deed5edb72de bundle2: first version of a bundle processing
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20887
diff changeset
   425
    """This function process a bundle, apply effect to/from a repo
deed5edb72de bundle2: first version of a bundle processing
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20887
diff changeset
   426
20947
c33d7bf53812 bundle2: feed a unbundle20 to the `processbundle` function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20892
diff changeset
   427
    It iterates over each part then searches for and uses the proper handling
c33d7bf53812 bundle2: feed a unbundle20 to the `processbundle` function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20892
diff changeset
   428
    code to process the part. Parts are processed in order.
20889
deed5edb72de bundle2: first version of a bundle processing
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20887
diff changeset
   429
20891
1c6cd23fc221 bundle2: add some distinction between mandatory and advisory part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20890
diff changeset
   430
    Unknown Mandatory part will abort the process.
24851
df0ce98c882f bundle2: also save output when error happens during part processing
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24849
diff changeset
   431
df0ce98c882f bundle2: also save output when error happens during part processing
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24849
diff changeset
   432
    It is temporarily possible to provide a prebuilt bundleoperation to the
df0ce98c882f bundle2: also save output when error happens during part processing
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24849
diff changeset
   433
    function. This is used to ensure output is properly propagated in case of
df0ce98c882f bundle2: also save output when error happens during part processing
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24849
diff changeset
   434
    an error during the unbundling. This output capturing part will likely be
df0ce98c882f bundle2: also save output when error happens during part processing
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24849
diff changeset
   435
    reworked and this ability will probably go away in the process.
20889
deed5edb72de bundle2: first version of a bundle processing
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20887
diff changeset
   436
    """
24851
df0ce98c882f bundle2: also save output when error happens during part processing
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24849
diff changeset
   437
    if op is None:
df0ce98c882f bundle2: also save output when error happens during part processing
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24849
diff changeset
   438
        if transactiongetter is None:
df0ce98c882f bundle2: also save output when error happens during part processing
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24849
diff changeset
   439
            transactiongetter = _notransaction
df0ce98c882f bundle2: also save output when error happens during part processing
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24849
diff changeset
   440
        op = bundleoperation(repo, transactiongetter)
20889
deed5edb72de bundle2: first version of a bundle processing
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20887
diff changeset
   441
    # todo:
deed5edb72de bundle2: first version of a bundle processing
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20887
diff changeset
   442
    # - replace this is a init function soon.
deed5edb72de bundle2: first version of a bundle processing
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20887
diff changeset
   443
    # - exception catching
deed5edb72de bundle2: first version of a bundle processing
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20887
diff changeset
   444
    unbundler.params
25331
53ca47ef645d bundle2: add generic debug output regarding processed bundle
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25324
diff changeset
   445
    if repo.ui.debugflag:
53ca47ef645d bundle2: add generic debug output regarding processed bundle
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25324
diff changeset
   446
        msg = ['bundle2-input-bundle:']
53ca47ef645d bundle2: add generic debug output regarding processed bundle
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25324
diff changeset
   447
        if unbundler.params:
33135
126eae7dae74 bundle2: add debug info about the number of stream params
Siddharth Agarwal <sid0@fb.com>
parents: 33134
diff changeset
   448
            msg.append(' %i params' % len(unbundler.params))
33791
119e1c6be1ce bundle2: fix transaction availability detection
Boris Feld <boris.feld@octobus.net>
parents: 33737
diff changeset
   449
        if op._gettransaction is None or op._gettransaction is _notransaction:
25331
53ca47ef645d bundle2: add generic debug output regarding processed bundle
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25324
diff changeset
   450
            msg.append(' no-transaction')
53ca47ef645d bundle2: add generic debug output regarding processed bundle
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25324
diff changeset
   451
        else:
53ca47ef645d bundle2: add generic debug output regarding processed bundle
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25324
diff changeset
   452
            msg.append(' with-transaction')
53ca47ef645d bundle2: add generic debug output regarding processed bundle
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25324
diff changeset
   453
        msg.append('\n')
53ca47ef645d bundle2: add generic debug output regarding processed bundle
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25324
diff changeset
   454
        repo.ui.debug(''.join(msg))
34162
e9e0e1143fc5 bundle2: move part iterator a separate class
Durham Goode <durham@fb.com>
parents: 34116
diff changeset
   455
34268
f010097c885c bundle2: move part processing to a separate function
Durham Goode <durham@fb.com>
parents: 34267
diff changeset
   456
    processparts(repo, op, unbundler)
f010097c885c bundle2: move part processing to a separate function
Durham Goode <durham@fb.com>
parents: 34267
diff changeset
   457
f010097c885c bundle2: move part processing to a separate function
Durham Goode <durham@fb.com>
parents: 34267
diff changeset
   458
    return op
f010097c885c bundle2: move part processing to a separate function
Durham Goode <durham@fb.com>
parents: 34267
diff changeset
   459
f010097c885c bundle2: move part processing to a separate function
Durham Goode <durham@fb.com>
parents: 34267
diff changeset
   460
def processparts(repo, op, unbundler):
34164
21c2df59a1da bundle2: move exception handling into part iterator
Durham Goode <durham@fb.com>
parents: 34163
diff changeset
   461
    with partiterator(repo, op, unbundler) as parts:
21c2df59a1da bundle2: move exception handling into part iterator
Durham Goode <durham@fb.com>
parents: 34163
diff changeset
   462
        for part in parts:
21c2df59a1da bundle2: move exception handling into part iterator
Durham Goode <durham@fb.com>
parents: 34163
diff changeset
   463
            _processpart(op, part)
25332
0cdd1c811705 bundle2: add generic debug output at the end of bundle processing
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25331
diff changeset
   464
33050
f0efd2bffe1e bundle: extract _processchangegroup() method
Martin von Zweigbergk <martinvonz@google.com>
parents: 33049
diff changeset
   465
def _processchangegroup(op, cg, tr, source, url, **kwargs):
33461
bb72031f0ea8 changegroup: stop returning and recording added nodes in 'cg.apply'
Boris Feld <boris.feld@octobus.net>
parents: 33460
diff changeset
   466
    ret = cg.apply(op.repo, tr, source, url, **kwargs)
33050
f0efd2bffe1e bundle: extract _processchangegroup() method
Martin von Zweigbergk <martinvonz@google.com>
parents: 33049
diff changeset
   467
    op.records.add('changegroup', {
f0efd2bffe1e bundle: extract _processchangegroup() method
Martin von Zweigbergk <martinvonz@google.com>
parents: 33049
diff changeset
   468
        'return': ret,
f0efd2bffe1e bundle: extract _processchangegroup() method
Martin von Zweigbergk <martinvonz@google.com>
parents: 33049
diff changeset
   469
    })
f0efd2bffe1e bundle: extract _processchangegroup() method
Martin von Zweigbergk <martinvonz@google.com>
parents: 33049
diff changeset
   470
    return ret
f0efd2bffe1e bundle: extract _processchangegroup() method
Martin von Zweigbergk <martinvonz@google.com>
parents: 33049
diff changeset
   471
34266
07e4170f02f3 bundle2: move handler validation out of processpart
Durham Goode <durham@fb.com>
parents: 34265
diff changeset
   472
def _gethandler(op, part):
07e4170f02f3 bundle2: move handler validation out of processpart
Durham Goode <durham@fb.com>
parents: 34265
diff changeset
   473
    status = 'unknown' # used by debug output
07e4170f02f3 bundle2: move handler validation out of processpart
Durham Goode <durham@fb.com>
parents: 34265
diff changeset
   474
    try:
07e4170f02f3 bundle2: move handler validation out of processpart
Durham Goode <durham@fb.com>
parents: 34265
diff changeset
   475
        handler = parthandlermapping.get(part.type)
07e4170f02f3 bundle2: move handler validation out of processpart
Durham Goode <durham@fb.com>
parents: 34265
diff changeset
   476
        if handler is None:
07e4170f02f3 bundle2: move handler validation out of processpart
Durham Goode <durham@fb.com>
parents: 34265
diff changeset
   477
            status = 'unsupported-type'
07e4170f02f3 bundle2: move handler validation out of processpart
Durham Goode <durham@fb.com>
parents: 34265
diff changeset
   478
            raise error.BundleUnknownFeatureError(parttype=part.type)
34284
a48b3db0baba bundle2: stop using %r to quote part names
Augie Fackler <augie@google.com>
parents: 34268
diff changeset
   479
        indebug(op.ui, 'found a handler for part %s' % part.type)
34266
07e4170f02f3 bundle2: move handler validation out of processpart
Durham Goode <durham@fb.com>
parents: 34265
diff changeset
   480
        unknownparams = part.mandatorykeys - handler.params
07e4170f02f3 bundle2: move handler validation out of processpart
Durham Goode <durham@fb.com>
parents: 34265
diff changeset
   481
        if unknownparams:
07e4170f02f3 bundle2: move handler validation out of processpart
Durham Goode <durham@fb.com>
parents: 34265
diff changeset
   482
            unknownparams = list(unknownparams)
07e4170f02f3 bundle2: move handler validation out of processpart
Durham Goode <durham@fb.com>
parents: 34265
diff changeset
   483
            unknownparams.sort()
34284
a48b3db0baba bundle2: stop using %r to quote part names
Augie Fackler <augie@google.com>
parents: 34268
diff changeset
   484
            status = 'unsupported-params (%s)' % ', '.join(unknownparams)
34266
07e4170f02f3 bundle2: move handler validation out of processpart
Durham Goode <durham@fb.com>
parents: 34265
diff changeset
   485
            raise error.BundleUnknownFeatureError(parttype=part.type,
07e4170f02f3 bundle2: move handler validation out of processpart
Durham Goode <durham@fb.com>
parents: 34265
diff changeset
   486
                                                  params=unknownparams)
07e4170f02f3 bundle2: move handler validation out of processpart
Durham Goode <durham@fb.com>
parents: 34265
diff changeset
   487
        status = 'supported'
07e4170f02f3 bundle2: move handler validation out of processpart
Durham Goode <durham@fb.com>
parents: 34265
diff changeset
   488
    except error.BundleUnknownFeatureError as exc:
07e4170f02f3 bundle2: move handler validation out of processpart
Durham Goode <durham@fb.com>
parents: 34265
diff changeset
   489
        if part.mandatory: # mandatory parts
07e4170f02f3 bundle2: move handler validation out of processpart
Durham Goode <durham@fb.com>
parents: 34265
diff changeset
   490
            raise
07e4170f02f3 bundle2: move handler validation out of processpart
Durham Goode <durham@fb.com>
parents: 34265
diff changeset
   491
        indebug(op.ui, 'ignoring unsupported advisory part %s' % exc)
07e4170f02f3 bundle2: move handler validation out of processpart
Durham Goode <durham@fb.com>
parents: 34265
diff changeset
   492
        return # skip to part processing
07e4170f02f3 bundle2: move handler validation out of processpart
Durham Goode <durham@fb.com>
parents: 34265
diff changeset
   493
    finally:
07e4170f02f3 bundle2: move handler validation out of processpart
Durham Goode <durham@fb.com>
parents: 34265
diff changeset
   494
        if op.ui.debugflag:
07e4170f02f3 bundle2: move handler validation out of processpart
Durham Goode <durham@fb.com>
parents: 34265
diff changeset
   495
            msg = ['bundle2-input-part: "%s"' % part.type]
07e4170f02f3 bundle2: move handler validation out of processpart
Durham Goode <durham@fb.com>
parents: 34265
diff changeset
   496
            if not part.mandatory:
07e4170f02f3 bundle2: move handler validation out of processpart
Durham Goode <durham@fb.com>
parents: 34265
diff changeset
   497
                msg.append(' (advisory)')
07e4170f02f3 bundle2: move handler validation out of processpart
Durham Goode <durham@fb.com>
parents: 34265
diff changeset
   498
            nbmp = len(part.mandatorykeys)
07e4170f02f3 bundle2: move handler validation out of processpart
Durham Goode <durham@fb.com>
parents: 34265
diff changeset
   499
            nbap = len(part.params) - nbmp
07e4170f02f3 bundle2: move handler validation out of processpart
Durham Goode <durham@fb.com>
parents: 34265
diff changeset
   500
            if nbmp or nbap:
07e4170f02f3 bundle2: move handler validation out of processpart
Durham Goode <durham@fb.com>
parents: 34265
diff changeset
   501
                msg.append(' (params:')
07e4170f02f3 bundle2: move handler validation out of processpart
Durham Goode <durham@fb.com>
parents: 34265
diff changeset
   502
                if nbmp:
07e4170f02f3 bundle2: move handler validation out of processpart
Durham Goode <durham@fb.com>
parents: 34265
diff changeset
   503
                    msg.append(' %i mandatory' % nbmp)
07e4170f02f3 bundle2: move handler validation out of processpart
Durham Goode <durham@fb.com>
parents: 34265
diff changeset
   504
                if nbap:
07e4170f02f3 bundle2: move handler validation out of processpart
Durham Goode <durham@fb.com>
parents: 34265
diff changeset
   505
                    msg.append(' %i advisory' % nbmp)
07e4170f02f3 bundle2: move handler validation out of processpart
Durham Goode <durham@fb.com>
parents: 34265
diff changeset
   506
                msg.append(')')
07e4170f02f3 bundle2: move handler validation out of processpart
Durham Goode <durham@fb.com>
parents: 34265
diff changeset
   507
            msg.append(' %s\n' % status)
07e4170f02f3 bundle2: move handler validation out of processpart
Durham Goode <durham@fb.com>
parents: 34265
diff changeset
   508
            op.ui.debug(''.join(msg))
07e4170f02f3 bundle2: move handler validation out of processpart
Durham Goode <durham@fb.com>
parents: 34265
diff changeset
   509
07e4170f02f3 bundle2: move handler validation out of processpart
Durham Goode <durham@fb.com>
parents: 34265
diff changeset
   510
    return handler
07e4170f02f3 bundle2: move handler validation out of processpart
Durham Goode <durham@fb.com>
parents: 34265
diff changeset
   511
23008
d3137827016a bundle2: extract processing of part into its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23001
diff changeset
   512
def _processpart(op, part):
d3137827016a bundle2: extract processing of part into its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23001
diff changeset
   513
    """process a single part from a bundle
d3137827016a bundle2: extract processing of part into its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23001
diff changeset
   514
d3137827016a bundle2: extract processing of part into its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23001
diff changeset
   515
    The part is guaranteed to have been fully consumed when the function exits
d3137827016a bundle2: extract processing of part into its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23001
diff changeset
   516
    (even if an exception is raised)."""
34267
cc7b37c90616 bundle2: remove unnecessary try finally
Durham Goode <durham@fb.com>
parents: 34266
diff changeset
   517
    handler = _gethandler(op, part)
cc7b37c90616 bundle2: remove unnecessary try finally
Durham Goode <durham@fb.com>
parents: 34266
diff changeset
   518
    if handler is None:
cc7b37c90616 bundle2: remove unnecessary try finally
Durham Goode <durham@fb.com>
parents: 34266
diff changeset
   519
        return
23008
d3137827016a bundle2: extract processing of part into its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23001
diff changeset
   520
34267
cc7b37c90616 bundle2: remove unnecessary try finally
Durham Goode <durham@fb.com>
parents: 34266
diff changeset
   521
    # handler is called outside the above try block so that we don't
cc7b37c90616 bundle2: remove unnecessary try finally
Durham Goode <durham@fb.com>
parents: 34266
diff changeset
   522
    # risk catching KeyErrors from anything other than the
cc7b37c90616 bundle2: remove unnecessary try finally
Durham Goode <durham@fb.com>
parents: 34266
diff changeset
   523
    # parthandlermapping lookup (any KeyError raised by handler()
cc7b37c90616 bundle2: remove unnecessary try finally
Durham Goode <durham@fb.com>
parents: 34266
diff changeset
   524
    # itself represents a defect of a different variety).
cc7b37c90616 bundle2: remove unnecessary try finally
Durham Goode <durham@fb.com>
parents: 34266
diff changeset
   525
    output = None
cc7b37c90616 bundle2: remove unnecessary try finally
Durham Goode <durham@fb.com>
parents: 34266
diff changeset
   526
    if op.captureoutput and op.reply is not None:
cc7b37c90616 bundle2: remove unnecessary try finally
Durham Goode <durham@fb.com>
parents: 34266
diff changeset
   527
        op.ui.pushbuffer(error=True, subproc=True)
cc7b37c90616 bundle2: remove unnecessary try finally
Durham Goode <durham@fb.com>
parents: 34266
diff changeset
   528
        output = ''
cc7b37c90616 bundle2: remove unnecessary try finally
Durham Goode <durham@fb.com>
parents: 34266
diff changeset
   529
    try:
cc7b37c90616 bundle2: remove unnecessary try finally
Durham Goode <durham@fb.com>
parents: 34266
diff changeset
   530
        handler(op, part)
23008
d3137827016a bundle2: extract processing of part into its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23001
diff changeset
   531
    finally:
34267
cc7b37c90616 bundle2: remove unnecessary try finally
Durham Goode <durham@fb.com>
parents: 34266
diff changeset
   532
        if output is not None:
cc7b37c90616 bundle2: remove unnecessary try finally
Durham Goode <durham@fb.com>
parents: 34266
diff changeset
   533
            output = op.ui.popbuffer()
cc7b37c90616 bundle2: remove unnecessary try finally
Durham Goode <durham@fb.com>
parents: 34266
diff changeset
   534
        if output:
cc7b37c90616 bundle2: remove unnecessary try finally
Durham Goode <durham@fb.com>
parents: 34266
diff changeset
   535
            outpart = op.reply.newpart('output', data=output,
cc7b37c90616 bundle2: remove unnecessary try finally
Durham Goode <durham@fb.com>
parents: 34266
diff changeset
   536
                                       mandatory=False)
cc7b37c90616 bundle2: remove unnecessary try finally
Durham Goode <durham@fb.com>
parents: 34266
diff changeset
   537
            outpart.addparam(
cc7b37c90616 bundle2: remove unnecessary try finally
Durham Goode <durham@fb.com>
parents: 34266
diff changeset
   538
                'in-reply-to', pycompat.bytestr(part.id), mandatory=False)
23008
d3137827016a bundle2: extract processing of part into its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23001
diff changeset
   539
21138
f469879d27ec bundle2: extract capabilities decoding
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21137
diff changeset
   540
def decodecaps(blob):
23139
e53f6b72a0e4 spelling: fixes from proofreading of spell checker issues
Mads Kiilerich <madski@unity3d.com>
parents: 23067
diff changeset
   541
    """decode a bundle2 caps bytes blob into a dictionary
21138
f469879d27ec bundle2: extract capabilities decoding
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21137
diff changeset
   542
f469879d27ec bundle2: extract capabilities decoding
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21137
diff changeset
   543
    The blob is a list of capabilities (one per line)
f469879d27ec bundle2: extract capabilities decoding
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21137
diff changeset
   544
    Capabilities may have values using a line of the form::
f469879d27ec bundle2: extract capabilities decoding
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21137
diff changeset
   545
f469879d27ec bundle2: extract capabilities decoding
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21137
diff changeset
   546
        capability=value1,value2,value3
f469879d27ec bundle2: extract capabilities decoding
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21137
diff changeset
   547
f469879d27ec bundle2: extract capabilities decoding
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21137
diff changeset
   548
    The values are always a list."""
f469879d27ec bundle2: extract capabilities decoding
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21137
diff changeset
   549
    caps = {}
f469879d27ec bundle2: extract capabilities decoding
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21137
diff changeset
   550
    for line in blob.splitlines():
f469879d27ec bundle2: extract capabilities decoding
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21137
diff changeset
   551
        if not line:
f469879d27ec bundle2: extract capabilities decoding
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21137
diff changeset
   552
            continue
f469879d27ec bundle2: extract capabilities decoding
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21137
diff changeset
   553
        if '=' not in line:
f469879d27ec bundle2: extract capabilities decoding
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21137
diff changeset
   554
            key, vals = line, ()
f469879d27ec bundle2: extract capabilities decoding
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21137
diff changeset
   555
        else:
f469879d27ec bundle2: extract capabilities decoding
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21137
diff changeset
   556
            key, vals = line.split('=', 1)
f469879d27ec bundle2: extract capabilities decoding
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21137
diff changeset
   557
            vals = vals.split(',')
28883
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents: 28672
diff changeset
   558
        key = urlreq.unquote(key)
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents: 28672
diff changeset
   559
        vals = [urlreq.unquote(v) for v in vals]
21138
f469879d27ec bundle2: extract capabilities decoding
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21137
diff changeset
   560
        caps[key] = vals
f469879d27ec bundle2: extract capabilities decoding
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21137
diff changeset
   561
    return caps
f469879d27ec bundle2: extract capabilities decoding
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21137
diff changeset
   562
21139
2b8c82f7f11d bundle2: capabilities encoding
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21138
diff changeset
   563
def encodecaps(caps):
2b8c82f7f11d bundle2: capabilities encoding
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21138
diff changeset
   564
    """encode a bundle2 caps dictionary into a bytes blob"""
2b8c82f7f11d bundle2: capabilities encoding
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21138
diff changeset
   565
    chunks = []
2b8c82f7f11d bundle2: capabilities encoding
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21138
diff changeset
   566
    for ca in sorted(caps):
2b8c82f7f11d bundle2: capabilities encoding
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21138
diff changeset
   567
        vals = caps[ca]
28883
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents: 28672
diff changeset
   568
        ca = urlreq.quote(ca)
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents: 28672
diff changeset
   569
        vals = [urlreq.quote(v) for v in vals]
21139
2b8c82f7f11d bundle2: capabilities encoding
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21138
diff changeset
   570
        if vals:
2b8c82f7f11d bundle2: capabilities encoding
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21138
diff changeset
   571
            ca = "%s=%s" % (ca, ','.join(vals))
2b8c82f7f11d bundle2: capabilities encoding
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21138
diff changeset
   572
        chunks.append(ca)
2b8c82f7f11d bundle2: capabilities encoding
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21138
diff changeset
   573
    return '\n'.join(chunks)
2b8c82f7f11d bundle2: capabilities encoding
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21138
diff changeset
   574
28666
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27953
diff changeset
   575
bundletypes = {
30361
f81002f736d7 bundle2: use new compression engine API for compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30177
diff changeset
   576
    "": ("", 'UN'),       # only when using unbundle on ssh and old http servers
28666
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27953
diff changeset
   577
                          # since the unification ssh accepts a header but there
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27953
diff changeset
   578
                          # is no capability signaling it.
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27953
diff changeset
   579
    "HG20": (), # special-cased below
30361
f81002f736d7 bundle2: use new compression engine API for compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30177
diff changeset
   580
    "HG10UN": ("HG10UN", 'UN'),
28666
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27953
diff changeset
   581
    "HG10BZ": ("HG10", 'BZ'),
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27953
diff changeset
   582
    "HG10GZ": ("HG10GZ", 'GZ'),
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27953
diff changeset
   583
}
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27953
diff changeset
   584
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27953
diff changeset
   585
# hgweb uses this list to communicate its preferred type
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27953
diff changeset
   586
bundlepriority = ['HG10GZ', 'HG10BZ', 'HG10UN']
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27953
diff changeset
   587
20801
9c5183cb9bca bundle2: very first version of a bundle2 bundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
diff changeset
   588
class bundle20(object):
9c5183cb9bca bundle2: very first version of a bundle2 bundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
diff changeset
   589
    """represent an outgoing bundle2 container
9c5183cb9bca bundle2: very first version of a bundle2 bundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
diff changeset
   590
21599
57cd844d7a5b bundle2: have ``newpart`` automatically add the part to the bundle
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21598
diff changeset
   591
    Use the `addparam` method to add stream level parameter. and `newpart` to
20856
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20844
diff changeset
   592
    populate it. Then call `getchunks` to retrieve all the binary chunks of
21024
7731a2281cf0 spelling: fixes from spell checker
Mads Kiilerich <madski@unity3d.com>
parents: 21020
diff changeset
   593
    data that compose the bundle2 container."""
20801
9c5183cb9bca bundle2: very first version of a bundle2 bundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
diff changeset
   594
24686
e0e28e910fa3 bundle2: rename format, parts and config to final names
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24648
diff changeset
   595
    _magicstring = 'HG20'
24640
685639f9430d bundle20: move magic string into the class
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24071
diff changeset
   596
21134
2f8c4fa237f5 bundle2: adds a capabilities attribute on bundler20
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21133
diff changeset
   597
    def __init__(self, ui, capabilities=()):
20842
938718d72624 bundle2: print debug information during bundling
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20814
diff changeset
   598
        self.ui = ui
20801
9c5183cb9bca bundle2: very first version of a bundle2 bundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
diff changeset
   599
        self._params = []
9c5183cb9bca bundle2: very first version of a bundle2 bundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
diff changeset
   600
        self._parts = []
21136
b6fd496e5c72 bundle2: support for capabilities with values
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21135
diff changeset
   601
        self.capabilities = dict(capabilities)
30361
f81002f736d7 bundle2: use new compression engine API for compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30177
diff changeset
   602
        self._compengine = util.compengines.forbundletype('UN')
30757
511a4bf52754 bundle2: allow compression options to be passed to compressor
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30448
diff changeset
   603
        self._compopts = None
35787
a84dbc87dae9 exchange: send bundle2 stream clones uncompressed
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35783
diff changeset
   604
        # If compression is being handled by a consumer of the raw
a84dbc87dae9 exchange: send bundle2 stream clones uncompressed
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35783
diff changeset
   605
        # data (e.g. the wire protocol), unsetting this flag tells
a84dbc87dae9 exchange: send bundle2 stream clones uncompressed
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35783
diff changeset
   606
        # consumers that the bundle is best left uncompressed.
a84dbc87dae9 exchange: send bundle2 stream clones uncompressed
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35783
diff changeset
   607
        self.prefercompressed = True
26404
795f02a24b87 bundle2: allow compressed bundle
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26396
diff changeset
   608
30757
511a4bf52754 bundle2: allow compression options to be passed to compressor
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30448
diff changeset
   609
    def setcompression(self, alg, compopts=None):
26404
795f02a24b87 bundle2: allow compressed bundle
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26396
diff changeset
   610
        """setup core part compression to <alg>"""
30448
71b368e3b590 bundle2: equate 'UN' with no compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30367
diff changeset
   611
        if alg in (None, 'UN'):
26404
795f02a24b87 bundle2: allow compressed bundle
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26396
diff changeset
   612
            return
30877
aa25989b0658 bundle2: fix assertion that 'compression' hasn't been set
Siddharth Agarwal <sid0@fb.com>
parents: 30757
diff changeset
   613
        assert not any(n.lower() == 'compression' for n, v in self._params)
26404
795f02a24b87 bundle2: allow compressed bundle
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26396
diff changeset
   614
        self.addparam('Compression', alg)
30361
f81002f736d7 bundle2: use new compression engine API for compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30177
diff changeset
   615
        self._compengine = util.compengines.forbundletype(alg)
30757
511a4bf52754 bundle2: allow compression options to be passed to compressor
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30448
diff changeset
   616
        self._compopts = compopts
20801
9c5183cb9bca bundle2: very first version of a bundle2 bundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
diff changeset
   617
21900
b8bd97085ec9 bundle2: add a ``bundle20.nbparts`` property
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21660
diff changeset
   618
    @property
b8bd97085ec9 bundle2: add a ``bundle20.nbparts`` property
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21660
diff changeset
   619
    def nbparts(self):
b8bd97085ec9 bundle2: add a ``bundle20.nbparts`` property
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21660
diff changeset
   620
        """total number of parts added to the bundler"""
b8bd97085ec9 bundle2: add a ``bundle20.nbparts`` property
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21660
diff changeset
   621
        return len(self._parts)
b8bd97085ec9 bundle2: add a ``bundle20.nbparts`` property
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21660
diff changeset
   622
21597
1daad9dcdba2 bundle2: small doc update on the bundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21186
diff changeset
   623
    # methods used to defines the bundle2 content
20804
db9d3991d2c6 bundle2: support bundling simple parameter
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20803
diff changeset
   624
    def addparam(self, name, value=None):
db9d3991d2c6 bundle2: support bundling simple parameter
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20803
diff changeset
   625
        """add a stream level parameter"""
20813
8c74b3ce5b70 bundle2: refuse empty parameter name
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20812
diff changeset
   626
        if not name:
34286
047f2af8ceb0 bundle2: make ValueError messages native strings
Augie Fackler <augie@google.com>
parents: 34285
diff changeset
   627
            raise ValueError(r'empty parameter name')
34287
2bdd55abd56c bundle2: portably grab first byte of part name for letter check
Augie Fackler <augie@google.com>
parents: 34286
diff changeset
   628
        if name[0:1] not in pycompat.bytestr(string.ascii_letters):
34286
047f2af8ceb0 bundle2: make ValueError messages native strings
Augie Fackler <augie@google.com>
parents: 34285
diff changeset
   629
            raise ValueError(r'non letter first character: %s' % name)
20804
db9d3991d2c6 bundle2: support bundling simple parameter
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20803
diff changeset
   630
        self._params.append((name, value))
db9d3991d2c6 bundle2: support bundling simple parameter
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20803
diff changeset
   631
20856
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20844
diff changeset
   632
    def addpart(self, part):
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20844
diff changeset
   633
        """add a new part to the bundle2 container
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20844
diff changeset
   634
21024
7731a2281cf0 spelling: fixes from spell checker
Mads Kiilerich <madski@unity3d.com>
parents: 21020
diff changeset
   635
        Parts contains the actual applicative payload."""
20995
e995d104c87f bundle2: add an integer id to part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20952
diff changeset
   636
        assert part.id is None
e995d104c87f bundle2: add an integer id to part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20952
diff changeset
   637
        part.id = len(self._parts) # very cheap counter
20856
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20844
diff changeset
   638
        self._parts.append(part)
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20844
diff changeset
   639
21598
1b0dbb91de5b bundle2: add a ``newpart`` method to ``bundle20``
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21597
diff changeset
   640
    def newpart(self, typeid, *args, **kwargs):
21602
cc33ae50bab3 bundle2: warn about error during initialization in ``newpart`` docstring
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21601
diff changeset
   641
        """create a new part and add it to the containers
cc33ae50bab3 bundle2: warn about error during initialization in ``newpart`` docstring
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21601
diff changeset
   642
cc33ae50bab3 bundle2: warn about error during initialization in ``newpart`` docstring
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21601
diff changeset
   643
        As the part is directly added to the containers. For now, this means
cc33ae50bab3 bundle2: warn about error during initialization in ``newpart`` docstring
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21601
diff changeset
   644
        that any failure to properly initialize the part after calling
cc33ae50bab3 bundle2: warn about error during initialization in ``newpart`` docstring
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21601
diff changeset
   645
        ``newpart`` should result in a failure of the whole bundling process.
cc33ae50bab3 bundle2: warn about error during initialization in ``newpart`` docstring
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21601
diff changeset
   646
cc33ae50bab3 bundle2: warn about error during initialization in ``newpart`` docstring
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21601
diff changeset
   647
        You can still fall back to manually create and add if you need better
cc33ae50bab3 bundle2: warn about error during initialization in ``newpart`` docstring
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21601
diff changeset
   648
        control."""
21598
1b0dbb91de5b bundle2: add a ``newpart`` method to ``bundle20``
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21597
diff changeset
   649
        part = bundlepart(typeid, *args, **kwargs)
21599
57cd844d7a5b bundle2: have ``newpart`` automatically add the part to the bundle
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21598
diff changeset
   650
        self.addpart(part)
21598
1b0dbb91de5b bundle2: add a ``newpart`` method to ``bundle20``
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21597
diff changeset
   651
        return part
1b0dbb91de5b bundle2: add a ``newpart`` method to ``bundle20``
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21597
diff changeset
   652
21597
1daad9dcdba2 bundle2: small doc update on the bundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21186
diff changeset
   653
    # methods used to generate the bundle2 stream
20801
9c5183cb9bca bundle2: very first version of a bundle2 bundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
diff changeset
   654
    def getchunks(self):
25322
1f7c0403f8be bundle2: add generic debug output regarding generated bundle
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25321
diff changeset
   655
        if self.ui.debugflag:
1f7c0403f8be bundle2: add generic debug output regarding generated bundle
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25321
diff changeset
   656
            msg = ['bundle2-output-bundle: "%s",' % self._magicstring]
1f7c0403f8be bundle2: add generic debug output regarding generated bundle
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25321
diff changeset
   657
            if self._params:
1f7c0403f8be bundle2: add generic debug output regarding generated bundle
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25321
diff changeset
   658
                msg.append(' (%i params)' % len(self._params))
1f7c0403f8be bundle2: add generic debug output regarding generated bundle
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25321
diff changeset
   659
            msg.append(' %i parts total\n' % len(self._parts))
1f7c0403f8be bundle2: add generic debug output regarding generated bundle
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25321
diff changeset
   660
            self.ui.debug(''.join(msg))
25315
e28245b0da56 bundle2: handle new line in 'outdebug' function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25314
diff changeset
   661
        outdebug(self.ui, 'start emission of %s stream' % self._magicstring)
24640
685639f9430d bundle20: move magic string into the class
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24071
diff changeset
   662
        yield self._magicstring
20804
db9d3991d2c6 bundle2: support bundling simple parameter
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20803
diff changeset
   663
        param = self._paramchunk()
25315
e28245b0da56 bundle2: handle new line in 'outdebug' function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25314
diff changeset
   664
        outdebug(self.ui, 'bundle parameter: %s' % param)
20804
db9d3991d2c6 bundle2: support bundling simple parameter
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20803
diff changeset
   665
        yield _pack(_fstreamparamsize, len(param))
db9d3991d2c6 bundle2: support bundling simple parameter
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20803
diff changeset
   666
        if param:
db9d3991d2c6 bundle2: support bundling simple parameter
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20803
diff changeset
   667
            yield param
30757
511a4bf52754 bundle2: allow compression options to be passed to compressor
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30448
diff changeset
   668
        for chunk in self._compengine.compressstream(self._getcorechunk(),
511a4bf52754 bundle2: allow compression options to be passed to compressor
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30448
diff changeset
   669
                                                     self._compopts):
30367
5925bda42dbd bundle2: use compressstream compression engine API
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30363
diff changeset
   670
            yield chunk
20802
520df53ad26a bundle2: a very first version of bundle2 unbundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20801
diff changeset
   671
20804
db9d3991d2c6 bundle2: support bundling simple parameter
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20803
diff changeset
   672
    def _paramchunk(self):
db9d3991d2c6 bundle2: support bundling simple parameter
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20803
diff changeset
   673
        """return a encoded version of all stream parameters"""
db9d3991d2c6 bundle2: support bundling simple parameter
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20803
diff changeset
   674
        blocks = []
20809
b93bb639451a bundle2: support for bundling parameter value
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20808
diff changeset
   675
        for par, value in self._params:
28883
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents: 28672
diff changeset
   676
            par = urlreq.quote(par)
20809
b93bb639451a bundle2: support for bundling parameter value
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20808
diff changeset
   677
            if value is not None:
28883
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents: 28672
diff changeset
   678
                value = urlreq.quote(value)
20809
b93bb639451a bundle2: support for bundling parameter value
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20808
diff changeset
   679
                par = '%s=%s' % (par, value)
b93bb639451a bundle2: support for bundling parameter value
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20808
diff changeset
   680
            blocks.append(par)
20804
db9d3991d2c6 bundle2: support bundling simple parameter
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20803
diff changeset
   681
        return ' '.join(blocks)
db9d3991d2c6 bundle2: support bundling simple parameter
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20803
diff changeset
   682
26396
d90c30801cdf bundle20: extract core payload generation in its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26395
diff changeset
   683
    def _getcorechunk(self):
d90c30801cdf bundle20: extract core payload generation in its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26395
diff changeset
   684
        """yield chunk for the core part of the bundle
d90c30801cdf bundle20: extract core payload generation in its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26395
diff changeset
   685
d90c30801cdf bundle20: extract core payload generation in its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26395
diff changeset
   686
        (all but headers and parameters)"""
d90c30801cdf bundle20: extract core payload generation in its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26395
diff changeset
   687
        outdebug(self.ui, 'start of parts')
d90c30801cdf bundle20: extract core payload generation in its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26395
diff changeset
   688
        for part in self._parts:
d90c30801cdf bundle20: extract core payload generation in its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26395
diff changeset
   689
            outdebug(self.ui, 'bundle part: "%s"' % part.type)
d90c30801cdf bundle20: extract core payload generation in its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26395
diff changeset
   690
            for chunk in part.getchunks(ui=self.ui):
d90c30801cdf bundle20: extract core payload generation in its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26395
diff changeset
   691
                yield chunk
d90c30801cdf bundle20: extract core payload generation in its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26395
diff changeset
   692
        outdebug(self.ui, 'end of bundle')
d90c30801cdf bundle20: extract core payload generation in its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26395
diff changeset
   693
        yield _pack(_fpartheadersize, 0)
d90c30801cdf bundle20: extract core payload generation in its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26395
diff changeset
   694
d90c30801cdf bundle20: extract core payload generation in its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26395
diff changeset
   695
24794
21f2e8f48ae1 bundle2: add a 'salvageoutput' method on bundle20
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24793
diff changeset
   696
    def salvageoutput(self):
21f2e8f48ae1 bundle2: add a 'salvageoutput' method on bundle20
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24793
diff changeset
   697
        """return a list with a copy of all output parts in the bundle
21f2e8f48ae1 bundle2: add a 'salvageoutput' method on bundle20
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24793
diff changeset
   698
21f2e8f48ae1 bundle2: add a 'salvageoutput' method on bundle20
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24793
diff changeset
   699
        This is meant to be used during error handling to make sure we preserve
21f2e8f48ae1 bundle2: add a 'salvageoutput' method on bundle20
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24793
diff changeset
   700
        server output"""
21f2e8f48ae1 bundle2: add a 'salvageoutput' method on bundle20
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24793
diff changeset
   701
        salvaged = []
21f2e8f48ae1 bundle2: add a 'salvageoutput' method on bundle20
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24793
diff changeset
   702
        for part in self._parts:
21f2e8f48ae1 bundle2: add a 'salvageoutput' method on bundle20
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24793
diff changeset
   703
            if part.type.startswith('output'):
21f2e8f48ae1 bundle2: add a 'salvageoutput' method on bundle20
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24793
diff changeset
   704
                salvaged.append(part.copy())
21f2e8f48ae1 bundle2: add a 'salvageoutput' method on bundle20
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24793
diff changeset
   705
        return salvaged
21f2e8f48ae1 bundle2: add a 'salvageoutput' method on bundle20
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24793
diff changeset
   706
21f2e8f48ae1 bundle2: add a 'salvageoutput' method on bundle20
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24793
diff changeset
   707
21013
a813caca89b3 bundle2: extract stream/unpack logic in an unpackermixin
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21005
diff changeset
   708
class unpackermixin(object):
a813caca89b3 bundle2: extract stream/unpack logic in an unpackermixin
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21005
diff changeset
   709
    """A mixin to extract bytes and struct data from a stream"""
20802
520df53ad26a bundle2: a very first version of bundle2 unbundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20801
diff changeset
   710
21013
a813caca89b3 bundle2: extract stream/unpack logic in an unpackermixin
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21005
diff changeset
   711
    def __init__(self, fp):
20802
520df53ad26a bundle2: a very first version of bundle2 unbundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20801
diff changeset
   712
        self._fp = fp
520df53ad26a bundle2: a very first version of bundle2 unbundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20801
diff changeset
   713
520df53ad26a bundle2: a very first version of bundle2 unbundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20801
diff changeset
   714
    def _unpack(self, format):
31862
9bd9e9cb3f69 bundle2: clarify the docstring of unpackermixin methods
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31861
diff changeset
   715
        """unpack this struct format from the stream
9bd9e9cb3f69 bundle2: clarify the docstring of unpackermixin methods
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31861
diff changeset
   716
9bd9e9cb3f69 bundle2: clarify the docstring of unpackermixin methods
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31861
diff changeset
   717
        This method is meant for internal usage by the bundle2 protocol only.
9bd9e9cb3f69 bundle2: clarify the docstring of unpackermixin methods
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31861
diff changeset
   718
        They directly manipulate the low level stream including bundle2 level
9bd9e9cb3f69 bundle2: clarify the docstring of unpackermixin methods
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31861
diff changeset
   719
        instruction.
9bd9e9cb3f69 bundle2: clarify the docstring of unpackermixin methods
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31861
diff changeset
   720
9bd9e9cb3f69 bundle2: clarify the docstring of unpackermixin methods
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31861
diff changeset
   721
        Do not use it to implement higher-level logic or methods."""
20802
520df53ad26a bundle2: a very first version of bundle2 unbundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20801
diff changeset
   722
        data = self._readexact(struct.calcsize(format))
520df53ad26a bundle2: a very first version of bundle2 unbundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20801
diff changeset
   723
        return _unpack(format, data)
520df53ad26a bundle2: a very first version of bundle2 unbundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20801
diff changeset
   724
520df53ad26a bundle2: a very first version of bundle2 unbundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20801
diff changeset
   725
    def _readexact(self, size):
31862
9bd9e9cb3f69 bundle2: clarify the docstring of unpackermixin methods
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31861
diff changeset
   726
        """read exactly <size> bytes from the stream
9bd9e9cb3f69 bundle2: clarify the docstring of unpackermixin methods
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31861
diff changeset
   727
9bd9e9cb3f69 bundle2: clarify the docstring of unpackermixin methods
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31861
diff changeset
   728
        This method is meant for internal usage by the bundle2 protocol only.
9bd9e9cb3f69 bundle2: clarify the docstring of unpackermixin methods
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31861
diff changeset
   729
        They directly manipulate the low level stream including bundle2 level
9bd9e9cb3f69 bundle2: clarify the docstring of unpackermixin methods
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31861
diff changeset
   730
        instruction.
9bd9e9cb3f69 bundle2: clarify the docstring of unpackermixin methods
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31861
diff changeset
   731
9bd9e9cb3f69 bundle2: clarify the docstring of unpackermixin methods
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31861
diff changeset
   732
        Do not use it to implement higher-level logic or methods."""
20802
520df53ad26a bundle2: a very first version of bundle2 unbundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20801
diff changeset
   733
        return changegroup.readexactly(self._fp, size)
520df53ad26a bundle2: a very first version of bundle2 unbundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20801
diff changeset
   734
25640
39f0064a3079 bundle2.getunbundler: rename "header" to "magicstring"
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25518
diff changeset
   735
def getunbundler(ui, fp, magicstring=None):
39f0064a3079 bundle2.getunbundler: rename "header" to "magicstring"
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25518
diff changeset
   736
    """return a valid unbundler object for a given magicstring"""
39f0064a3079 bundle2.getunbundler: rename "header" to "magicstring"
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25518
diff changeset
   737
    if magicstring is None:
39f0064a3079 bundle2.getunbundler: rename "header" to "magicstring"
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25518
diff changeset
   738
        magicstring = changegroup.readexactly(fp, 4)
39f0064a3079 bundle2.getunbundler: rename "header" to "magicstring"
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25518
diff changeset
   739
    magic, version = magicstring[0:2], magicstring[2:4]
24648
5cac3accdaa1 unbundle20: allow generic dispatch between unbundlers
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24642
diff changeset
   740
    if magic != 'HG':
33134
0aae80d14fed bundle2: add some debugging information to the not-a-bundle error
Siddharth Agarwal <sid0@fb.com>
parents: 33056
diff changeset
   741
        ui.debug(
0aae80d14fed bundle2: add some debugging information to the not-a-bundle error
Siddharth Agarwal <sid0@fb.com>
parents: 33056
diff changeset
   742
            "error: invalid magic: %r (version %r), should be 'HG'\n"
0aae80d14fed bundle2: add some debugging information to the not-a-bundle error
Siddharth Agarwal <sid0@fb.com>
parents: 33056
diff changeset
   743
            % (magic, version))
26587
56b2bcea2529 error: get Abort from 'error' instead of 'util'
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26565
diff changeset
   744
        raise error.Abort(_('not a Mercurial bundle'))
24648
5cac3accdaa1 unbundle20: allow generic dispatch between unbundlers
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24642
diff changeset
   745
    unbundlerclass = formatmap.get(version)
5cac3accdaa1 unbundle20: allow generic dispatch between unbundlers
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24642
diff changeset
   746
    if unbundlerclass is None:
26587
56b2bcea2529 error: get Abort from 'error' instead of 'util'
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26565
diff changeset
   747
        raise error.Abort(_('unknown bundle version %s') % version)
24648
5cac3accdaa1 unbundle20: allow generic dispatch between unbundlers
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24642
diff changeset
   748
    unbundler = unbundlerclass(ui, fp)
25640
39f0064a3079 bundle2.getunbundler: rename "header" to "magicstring"
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25518
diff changeset
   749
    indebug(ui, 'start processing of %s stream' % magicstring)
24642
54e5c239c2d9 unbundle20: move header parsing into the 'getunbundler' function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24641
diff changeset
   750
    return unbundler
24641
60fecc5b14a4 unbundle20: retrieve unbundler instances through a factory function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24640
diff changeset
   751
21013
a813caca89b3 bundle2: extract stream/unpack logic in an unpackermixin
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21005
diff changeset
   752
class unbundle20(unpackermixin):
a813caca89b3 bundle2: extract stream/unpack logic in an unpackermixin
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21005
diff changeset
   753
    """interpret a bundle2 stream
a813caca89b3 bundle2: extract stream/unpack logic in an unpackermixin
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21005
diff changeset
   754
21129
07bcbf326c8d bundle2: use an official iterparts method to unbundle parts
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21066
diff changeset
   755
    This class is fed with a binary stream and yields parts through its
07bcbf326c8d bundle2: use an official iterparts method to unbundle parts
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21066
diff changeset
   756
    `iterparts` methods."""
21013
a813caca89b3 bundle2: extract stream/unpack logic in an unpackermixin
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21005
diff changeset
   757
26542
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26541
diff changeset
   758
    _magicstring = 'HG20'
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26541
diff changeset
   759
24642
54e5c239c2d9 unbundle20: move header parsing into the 'getunbundler' function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24641
diff changeset
   760
    def __init__(self, ui, fp):
21066
5ecfe76d0d96 bundle2: make header reading optional
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21062
diff changeset
   761
        """If header is specified, we do not read it out of the stream."""
21013
a813caca89b3 bundle2: extract stream/unpack logic in an unpackermixin
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21005
diff changeset
   762
        self.ui = ui
30363
d045b4091197 bundle2: use compression engines API to obtain decompressor
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30361
diff changeset
   763
        self._compengine = util.compengines.forbundletype('UN')
26802
42f705f2c02d bundle2: make unbundle.compressed return True when compressed
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26795
diff changeset
   764
        self._compressed = None
21013
a813caca89b3 bundle2: extract stream/unpack logic in an unpackermixin
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21005
diff changeset
   765
        super(unbundle20, self).__init__(fp)
a813caca89b3 bundle2: extract stream/unpack logic in an unpackermixin
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21005
diff changeset
   766
20802
520df53ad26a bundle2: a very first version of bundle2 unbundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20801
diff changeset
   767
    @util.propertycache
520df53ad26a bundle2: a very first version of bundle2 unbundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20801
diff changeset
   768
    def params(self):
21024
7731a2281cf0 spelling: fixes from spell checker
Mads Kiilerich <madski@unity3d.com>
parents: 21020
diff changeset
   769
        """dictionary of stream level parameters"""
25320
697d8953b04d bundle2: handle new line in 'indebug' function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25319
diff changeset
   770
        indebug(self.ui, 'reading bundle2 stream parameters')
20805
c5aaeca0cfbf bundle2: support for unbundling simple parameter
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20804
diff changeset
   771
        params = {}
c5aaeca0cfbf bundle2: support for unbundling simple parameter
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20804
diff changeset
   772
        paramssize = self._unpack(_fstreamparamsize)[0]
23011
006a81d07e57 bundle2: detect and disallow a negative chunk size
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23010
diff changeset
   773
        if paramssize < 0:
006a81d07e57 bundle2: detect and disallow a negative chunk size
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23010
diff changeset
   774
            raise error.BundleValueError('negative bundle param size: %i'
006a81d07e57 bundle2: detect and disallow a negative chunk size
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23010
diff changeset
   775
                                         % paramssize)
20805
c5aaeca0cfbf bundle2: support for unbundling simple parameter
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20804
diff changeset
   776
        if paramssize:
26541
d40029b4296e bundle2: split parameter retrieval and processing
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26404
diff changeset
   777
            params = self._readexact(paramssize)
d40029b4296e bundle2: split parameter retrieval and processing
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26404
diff changeset
   778
            params = self._processallparams(params)
20805
c5aaeca0cfbf bundle2: support for unbundling simple parameter
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20804
diff changeset
   779
        return params
20802
520df53ad26a bundle2: a very first version of bundle2 unbundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20801
diff changeset
   780
26541
d40029b4296e bundle2: split parameter retrieval and processing
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26404
diff changeset
   781
    def _processallparams(self, paramsblock):
d40029b4296e bundle2: split parameter retrieval and processing
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26404
diff changeset
   782
        """"""
29591
6215b5537ba5 bundle2: use a sorted dict for holding parameters
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29294
diff changeset
   783
        params = util.sortdict()
26541
d40029b4296e bundle2: split parameter retrieval and processing
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26404
diff changeset
   784
        for p in paramsblock.split(' '):
d40029b4296e bundle2: split parameter retrieval and processing
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26404
diff changeset
   785
            p = p.split('=', 1)
28883
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents: 28672
diff changeset
   786
            p = [urlreq.unquote(i) for i in p]
26541
d40029b4296e bundle2: split parameter retrieval and processing
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26404
diff changeset
   787
            if len(p) < 2:
d40029b4296e bundle2: split parameter retrieval and processing
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26404
diff changeset
   788
                p.append(None)
d40029b4296e bundle2: split parameter retrieval and processing
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26404
diff changeset
   789
            self._processparam(*p)
d40029b4296e bundle2: split parameter retrieval and processing
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26404
diff changeset
   790
            params[p[0]] = p[1]
d40029b4296e bundle2: split parameter retrieval and processing
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26404
diff changeset
   791
        return params
d40029b4296e bundle2: split parameter retrieval and processing
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26404
diff changeset
   792
d40029b4296e bundle2: split parameter retrieval and processing
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26404
diff changeset
   793
20844
2631204d7305 bundle2: implement the mandatory/advisory logic for parameter
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20843
diff changeset
   794
    def _processparam(self, name, value):
2631204d7305 bundle2: implement the mandatory/advisory logic for parameter
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20843
diff changeset
   795
        """process a parameter, applying its effect if needed
2631204d7305 bundle2: implement the mandatory/advisory logic for parameter
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20843
diff changeset
   796
2631204d7305 bundle2: implement the mandatory/advisory logic for parameter
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20843
diff changeset
   797
        Parameter starting with a lower case letter are advisory and will be
2631204d7305 bundle2: implement the mandatory/advisory logic for parameter
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20843
diff changeset
   798
        ignored when unknown.  Those starting with an upper case letter are
2631204d7305 bundle2: implement the mandatory/advisory logic for parameter
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20843
diff changeset
   799
        mandatory and will this function will raise a KeyError when unknown.
2631204d7305 bundle2: implement the mandatory/advisory logic for parameter
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20843
diff changeset
   800
2631204d7305 bundle2: implement the mandatory/advisory logic for parameter
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20843
diff changeset
   801
        Note: no option are currently supported. Any input will be either
2631204d7305 bundle2: implement the mandatory/advisory logic for parameter
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20843
diff changeset
   802
              ignored or failing.
2631204d7305 bundle2: implement the mandatory/advisory logic for parameter
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20843
diff changeset
   803
        """
2631204d7305 bundle2: implement the mandatory/advisory logic for parameter
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20843
diff changeset
   804
        if not name:
34284
a48b3db0baba bundle2: stop using %r to quote part names
Augie Fackler <augie@google.com>
parents: 34268
diff changeset
   805
            raise ValueError(r'empty parameter name')
a48b3db0baba bundle2: stop using %r to quote part names
Augie Fackler <augie@google.com>
parents: 34268
diff changeset
   806
        if name[0:1] not in pycompat.bytestr(string.ascii_letters):
a48b3db0baba bundle2: stop using %r to quote part names
Augie Fackler <augie@google.com>
parents: 34268
diff changeset
   807
            raise ValueError(r'non letter first character: %s' % name)
26395
4e7b0bf9f0b1 unbundle20: allow registering handlers for stream level parameters
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26393
diff changeset
   808
        try:
4e7b0bf9f0b1 unbundle20: allow registering handlers for stream level parameters
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26393
diff changeset
   809
            handler = b2streamparamsmap[name.lower()]
4e7b0bf9f0b1 unbundle20: allow registering handlers for stream level parameters
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26393
diff changeset
   810
        except KeyError:
34284
a48b3db0baba bundle2: stop using %r to quote part names
Augie Fackler <augie@google.com>
parents: 34268
diff changeset
   811
            if name[0:1].islower():
a48b3db0baba bundle2: stop using %r to quote part names
Augie Fackler <augie@google.com>
parents: 34268
diff changeset
   812
                indebug(self.ui, "ignoring unknown parameter %s" % name)
26395
4e7b0bf9f0b1 unbundle20: allow registering handlers for stream level parameters
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26393
diff changeset
   813
            else:
4e7b0bf9f0b1 unbundle20: allow registering handlers for stream level parameters
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26393
diff changeset
   814
                raise error.BundleUnknownFeatureError(params=(name,))
20844
2631204d7305 bundle2: implement the mandatory/advisory logic for parameter
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20843
diff changeset
   815
        else:
26395
4e7b0bf9f0b1 unbundle20: allow registering handlers for stream level parameters
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26393
diff changeset
   816
            handler(self, name, value)
20844
2631204d7305 bundle2: implement the mandatory/advisory logic for parameter
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20843
diff changeset
   817
26542
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26541
diff changeset
   818
    def _forwardchunks(self):
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26541
diff changeset
   819
        """utility to transfer a bundle2 as binary
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26541
diff changeset
   820
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26541
diff changeset
   821
        This is made necessary by the fact the 'getbundle' command over 'ssh'
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26541
diff changeset
   822
        have no way to know then the reply end, relying on the bundle to be
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26541
diff changeset
   823
        interpreted to know its end. This is terrible and we are sorry, but we
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26541
diff changeset
   824
        needed to move forward to get general delta enabled.
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26541
diff changeset
   825
        """
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26541
diff changeset
   826
        yield self._magicstring
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26541
diff changeset
   827
        assert 'params' not in vars(self)
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26541
diff changeset
   828
        paramssize = self._unpack(_fstreamparamsize)[0]
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26541
diff changeset
   829
        if paramssize < 0:
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26541
diff changeset
   830
            raise error.BundleValueError('negative bundle param size: %i'
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26541
diff changeset
   831
                                         % paramssize)
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26541
diff changeset
   832
        yield _pack(_fstreamparamsize, paramssize)
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26541
diff changeset
   833
        if paramssize:
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26541
diff changeset
   834
            params = self._readexact(paramssize)
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26541
diff changeset
   835
            self._processallparams(params)
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26541
diff changeset
   836
            yield params
30363
d045b4091197 bundle2: use compression engines API to obtain decompressor
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30361
diff changeset
   837
            assert self._compengine.bundletype == 'UN'
26542
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26541
diff changeset
   838
        # From there, payload might need to be decompressed
30363
d045b4091197 bundle2: use compression engines API to obtain decompressor
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30361
diff changeset
   839
        self._fp = self._compengine.decompressorreader(self._fp)
26542
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26541
diff changeset
   840
        emptycount = 0
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26541
diff changeset
   841
        while emptycount < 2:
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26541
diff changeset
   842
            # so we can brainlessly loop
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26541
diff changeset
   843
            assert _fpartheadersize == _fpayloadsize
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26541
diff changeset
   844
            size = self._unpack(_fpartheadersize)[0]
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26541
diff changeset
   845
            yield _pack(_fpartheadersize, size)
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26541
diff changeset
   846
            if size:
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26541
diff changeset
   847
                emptycount = 0
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26541
diff changeset
   848
            else:
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26541
diff changeset
   849
                emptycount += 1
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26541
diff changeset
   850
                continue
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26541
diff changeset
   851
            if size == flaginterrupt:
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26541
diff changeset
   852
                continue
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26541
diff changeset
   853
            elif size < 0:
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26541
diff changeset
   854
                raise error.BundleValueError('negative chunk size: %i')
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26541
diff changeset
   855
            yield self._readexact(size)
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26541
diff changeset
   856
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26541
diff changeset
   857
35137
da91e7309daf bundle2: don't use seekable bundle2 parts by default (issue5691)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35136
diff changeset
   858
    def iterparts(self, seekable=False):
20802
520df53ad26a bundle2: a very first version of bundle2 unbundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20801
diff changeset
   859
        """yield all parts contained in the stream"""
35137
da91e7309daf bundle2: don't use seekable bundle2 parts by default (issue5691)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35136
diff changeset
   860
        cls = seekableunbundlepart if seekable else unbundlepart
20802
520df53ad26a bundle2: a very first version of bundle2 unbundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20801
diff changeset
   861
        # make sure param have been loaded
520df53ad26a bundle2: a very first version of bundle2 unbundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20801
diff changeset
   862
        self.params
26404
795f02a24b87 bundle2: allow compressed bundle
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26396
diff changeset
   863
        # From there, payload need to be decompressed
30363
d045b4091197 bundle2: use compression engines API to obtain decompressor
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30361
diff changeset
   864
        self._fp = self._compengine.decompressorreader(self._fp)
25320
697d8953b04d bundle2: handle new line in 'indebug' function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25319
diff changeset
   865
        indebug(self.ui, 'start extraction of bundle2 parts')
21014
a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21013
diff changeset
   866
        headerblock = self._readpartheader()
a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21013
diff changeset
   867
        while headerblock is not None:
35137
da91e7309daf bundle2: don't use seekable bundle2 parts by default (issue5691)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35136
diff changeset
   868
            part = cls(self.ui, headerblock, self._fp)
20802
520df53ad26a bundle2: a very first version of bundle2 unbundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20801
diff changeset
   869
            yield part
35135
db5038525718 bundle2: implement consume() API on unbundlepart
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35134
diff changeset
   870
            # Ensure part is fully consumed so we can start reading the next
db5038525718 bundle2: implement consume() API on unbundlepart
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35134
diff changeset
   871
            # part.
db5038525718 bundle2: implement consume() API on unbundlepart
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35134
diff changeset
   872
            part.consume()
35136
2b72bc88043f bundle2: only seek to beginning of part in bundlerepo
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35135
diff changeset
   873
21014
a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21013
diff changeset
   874
            headerblock = self._readpartheader()
25320
697d8953b04d bundle2: handle new line in 'indebug' function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25319
diff changeset
   875
        indebug(self.ui, 'end of bundle2 stream')
20802
520df53ad26a bundle2: a very first version of bundle2 unbundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20801
diff changeset
   876
21014
a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21013
diff changeset
   877
    def _readpartheader(self):
a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21013
diff changeset
   878
        """reads a part header size and return the bytes blob
20864
9a75d2559cff bundle2: support unbundling empty part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20856
diff changeset
   879
21014
a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21013
diff changeset
   880
        returns None if empty"""
20864
9a75d2559cff bundle2: support unbundling empty part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20856
diff changeset
   881
        headersize = self._unpack(_fpartheadersize)[0]
23011
006a81d07e57 bundle2: detect and disallow a negative chunk size
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23010
diff changeset
   882
        if headersize < 0:
006a81d07e57 bundle2: detect and disallow a negative chunk size
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23010
diff changeset
   883
            raise error.BundleValueError('negative part header size: %i'
006a81d07e57 bundle2: detect and disallow a negative chunk size
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23010
diff changeset
   884
                                         % headersize)
25320
697d8953b04d bundle2: handle new line in 'indebug' function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25319
diff changeset
   885
        indebug(self.ui, 'part header size: %i' % headersize)
21014
a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21013
diff changeset
   886
        if headersize:
a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21013
diff changeset
   887
            return self._readexact(headersize)
a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21013
diff changeset
   888
        return None
20864
9a75d2559cff bundle2: support unbundling empty part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20856
diff changeset
   889
24071
184a2f6f40da bundle2.unbundle20: add compressed() method
Eric Sumner <ericsumner@fb.com>
parents: 24070
diff changeset
   890
    def compressed(self):
26802
42f705f2c02d bundle2: make unbundle.compressed return True when compressed
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26795
diff changeset
   891
        self.params # load params
42f705f2c02d bundle2: make unbundle.compressed return True when compressed
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26795
diff changeset
   892
        return self._compressed
20802
520df53ad26a bundle2: a very first version of bundle2 unbundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20801
diff changeset
   893
31863
cd7aaf344d83 bundle2: move the 'close' method off the unpackermixin
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31862
diff changeset
   894
    def close(self):
cd7aaf344d83 bundle2: move the 'close' method off the unpackermixin
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31862
diff changeset
   895
        """close underlying file"""
cd7aaf344d83 bundle2: move the 'close' method off the unpackermixin
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31862
diff changeset
   896
        if util.safehasattr(self._fp, 'close'):
cd7aaf344d83 bundle2: move the 'close' method off the unpackermixin
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31862
diff changeset
   897
            return self._fp.close()
cd7aaf344d83 bundle2: move the 'close' method off the unpackermixin
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31862
diff changeset
   898
24686
e0e28e910fa3 bundle2: rename format, parts and config to final names
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24648
diff changeset
   899
formatmap = {'20': unbundle20}
24648
5cac3accdaa1 unbundle20: allow generic dispatch between unbundlers
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24642
diff changeset
   900
26395
4e7b0bf9f0b1 unbundle20: allow registering handlers for stream level parameters
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26393
diff changeset
   901
b2streamparamsmap = {}
4e7b0bf9f0b1 unbundle20: allow registering handlers for stream level parameters
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26393
diff changeset
   902
4e7b0bf9f0b1 unbundle20: allow registering handlers for stream level parameters
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26393
diff changeset
   903
def b2streamparamhandler(name):
4e7b0bf9f0b1 unbundle20: allow registering handlers for stream level parameters
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26393
diff changeset
   904
    """register a handler for a stream level parameter"""
4e7b0bf9f0b1 unbundle20: allow registering handlers for stream level parameters
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26393
diff changeset
   905
    def decorator(func):
4e7b0bf9f0b1 unbundle20: allow registering handlers for stream level parameters
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26393
diff changeset
   906
        assert name not in formatmap
4e7b0bf9f0b1 unbundle20: allow registering handlers for stream level parameters
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26393
diff changeset
   907
        b2streamparamsmap[name] = func
4e7b0bf9f0b1 unbundle20: allow registering handlers for stream level parameters
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26393
diff changeset
   908
        return func
4e7b0bf9f0b1 unbundle20: allow registering handlers for stream level parameters
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26393
diff changeset
   909
    return decorator
4e7b0bf9f0b1 unbundle20: allow registering handlers for stream level parameters
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26393
diff changeset
   910
26404
795f02a24b87 bundle2: allow compressed bundle
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26396
diff changeset
   911
@b2streamparamhandler('compression')
795f02a24b87 bundle2: allow compressed bundle
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26396
diff changeset
   912
def processcompression(unbundler, param, value):
795f02a24b87 bundle2: allow compressed bundle
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26396
diff changeset
   913
    """read compression parameter and install payload decompression"""
30363
d045b4091197 bundle2: use compression engines API to obtain decompressor
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30361
diff changeset
   914
    if value not in util.compengines.supportedbundletypes:
26404
795f02a24b87 bundle2: allow compressed bundle
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26396
diff changeset
   915
        raise error.BundleUnknownFeatureError(params=(param,),
795f02a24b87 bundle2: allow compressed bundle
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26396
diff changeset
   916
                                              values=(value,))
30363
d045b4091197 bundle2: use compression engines API to obtain decompressor
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30361
diff changeset
   917
    unbundler._compengine = util.compengines.forbundletype(value)
26802
42f705f2c02d bundle2: make unbundle.compressed return True when compressed
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26795
diff changeset
   918
    if value is not None:
42f705f2c02d bundle2: make unbundle.compressed return True when compressed
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26795
diff changeset
   919
        unbundler._compressed = True
26404
795f02a24b87 bundle2: allow compressed bundle
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26396
diff changeset
   920
21005
3d38ebb586fe bundle2: rename part to bundlepart
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21004
diff changeset
   921
class bundlepart(object):
20856
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20844
diff changeset
   922
    """A bundle2 part contains application level payload
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20844
diff changeset
   923
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20844
diff changeset
   924
    The part `type` is used to route the part to the application level
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20844
diff changeset
   925
    handler.
21604
c399bf961cb9 bundle2: the ability to set ``data`` attribute of the part is now official
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21603
diff changeset
   926
c399bf961cb9 bundle2: the ability to set ``data`` attribute of the part is now official
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21603
diff changeset
   927
    The part payload is contained in ``part.data``. It could be raw bytes or a
21605
f9dabfceb259 bundle2: introduce a ``addparam`` method on part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21604
diff changeset
   928
    generator of byte chunks.
f9dabfceb259 bundle2: introduce a ``addparam`` method on part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21604
diff changeset
   929
f9dabfceb259 bundle2: introduce a ``addparam`` method on part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21604
diff changeset
   930
    You can add parameters to the part using the ``addparam`` method.
f9dabfceb259 bundle2: introduce a ``addparam`` method on part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21604
diff changeset
   931
    Parameters can be either mandatory (default) or advisory. Remote side
f9dabfceb259 bundle2: introduce a ``addparam`` method on part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21604
diff changeset
   932
    should be able to safely ignore the advisory ones.
f9dabfceb259 bundle2: introduce a ``addparam`` method on part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21604
diff changeset
   933
f9dabfceb259 bundle2: introduce a ``addparam`` method on part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21604
diff changeset
   934
    Both data and parameters cannot be modified after the generation has begun.
20856
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20844
diff changeset
   935
    """
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20844
diff changeset
   936
20877
9e9e3a4e9261 bundle2: part params
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20876
diff changeset
   937
    def __init__(self, parttype, mandatoryparams=(), advisoryparams=(),
23590
4440c7cc3728 bundle2.bundlepart: make mandatory part flag explicit in API
Eric Sumner <ericsumner@fb.com>
parents: 23586
diff changeset
   938
                 data='', mandatory=True):
23868
405eacbecc52 bundle2: enforce parttype as alphanumerical
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23590
diff changeset
   939
        validateparttype(parttype)
20995
e995d104c87f bundle2: add an integer id to part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20952
diff changeset
   940
        self.id = None
20856
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20844
diff changeset
   941
        self.type = parttype
21604
c399bf961cb9 bundle2: the ability to set ``data`` attribute of the part is now official
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21603
diff changeset
   942
        self._data = data
21605
f9dabfceb259 bundle2: introduce a ``addparam`` method on part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21604
diff changeset
   943
        self._mandatoryparams = list(mandatoryparams)
f9dabfceb259 bundle2: introduce a ``addparam`` method on part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21604
diff changeset
   944
        self._advisoryparams = list(advisoryparams)
21607
054fa5176fa7 bundle2: forbid duplicate parameter keys
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21606
diff changeset
   945
        # checking for duplicated entries
054fa5176fa7 bundle2: forbid duplicate parameter keys
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21606
diff changeset
   946
        self._seenparams = set()
054fa5176fa7 bundle2: forbid duplicate parameter keys
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21606
diff changeset
   947
        for pname, __ in self._mandatoryparams + self._advisoryparams:
054fa5176fa7 bundle2: forbid duplicate parameter keys
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21606
diff changeset
   948
            if pname in self._seenparams:
31652
4dbef666c6c9 bundle2: use ProgrammingError
Jun Wu <quark@fb.com>
parents: 31483
diff changeset
   949
                raise error.ProgrammingError('duplicated params: %s' % pname)
21607
054fa5176fa7 bundle2: forbid duplicate parameter keys
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21606
diff changeset
   950
            self._seenparams.add(pname)
21601
7ff01befc7ec bundle2: track life cycle of parts
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21600
diff changeset
   951
        # status of the part's generation:
7ff01befc7ec bundle2: track life cycle of parts
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21600
diff changeset
   952
        # - None: not started,
7ff01befc7ec bundle2: track life cycle of parts
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21600
diff changeset
   953
        # - False: currently generated,
7ff01befc7ec bundle2: track life cycle of parts
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21600
diff changeset
   954
        # - True: generation done.
7ff01befc7ec bundle2: track life cycle of parts
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21600
diff changeset
   955
        self._generated = None
23590
4440c7cc3728 bundle2.bundlepart: make mandatory part flag explicit in API
Eric Sumner <ericsumner@fb.com>
parents: 23586
diff changeset
   956
        self.mandatory = mandatory
20856
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20844
diff changeset
   957
30906
1f51b4658f21 bundle2: implement a basic __repr__ for bundle2 part
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 30905
diff changeset
   958
    def __repr__(self):
1f51b4658f21 bundle2: implement a basic __repr__ for bundle2 part
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 30905
diff changeset
   959
        cls = "%s.%s" % (self.__class__.__module__, self.__class__.__name__)
1f51b4658f21 bundle2: implement a basic __repr__ for bundle2 part
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 30905
diff changeset
   960
        return ('<%s object at %x; id: %s; type: %s; mandatory: %s>'
1f51b4658f21 bundle2: implement a basic __repr__ for bundle2 part
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 30905
diff changeset
   961
                % (cls, id(self), self.id, self.type, self.mandatory))
1f51b4658f21 bundle2: implement a basic __repr__ for bundle2 part
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 30905
diff changeset
   962
24793
2ec894582ea2 bundle2: add a 'copy' method on parts
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24743
diff changeset
   963
    def copy(self):
2ec894582ea2 bundle2: add a 'copy' method on parts
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24743
diff changeset
   964
        """return a copy of the part
2ec894582ea2 bundle2: add a 'copy' method on parts
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24743
diff changeset
   965
2ec894582ea2 bundle2: add a 'copy' method on parts
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24743
diff changeset
   966
        The new part have the very same content but no partid assigned yet.
2ec894582ea2 bundle2: add a 'copy' method on parts
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24743
diff changeset
   967
        Parts with generated data cannot be copied."""
2ec894582ea2 bundle2: add a 'copy' method on parts
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24743
diff changeset
   968
        assert not util.safehasattr(self.data, 'next')
2ec894582ea2 bundle2: add a 'copy' method on parts
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24743
diff changeset
   969
        return self.__class__(self.type, self._mandatoryparams,
2ec894582ea2 bundle2: add a 'copy' method on parts
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24743
diff changeset
   970
                              self._advisoryparams, self._data, self.mandatory)
2ec894582ea2 bundle2: add a 'copy' method on parts
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24743
diff changeset
   971
21604
c399bf961cb9 bundle2: the ability to set ``data`` attribute of the part is now official
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21603
diff changeset
   972
    # methods used to defines the part content
27879
52a4ad62b006 cleanup: use modern @property/@foo.setter property specification
Augie Fackler <augie@google.com>
parents: 27751
diff changeset
   973
    @property
52a4ad62b006 cleanup: use modern @property/@foo.setter property specification
Augie Fackler <augie@google.com>
parents: 27751
diff changeset
   974
    def data(self):
52a4ad62b006 cleanup: use modern @property/@foo.setter property specification
Augie Fackler <augie@google.com>
parents: 27751
diff changeset
   975
        return self._data
52a4ad62b006 cleanup: use modern @property/@foo.setter property specification
Augie Fackler <augie@google.com>
parents: 27751
diff changeset
   976
52a4ad62b006 cleanup: use modern @property/@foo.setter property specification
Augie Fackler <augie@google.com>
parents: 27751
diff changeset
   977
    @data.setter
52a4ad62b006 cleanup: use modern @property/@foo.setter property specification
Augie Fackler <augie@google.com>
parents: 27751
diff changeset
   978
    def data(self, data):
21604
c399bf961cb9 bundle2: the ability to set ``data`` attribute of the part is now official
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21603
diff changeset
   979
        if self._generated is not None:
21618
7568f5c1c801 bundle2: move exception classes into the error module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21617
diff changeset
   980
            raise error.ReadOnlyPartError('part is being generated')
21604
c399bf961cb9 bundle2: the ability to set ``data`` attribute of the part is now official
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21603
diff changeset
   981
        self._data = data
c399bf961cb9 bundle2: the ability to set ``data`` attribute of the part is now official
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21603
diff changeset
   982
21605
f9dabfceb259 bundle2: introduce a ``addparam`` method on part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21604
diff changeset
   983
    @property
f9dabfceb259 bundle2: introduce a ``addparam`` method on part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21604
diff changeset
   984
    def mandatoryparams(self):
f9dabfceb259 bundle2: introduce a ``addparam`` method on part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21604
diff changeset
   985
        # make it an immutable tuple to force people through ``addparam``
f9dabfceb259 bundle2: introduce a ``addparam`` method on part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21604
diff changeset
   986
        return tuple(self._mandatoryparams)
f9dabfceb259 bundle2: introduce a ``addparam`` method on part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21604
diff changeset
   987
f9dabfceb259 bundle2: introduce a ``addparam`` method on part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21604
diff changeset
   988
    @property
f9dabfceb259 bundle2: introduce a ``addparam`` method on part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21604
diff changeset
   989
    def advisoryparams(self):
f9dabfceb259 bundle2: introduce a ``addparam`` method on part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21604
diff changeset
   990
        # make it an immutable tuple to force people through ``addparam``
f9dabfceb259 bundle2: introduce a ``addparam`` method on part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21604
diff changeset
   991
        return tuple(self._advisoryparams)
f9dabfceb259 bundle2: introduce a ``addparam`` method on part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21604
diff changeset
   992
f9dabfceb259 bundle2: introduce a ``addparam`` method on part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21604
diff changeset
   993
    def addparam(self, name, value='', mandatory=True):
31861
6d055cd6669f bundle2: add documention to 'part.addparams'
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31652
diff changeset
   994
        """add a parameter to the part
6d055cd6669f bundle2: add documention to 'part.addparams'
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31652
diff changeset
   995
6d055cd6669f bundle2: add documention to 'part.addparams'
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31652
diff changeset
   996
        If 'mandatory' is set to True, the remote handler must claim support
6d055cd6669f bundle2: add documention to 'part.addparams'
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31652
diff changeset
   997
        for this parameter or the unbundling will be aborted.
6d055cd6669f bundle2: add documention to 'part.addparams'
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31652
diff changeset
   998
6d055cd6669f bundle2: add documention to 'part.addparams'
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31652
diff changeset
   999
        The 'name' and 'value' cannot exceed 255 bytes each.
6d055cd6669f bundle2: add documention to 'part.addparams'
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31652
diff changeset
  1000
        """
21605
f9dabfceb259 bundle2: introduce a ``addparam`` method on part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21604
diff changeset
  1001
        if self._generated is not None:
21618
7568f5c1c801 bundle2: move exception classes into the error module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21617
diff changeset
  1002
            raise error.ReadOnlyPartError('part is being generated')
21607
054fa5176fa7 bundle2: forbid duplicate parameter keys
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21606
diff changeset
  1003
        if name in self._seenparams:
054fa5176fa7 bundle2: forbid duplicate parameter keys
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21606
diff changeset
  1004
            raise ValueError('duplicated params: %s' % name)
054fa5176fa7 bundle2: forbid duplicate parameter keys
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21606
diff changeset
  1005
        self._seenparams.add(name)
21605
f9dabfceb259 bundle2: introduce a ``addparam`` method on part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21604
diff changeset
  1006
        params = self._advisoryparams
f9dabfceb259 bundle2: introduce a ``addparam`` method on part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21604
diff changeset
  1007
        if mandatory:
f9dabfceb259 bundle2: introduce a ``addparam`` method on part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21604
diff changeset
  1008
            params = self._mandatoryparams
f9dabfceb259 bundle2: introduce a ``addparam`` method on part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21604
diff changeset
  1009
        params.append((name, value))
f9dabfceb259 bundle2: introduce a ``addparam`` method on part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21604
diff changeset
  1010
21601
7ff01befc7ec bundle2: track life cycle of parts
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21600
diff changeset
  1011
    # methods used to generates the bundle2 stream
25321
b44ee346211f bundle2: add debug output for part generation
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25320
diff changeset
  1012
    def getchunks(self, ui):
21601
7ff01befc7ec bundle2: track life cycle of parts
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21600
diff changeset
  1013
        if self._generated is not None:
31652
4dbef666c6c9 bundle2: use ProgrammingError
Jun Wu <quark@fb.com>
parents: 31483
diff changeset
  1014
            raise error.ProgrammingError('part can only be consumed once')
21601
7ff01befc7ec bundle2: track life cycle of parts
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21600
diff changeset
  1015
        self._generated = False
25323
21a25fb81d2c bundle2: add generic debug output regarding generated parts
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25322
diff changeset
  1016
21a25fb81d2c bundle2: add generic debug output regarding generated parts
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25322
diff changeset
  1017
        if ui.debugflag:
21a25fb81d2c bundle2: add generic debug output regarding generated parts
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25322
diff changeset
  1018
            msg = ['bundle2-output-part: "%s"' % self.type]
21a25fb81d2c bundle2: add generic debug output regarding generated parts
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25322
diff changeset
  1019
            if not self.mandatory:
21a25fb81d2c bundle2: add generic debug output regarding generated parts
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25322
diff changeset
  1020
                msg.append(' (advisory)')
21a25fb81d2c bundle2: add generic debug output regarding generated parts
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25322
diff changeset
  1021
            nbmp = len(self.mandatoryparams)
21a25fb81d2c bundle2: add generic debug output regarding generated parts
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25322
diff changeset
  1022
            nbap = len(self.advisoryparams)
21a25fb81d2c bundle2: add generic debug output regarding generated parts
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25322
diff changeset
  1023
            if nbmp or nbap:
21a25fb81d2c bundle2: add generic debug output regarding generated parts
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25322
diff changeset
  1024
                msg.append(' (params:')
21a25fb81d2c bundle2: add generic debug output regarding generated parts
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25322
diff changeset
  1025
                if nbmp:
21a25fb81d2c bundle2: add generic debug output regarding generated parts
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25322
diff changeset
  1026
                    msg.append(' %i mandatory' % nbmp)
21a25fb81d2c bundle2: add generic debug output regarding generated parts
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25322
diff changeset
  1027
                if nbap:
21a25fb81d2c bundle2: add generic debug output regarding generated parts
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25322
diff changeset
  1028
                    msg.append(' %i advisory' % nbmp)
21a25fb81d2c bundle2: add generic debug output regarding generated parts
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25322
diff changeset
  1029
                msg.append(')')
21a25fb81d2c bundle2: add generic debug output regarding generated parts
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25322
diff changeset
  1030
            if not self.data:
21a25fb81d2c bundle2: add generic debug output regarding generated parts
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25322
diff changeset
  1031
                msg.append(' empty payload')
34285
ea7588c11c36 bundle2: update check for a generator to work on Python 3
Augie Fackler <augie@google.com>
parents: 34284
diff changeset
  1032
            elif (util.safehasattr(self.data, 'next')
ea7588c11c36 bundle2: update check for a generator to work on Python 3
Augie Fackler <augie@google.com>
parents: 34284
diff changeset
  1033
                  or util.safehasattr(self.data, '__next__')):
25323
21a25fb81d2c bundle2: add generic debug output regarding generated parts
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25322
diff changeset
  1034
                msg.append(' streamed payload')
21a25fb81d2c bundle2: add generic debug output regarding generated parts
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25322
diff changeset
  1035
            else:
21a25fb81d2c bundle2: add generic debug output regarding generated parts
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25322
diff changeset
  1036
                msg.append(' %i bytes payload' % len(self.data))
21a25fb81d2c bundle2: add generic debug output regarding generated parts
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25322
diff changeset
  1037
            msg.append('\n')
21a25fb81d2c bundle2: add generic debug output regarding generated parts
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25322
diff changeset
  1038
            ui.debug(''.join(msg))
21a25fb81d2c bundle2: add generic debug output regarding generated parts
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25322
diff changeset
  1039
20877
9e9e3a4e9261 bundle2: part params
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20876
diff changeset
  1040
        #### header
23590
4440c7cc3728 bundle2.bundlepart: make mandatory part flag explicit in API
Eric Sumner <ericsumner@fb.com>
parents: 23586
diff changeset
  1041
        if self.mandatory:
4440c7cc3728 bundle2.bundlepart: make mandatory part flag explicit in API
Eric Sumner <ericsumner@fb.com>
parents: 23586
diff changeset
  1042
            parttype = self.type.upper()
4440c7cc3728 bundle2.bundlepart: make mandatory part flag explicit in API
Eric Sumner <ericsumner@fb.com>
parents: 23586
diff changeset
  1043
        else:
4440c7cc3728 bundle2.bundlepart: make mandatory part flag explicit in API
Eric Sumner <ericsumner@fb.com>
parents: 23586
diff changeset
  1044
            parttype = self.type.lower()
33737
373ca5103ba8 bundle2: convert ints to strings using pycompat.bytestring()
Augie Fackler <augie@google.com>
parents: 33719
diff changeset
  1045
        outdebug(ui, 'part %s: "%s"' % (pycompat.bytestr(self.id), parttype))
20877
9e9e3a4e9261 bundle2: part params
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20876
diff changeset
  1046
        ## parttype
23590
4440c7cc3728 bundle2.bundlepart: make mandatory part flag explicit in API
Eric Sumner <ericsumner@fb.com>
parents: 23586
diff changeset
  1047
        header = [_pack(_fparttypesize, len(parttype)),
4440c7cc3728 bundle2.bundlepart: make mandatory part flag explicit in API
Eric Sumner <ericsumner@fb.com>
parents: 23586
diff changeset
  1048
                  parttype, _pack(_fpartid, self.id),
20856
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20844
diff changeset
  1049
                 ]
20877
9e9e3a4e9261 bundle2: part params
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20876
diff changeset
  1050
        ## parameters
9e9e3a4e9261 bundle2: part params
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20876
diff changeset
  1051
        # count
9e9e3a4e9261 bundle2: part params
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20876
diff changeset
  1052
        manpar = self.mandatoryparams
9e9e3a4e9261 bundle2: part params
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20876
diff changeset
  1053
        advpar = self.advisoryparams
9e9e3a4e9261 bundle2: part params
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20876
diff changeset
  1054
        header.append(_pack(_fpartparamcount, len(manpar), len(advpar)))
9e9e3a4e9261 bundle2: part params
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20876
diff changeset
  1055
        # size
9e9e3a4e9261 bundle2: part params
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20876
diff changeset
  1056
        parsizes = []
9e9e3a4e9261 bundle2: part params
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20876
diff changeset
  1057
        for key, value in manpar:
9e9e3a4e9261 bundle2: part params
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20876
diff changeset
  1058
            parsizes.append(len(key))
9e9e3a4e9261 bundle2: part params
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20876
diff changeset
  1059
            parsizes.append(len(value))
9e9e3a4e9261 bundle2: part params
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20876
diff changeset
  1060
        for key, value in advpar:
9e9e3a4e9261 bundle2: part params
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20876
diff changeset
  1061
            parsizes.append(len(key))
9e9e3a4e9261 bundle2: part params
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20876
diff changeset
  1062
            parsizes.append(len(value))
33698
da7c285ec6da bundle2: use modern Python division
Augie Fackler <augie@google.com>
parents: 33693
diff changeset
  1063
        paramsizes = _pack(_makefpartparamsizes(len(parsizes) // 2), *parsizes)
20877
9e9e3a4e9261 bundle2: part params
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20876
diff changeset
  1064
        header.append(paramsizes)
9e9e3a4e9261 bundle2: part params
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20876
diff changeset
  1065
        # key, value
9e9e3a4e9261 bundle2: part params
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20876
diff changeset
  1066
        for key, value in manpar:
9e9e3a4e9261 bundle2: part params
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20876
diff changeset
  1067
            header.append(key)
9e9e3a4e9261 bundle2: part params
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20876
diff changeset
  1068
            header.append(value)
9e9e3a4e9261 bundle2: part params
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20876
diff changeset
  1069
        for key, value in advpar:
9e9e3a4e9261 bundle2: part params
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20876
diff changeset
  1070
            header.append(key)
9e9e3a4e9261 bundle2: part params
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20876
diff changeset
  1071
            header.append(value)
9e9e3a4e9261 bundle2: part params
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20876
diff changeset
  1072
        ## finalize header
34256
ab379eed2e31 bundle2: raise a more helpful error if building a bundle part header fails
Augie Fackler <raf@durin42.com>
parents: 34228
diff changeset
  1073
        try:
ab379eed2e31 bundle2: raise a more helpful error if building a bundle part header fails
Augie Fackler <raf@durin42.com>
parents: 34228
diff changeset
  1074
            headerchunk = ''.join(header)
ab379eed2e31 bundle2: raise a more helpful error if building a bundle part header fails
Augie Fackler <raf@durin42.com>
parents: 34228
diff changeset
  1075
        except TypeError:
ab379eed2e31 bundle2: raise a more helpful error if building a bundle part header fails
Augie Fackler <raf@durin42.com>
parents: 34228
diff changeset
  1076
            raise TypeError(r'Found a non-bytes trying to '
ab379eed2e31 bundle2: raise a more helpful error if building a bundle part header fails
Augie Fackler <raf@durin42.com>
parents: 34228
diff changeset
  1077
                            r'build bundle part header: %r' % header)
25321
b44ee346211f bundle2: add debug output for part generation
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25320
diff changeset
  1078
        outdebug(ui, 'header chunk size: %i' % len(headerchunk))
20856
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20844
diff changeset
  1079
        yield _pack(_fpartheadersize, len(headerchunk))
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20844
diff changeset
  1080
        yield headerchunk
20877
9e9e3a4e9261 bundle2: part params
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20876
diff changeset
  1081
        ## payload
23067
420a051616ce bundle2: transmit exception during part generation
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23066
diff changeset
  1082
        try:
420a051616ce bundle2: transmit exception during part generation
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23066
diff changeset
  1083
            for chunk in self._payloadchunks():
25321
b44ee346211f bundle2: add debug output for part generation
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25320
diff changeset
  1084
                outdebug(ui, 'payload chunk size: %i' % len(chunk))
23067
420a051616ce bundle2: transmit exception during part generation
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23066
diff changeset
  1085
                yield _pack(_fpayloadsize, len(chunk))
420a051616ce bundle2: transmit exception during part generation
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23066
diff changeset
  1086
                yield chunk
26144
4bc3707f3e67 bundle2: don't try to recover from a GeneratorExit (issue4785)
Augie Fackler <augie@google.com>
parents: 25919
diff changeset
  1087
        except GeneratorExit:
4bc3707f3e67 bundle2: don't try to recover from a GeneratorExit (issue4785)
Augie Fackler <augie@google.com>
parents: 25919
diff changeset
  1088
            # GeneratorExit means that nobody is listening for our
4bc3707f3e67 bundle2: don't try to recover from a GeneratorExit (issue4785)
Augie Fackler <augie@google.com>
parents: 25919
diff changeset
  1089
            # results anyway, so just bail quickly rather than trying
4bc3707f3e67 bundle2: don't try to recover from a GeneratorExit (issue4785)
Augie Fackler <augie@google.com>
parents: 25919
diff changeset
  1090
            # to produce an error part.
4bc3707f3e67 bundle2: don't try to recover from a GeneratorExit (issue4785)
Augie Fackler <augie@google.com>
parents: 25919
diff changeset
  1091
            ui.debug('bundle2-generatorexit\n')
4bc3707f3e67 bundle2: don't try to recover from a GeneratorExit (issue4785)
Augie Fackler <augie@google.com>
parents: 25919
diff changeset
  1092
            raise
25660
328739ea70c3 global: mass rewrite to use modern exception syntax
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25641
diff changeset
  1093
        except BaseException as exc:
33708
1d5e497c08b3 py3: convert arbitrary exception object to byte string more reliably
Yuya Nishihara <yuya@tcha.org>
parents: 33702
diff changeset
  1094
            bexc = util.forcebytestr(exc)
23067
420a051616ce bundle2: transmit exception during part generation
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23066
diff changeset
  1095
            # backup exception data for later
25324
deed9c6b12d2 bundle2: add generic debug output regarding generated interruption
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25323
diff changeset
  1096
            ui.debug('bundle2-input-stream-interrupt: encoding exception %s'
33702
9f862fe7a8bd bundle2: obtain repr() of exception in a python3-safe way
Augie Fackler <augie@google.com>
parents: 33701
diff changeset
  1097
                     % bexc)
32232
76f9a0009b4b pycompat: extract helper to raise exception with traceback
Yuya Nishihara <yuya@tcha.org>
parents: 32024
diff changeset
  1098
            tb = sys.exc_info()[2]
33702
9f862fe7a8bd bundle2: obtain repr() of exception in a python3-safe way
Augie Fackler <augie@google.com>
parents: 33701
diff changeset
  1099
            msg = 'unexpected error: %s' % bexc
24686
e0e28e910fa3 bundle2: rename format, parts and config to final names
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24648
diff changeset
  1100
            interpart = bundlepart('error:abort', [('message', msg)],
23590
4440c7cc3728 bundle2.bundlepart: make mandatory part flag explicit in API
Eric Sumner <ericsumner@fb.com>
parents: 23586
diff changeset
  1101
                                   mandatory=False)
23067
420a051616ce bundle2: transmit exception during part generation
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23066
diff changeset
  1102
            interpart.id = 0
420a051616ce bundle2: transmit exception during part generation
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23066
diff changeset
  1103
            yield _pack(_fpayloadsize, -1)
25321
b44ee346211f bundle2: add debug output for part generation
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25320
diff changeset
  1104
            for chunk in interpart.getchunks(ui=ui):
23067
420a051616ce bundle2: transmit exception during part generation
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23066
diff changeset
  1105
                yield chunk
25321
b44ee346211f bundle2: add debug output for part generation
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25320
diff changeset
  1106
            outdebug(ui, 'closing payload chunk')
23067
420a051616ce bundle2: transmit exception during part generation
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23066
diff changeset
  1107
            # abort current part payload
420a051616ce bundle2: transmit exception during part generation
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23066
diff changeset
  1108
            yield _pack(_fpayloadsize, 0)
32232
76f9a0009b4b pycompat: extract helper to raise exception with traceback
Yuya Nishihara <yuya@tcha.org>
parents: 32024
diff changeset
  1109
            pycompat.raisewithtb(exc, tb)
21000
4cae06ae1562 bundle2: extract a _payloadchunks method for part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20998
diff changeset
  1110
        # end of payload
25321
b44ee346211f bundle2: add debug output for part generation
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25320
diff changeset
  1111
        outdebug(ui, 'closing payload chunk')
21000
4cae06ae1562 bundle2: extract a _payloadchunks method for part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20998
diff changeset
  1112
        yield _pack(_fpayloadsize, 0)
21601
7ff01befc7ec bundle2: track life cycle of parts
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21600
diff changeset
  1113
        self._generated = True
21000
4cae06ae1562 bundle2: extract a _payloadchunks method for part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20998
diff changeset
  1114
4cae06ae1562 bundle2: extract a _payloadchunks method for part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20998
diff changeset
  1115
    def _payloadchunks(self):
4cae06ae1562 bundle2: extract a _payloadchunks method for part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20998
diff changeset
  1116
        """yield chunks of a the part payload
4cae06ae1562 bundle2: extract a _payloadchunks method for part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20998
diff changeset
  1117
4cae06ae1562 bundle2: extract a _payloadchunks method for part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20998
diff changeset
  1118
        Exists to handle the different methods to provide data to a part."""
20876
ddd56f3eb786 bundle2: support for bundling and unbundling payload
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20864
diff changeset
  1119
        # we only support fixed size data now.
ddd56f3eb786 bundle2: support for bundling and unbundling payload
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20864
diff changeset
  1120
        # This will be improved in the future.
33699
5ae35a1347fd bundle2: look for __next__ as well as next to identify iterators
Augie Fackler <augie@google.com>
parents: 33698
diff changeset
  1121
        if (util.safehasattr(self.data, 'next')
5ae35a1347fd bundle2: look for __next__ as well as next to identify iterators
Augie Fackler <augie@google.com>
parents: 33698
diff changeset
  1122
            or util.safehasattr(self.data, '__next__')):
21001
c93bb6a08fa1 bundle2: support chunk iterator as part data
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21000
diff changeset
  1123
            buff = util.chunkbuffer(self.data)
c93bb6a08fa1 bundle2: support chunk iterator as part data
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21000
diff changeset
  1124
            chunk = buff.read(preferedchunksize)
c93bb6a08fa1 bundle2: support chunk iterator as part data
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21000
diff changeset
  1125
            while chunk:
c93bb6a08fa1 bundle2: support chunk iterator as part data
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21000
diff changeset
  1126
                yield chunk
c93bb6a08fa1 bundle2: support chunk iterator as part data
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21000
diff changeset
  1127
                chunk = buff.read(preferedchunksize)
c93bb6a08fa1 bundle2: support chunk iterator as part data
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21000
diff changeset
  1128
        elif len(self.data):
20876
ddd56f3eb786 bundle2: support for bundling and unbundling payload
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20864
diff changeset
  1129
            yield self.data
20802
520df53ad26a bundle2: a very first version of bundle2 unbundler
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20801
diff changeset
  1130
23066
ad144882318d bundle2: add a interrupt mechanism
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23029
diff changeset
  1131
ad144882318d bundle2: add a interrupt mechanism
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23029
diff changeset
  1132
flaginterrupt = -1
ad144882318d bundle2: add a interrupt mechanism
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23029
diff changeset
  1133
ad144882318d bundle2: add a interrupt mechanism
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23029
diff changeset
  1134
class interrupthandler(unpackermixin):
ad144882318d bundle2: add a interrupt mechanism
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23029
diff changeset
  1135
    """read one part and process it with restricted capability
ad144882318d bundle2: add a interrupt mechanism
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23029
diff changeset
  1136
ad144882318d bundle2: add a interrupt mechanism
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23029
diff changeset
  1137
    This allows to transmit exception raised on the producer size during part
ad144882318d bundle2: add a interrupt mechanism
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23029
diff changeset
  1138
    iteration while the consumer is reading a part.
ad144882318d bundle2: add a interrupt mechanism
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23029
diff changeset
  1139
ad144882318d bundle2: add a interrupt mechanism
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23029
diff changeset
  1140
    Part processed in this manner only have access to a ui object,"""
ad144882318d bundle2: add a interrupt mechanism
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23029
diff changeset
  1141
ad144882318d bundle2: add a interrupt mechanism
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23029
diff changeset
  1142
    def __init__(self, ui, fp):
ad144882318d bundle2: add a interrupt mechanism
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23029
diff changeset
  1143
        super(interrupthandler, self).__init__(fp)
ad144882318d bundle2: add a interrupt mechanism
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23029
diff changeset
  1144
        self.ui = ui
ad144882318d bundle2: add a interrupt mechanism
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23029
diff changeset
  1145
ad144882318d bundle2: add a interrupt mechanism
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23029
diff changeset
  1146
    def _readpartheader(self):
ad144882318d bundle2: add a interrupt mechanism
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23029
diff changeset
  1147
        """reads a part header size and return the bytes blob
ad144882318d bundle2: add a interrupt mechanism
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23029
diff changeset
  1148
ad144882318d bundle2: add a interrupt mechanism
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23029
diff changeset
  1149
        returns None if empty"""
ad144882318d bundle2: add a interrupt mechanism
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23029
diff changeset
  1150
        headersize = self._unpack(_fpartheadersize)[0]
ad144882318d bundle2: add a interrupt mechanism
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23029
diff changeset
  1151
        if headersize < 0:
ad144882318d bundle2: add a interrupt mechanism
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23029
diff changeset
  1152
            raise error.BundleValueError('negative part header size: %i'
ad144882318d bundle2: add a interrupt mechanism
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23029
diff changeset
  1153
                                         % headersize)
25318
64dd8f25fcf4 bundle2: introduce a specific function for debug messages while unbundling
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25317
diff changeset
  1154
        indebug(self.ui, 'part header size: %i\n' % headersize)
23066
ad144882318d bundle2: add a interrupt mechanism
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23029
diff changeset
  1155
        if headersize:
ad144882318d bundle2: add a interrupt mechanism
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23029
diff changeset
  1156
            return self._readexact(headersize)
ad144882318d bundle2: add a interrupt mechanism
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23029
diff changeset
  1157
        return None
ad144882318d bundle2: add a interrupt mechanism
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23029
diff changeset
  1158
ad144882318d bundle2: add a interrupt mechanism
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23029
diff changeset
  1159
    def __call__(self):
25335
8f7137a85a0e bundle2: add generic debug output regarding processed interruption
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25334
diff changeset
  1160
8f7137a85a0e bundle2: add generic debug output regarding processed interruption
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25334
diff changeset
  1161
        self.ui.debug('bundle2-input-stream-interrupt:'
8f7137a85a0e bundle2: add generic debug output regarding processed interruption
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25334
diff changeset
  1162
                      ' opening out of band context\n')
25320
697d8953b04d bundle2: handle new line in 'indebug' function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25319
diff changeset
  1163
        indebug(self.ui, 'bundle2 stream interruption, looking for a part.')
23066
ad144882318d bundle2: add a interrupt mechanism
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23029
diff changeset
  1164
        headerblock = self._readpartheader()
ad144882318d bundle2: add a interrupt mechanism
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23029
diff changeset
  1165
        if headerblock is None:
25320
697d8953b04d bundle2: handle new line in 'indebug' function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25319
diff changeset
  1166
            indebug(self.ui, 'no part found during interruption.')
23066
ad144882318d bundle2: add a interrupt mechanism
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23029
diff changeset
  1167
            return
35137
da91e7309daf bundle2: don't use seekable bundle2 parts by default (issue5691)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35136
diff changeset
  1168
        part = unbundlepart(self.ui, headerblock, self._fp)
23066
ad144882318d bundle2: add a interrupt mechanism
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23029
diff changeset
  1169
        op = interruptoperation(self.ui)
34265
e71890f27767 bundle2: move processpart stream maintenance into part iterator
Durham Goode <durham@fb.com>
parents: 34256
diff changeset
  1170
        hardabort = False
e71890f27767 bundle2: move processpart stream maintenance into part iterator
Durham Goode <durham@fb.com>
parents: 34256
diff changeset
  1171
        try:
e71890f27767 bundle2: move processpart stream maintenance into part iterator
Durham Goode <durham@fb.com>
parents: 34256
diff changeset
  1172
            _processpart(op, part)
e71890f27767 bundle2: move processpart stream maintenance into part iterator
Durham Goode <durham@fb.com>
parents: 34256
diff changeset
  1173
        except (SystemExit, KeyboardInterrupt):
e71890f27767 bundle2: move processpart stream maintenance into part iterator
Durham Goode <durham@fb.com>
parents: 34256
diff changeset
  1174
            hardabort = True
e71890f27767 bundle2: move processpart stream maintenance into part iterator
Durham Goode <durham@fb.com>
parents: 34256
diff changeset
  1175
            raise
e71890f27767 bundle2: move processpart stream maintenance into part iterator
Durham Goode <durham@fb.com>
parents: 34256
diff changeset
  1176
        finally:
e71890f27767 bundle2: move processpart stream maintenance into part iterator
Durham Goode <durham@fb.com>
parents: 34256
diff changeset
  1177
            if not hardabort:
35135
db5038525718 bundle2: implement consume() API on unbundlepart
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35134
diff changeset
  1178
                part.consume()
25335
8f7137a85a0e bundle2: add generic debug output regarding processed interruption
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25334
diff changeset
  1179
        self.ui.debug('bundle2-input-stream-interrupt:'
8f7137a85a0e bundle2: add generic debug output regarding processed interruption
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25334
diff changeset
  1180
                      ' closing out of band context\n')
23066
ad144882318d bundle2: add a interrupt mechanism
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23029
diff changeset
  1181
ad144882318d bundle2: add a interrupt mechanism
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23029
diff changeset
  1182
class interruptoperation(object):
ad144882318d bundle2: add a interrupt mechanism
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23029
diff changeset
  1183
    """A limited operation to be use by part handler during interruption
ad144882318d bundle2: add a interrupt mechanism
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23029
diff changeset
  1184
ad144882318d bundle2: add a interrupt mechanism
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23029
diff changeset
  1185
    It only have access to an ui object.
ad144882318d bundle2: add a interrupt mechanism
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23029
diff changeset
  1186
    """
ad144882318d bundle2: add a interrupt mechanism
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23029
diff changeset
  1187
ad144882318d bundle2: add a interrupt mechanism
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23029
diff changeset
  1188
    def __init__(self, ui):
ad144882318d bundle2: add a interrupt mechanism
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23029
diff changeset
  1189
        self.ui = ui
ad144882318d bundle2: add a interrupt mechanism
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23029
diff changeset
  1190
        self.reply = None
24878
e530cde6d115 bundle2: disable ouput capture unless we use http (issue4613 issue4615)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24851
diff changeset
  1191
        self.captureoutput = False
23066
ad144882318d bundle2: add a interrupt mechanism
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23029
diff changeset
  1192
ad144882318d bundle2: add a interrupt mechanism
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23029
diff changeset
  1193
    @property
ad144882318d bundle2: add a interrupt mechanism
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23029
diff changeset
  1194
    def repo(self):
31652
4dbef666c6c9 bundle2: use ProgrammingError
Jun Wu <quark@fb.com>
parents: 31483
diff changeset
  1195
        raise error.ProgrammingError('no repo access from stream interruption')
23066
ad144882318d bundle2: add a interrupt mechanism
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23029
diff changeset
  1196
ad144882318d bundle2: add a interrupt mechanism
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23029
diff changeset
  1197
    def gettransaction(self):
ad144882318d bundle2: add a interrupt mechanism
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23029
diff changeset
  1198
        raise TransactionUnavailable('no repo access from stream interruption')
ad144882318d bundle2: add a interrupt mechanism
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23029
diff changeset
  1199
35134
8aa43ff9c12c bundle2: implement generic part payload decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35133
diff changeset
  1200
def decodepayloadchunks(ui, fh):
8aa43ff9c12c bundle2: implement generic part payload decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35133
diff changeset
  1201
    """Reads bundle2 part payload data into chunks.
8aa43ff9c12c bundle2: implement generic part payload decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35133
diff changeset
  1202
8aa43ff9c12c bundle2: implement generic part payload decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35133
diff changeset
  1203
    Part payload data consists of framed chunks. This function takes
8aa43ff9c12c bundle2: implement generic part payload decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35133
diff changeset
  1204
    a file handle and emits those chunks.
8aa43ff9c12c bundle2: implement generic part payload decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35133
diff changeset
  1205
    """
35138
589eed45a452 bundle2: inline debug logging
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35137
diff changeset
  1206
    dolog = ui.configbool('devel', 'bundle2.debug')
589eed45a452 bundle2: inline debug logging
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35137
diff changeset
  1207
    debug = ui.debug
589eed45a452 bundle2: inline debug logging
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35137
diff changeset
  1208
35140
764e3ad1cf54 bundle2: inline struct operations
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35139
diff changeset
  1209
    headerstruct = struct.Struct(_fpayloadsize)
764e3ad1cf54 bundle2: inline struct operations
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35139
diff changeset
  1210
    headersize = headerstruct.size
764e3ad1cf54 bundle2: inline struct operations
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35139
diff changeset
  1211
    unpack = headerstruct.unpack
764e3ad1cf54 bundle2: inline struct operations
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35139
diff changeset
  1212
35134
8aa43ff9c12c bundle2: implement generic part payload decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35133
diff changeset
  1213
    readexactly = changegroup.readexactly
35139
1fb0846ad792 bundle2: inline changegroup.readexactly()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35138
diff changeset
  1214
    read = fh.read
35134
8aa43ff9c12c bundle2: implement generic part payload decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35133
diff changeset
  1215
35140
764e3ad1cf54 bundle2: inline struct operations
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35139
diff changeset
  1216
    chunksize = unpack(readexactly(fh, headersize))[0]
35134
8aa43ff9c12c bundle2: implement generic part payload decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35133
diff changeset
  1217
    indebug(ui, 'payload chunk size: %i' % chunksize)
8aa43ff9c12c bundle2: implement generic part payload decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35133
diff changeset
  1218
35139
1fb0846ad792 bundle2: inline changegroup.readexactly()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35138
diff changeset
  1219
    # changegroup.readexactly() is inlined below for performance.
35134
8aa43ff9c12c bundle2: implement generic part payload decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35133
diff changeset
  1220
    while chunksize:
8aa43ff9c12c bundle2: implement generic part payload decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35133
diff changeset
  1221
        if chunksize >= 0:
35139
1fb0846ad792 bundle2: inline changegroup.readexactly()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35138
diff changeset
  1222
            s = read(chunksize)
1fb0846ad792 bundle2: inline changegroup.readexactly()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35138
diff changeset
  1223
            if len(s) < chunksize:
1fb0846ad792 bundle2: inline changegroup.readexactly()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35138
diff changeset
  1224
                raise error.Abort(_('stream ended unexpectedly '
1fb0846ad792 bundle2: inline changegroup.readexactly()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35138
diff changeset
  1225
                                    ' (got %d bytes, expected %d)') %
1fb0846ad792 bundle2: inline changegroup.readexactly()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35138
diff changeset
  1226
                                  (len(s), chunksize))
1fb0846ad792 bundle2: inline changegroup.readexactly()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35138
diff changeset
  1227
1fb0846ad792 bundle2: inline changegroup.readexactly()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35138
diff changeset
  1228
            yield s
35134
8aa43ff9c12c bundle2: implement generic part payload decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35133
diff changeset
  1229
        elif chunksize == flaginterrupt:
8aa43ff9c12c bundle2: implement generic part payload decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35133
diff changeset
  1230
            # Interrupt "signal" detected. The regular stream is interrupted
8aa43ff9c12c bundle2: implement generic part payload decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35133
diff changeset
  1231
            # and a bundle2 part follows. Consume it.
8aa43ff9c12c bundle2: implement generic part payload decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35133
diff changeset
  1232
            interrupthandler(ui, fh)()
8aa43ff9c12c bundle2: implement generic part payload decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35133
diff changeset
  1233
        else:
8aa43ff9c12c bundle2: implement generic part payload decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35133
diff changeset
  1234
            raise error.BundleValueError(
8aa43ff9c12c bundle2: implement generic part payload decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35133
diff changeset
  1235
                'negative payload chunk size: %s' % chunksize)
8aa43ff9c12c bundle2: implement generic part payload decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35133
diff changeset
  1236
35139
1fb0846ad792 bundle2: inline changegroup.readexactly()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35138
diff changeset
  1237
        s = read(headersize)
1fb0846ad792 bundle2: inline changegroup.readexactly()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35138
diff changeset
  1238
        if len(s) < headersize:
1fb0846ad792 bundle2: inline changegroup.readexactly()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35138
diff changeset
  1239
            raise error.Abort(_('stream ended unexpectedly '
1fb0846ad792 bundle2: inline changegroup.readexactly()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35138
diff changeset
  1240
                                ' (got %d bytes, expected %d)') %
1fb0846ad792 bundle2: inline changegroup.readexactly()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35138
diff changeset
  1241
                              (len(s), chunksize))
1fb0846ad792 bundle2: inline changegroup.readexactly()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35138
diff changeset
  1242
35140
764e3ad1cf54 bundle2: inline struct operations
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35139
diff changeset
  1243
        chunksize = unpack(s)[0]
35138
589eed45a452 bundle2: inline debug logging
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35137
diff changeset
  1244
589eed45a452 bundle2: inline debug logging
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35137
diff changeset
  1245
        # indebug() inlined for performance.
589eed45a452 bundle2: inline debug logging
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35137
diff changeset
  1246
        if dolog:
589eed45a452 bundle2: inline debug logging
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35137
diff changeset
  1247
            debug('bundle2-input: payload chunk size: %i\n' % chunksize)
35134
8aa43ff9c12c bundle2: implement generic part payload decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35133
diff changeset
  1248
21014
a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21013
diff changeset
  1249
class unbundlepart(unpackermixin):
a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21013
diff changeset
  1250
    """a bundle part read from a bundle"""
a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21013
diff changeset
  1251
a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21013
diff changeset
  1252
    def __init__(self, ui, header, fp):
a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21013
diff changeset
  1253
        super(unbundlepart, self).__init__(fp)
31889
a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31863
diff changeset
  1254
        self._seekable = (util.safehasattr(fp, 'seek') and
a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31863
diff changeset
  1255
                          util.safehasattr(fp, 'tell'))
21014
a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21013
diff changeset
  1256
        self.ui = ui
a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21013
diff changeset
  1257
        # unbundle state attr
a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21013
diff changeset
  1258
        self._headerdata = header
21015
14dd49260246 bundle2: move the fromheader closure into the class itself
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21014
diff changeset
  1259
        self._headeroffset = 0
21019
3dc09f831a2e bundle2: lazy unbundle of part payload
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21016
diff changeset
  1260
        self._initialized = False
3dc09f831a2e bundle2: lazy unbundle of part payload
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21016
diff changeset
  1261
        self.consumed = False
21014
a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21013
diff changeset
  1262
        # part data
a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21013
diff changeset
  1263
        self.id = None
a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21013
diff changeset
  1264
        self.type = None
a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21013
diff changeset
  1265
        self.mandatoryparams = None
a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21013
diff changeset
  1266
        self.advisoryparams = None
21610
d6056805f8f4 bundle2: introduce a ``params`` dictionary on unbundled parts
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21609
diff changeset
  1267
        self.params = None
21612
f221eb0531d9 bundle2: expose mandatory params in a mandatorykeys attribute
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21611
diff changeset
  1268
        self.mandatorykeys = ()
21019
3dc09f831a2e bundle2: lazy unbundle of part payload
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21016
diff changeset
  1269
        self._readheader()
23585
94b25d71dd0f bundle2.unbundlepart: decouple mandatory from parttype
Eric Sumner <ericsumner@fb.com>
parents: 23439
diff changeset
  1270
        self._mandatory = None
24036
c7601086338a bundle2.unbundlepart: tell() implementation
Eric Sumner <ericsumner@fb.com>
parents: 24035
diff changeset
  1271
        self._pos = 0
21014
a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21013
diff changeset
  1272
21015
14dd49260246 bundle2: move the fromheader closure into the class itself
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21014
diff changeset
  1273
    def _fromheader(self, size):
14dd49260246 bundle2: move the fromheader closure into the class itself
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21014
diff changeset
  1274
        """return the next <size> byte from the header"""
14dd49260246 bundle2: move the fromheader closure into the class itself
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21014
diff changeset
  1275
        offset = self._headeroffset
14dd49260246 bundle2: move the fromheader closure into the class itself
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21014
diff changeset
  1276
        data = self._headerdata[offset:(offset + size)]
21019
3dc09f831a2e bundle2: lazy unbundle of part payload
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21016
diff changeset
  1277
        self._headeroffset = offset + size
21015
14dd49260246 bundle2: move the fromheader closure into the class itself
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21014
diff changeset
  1278
        return data
14dd49260246 bundle2: move the fromheader closure into the class itself
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21014
diff changeset
  1279
21016
b477afb1c81e bundle2: move unpackheader closure into the class
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21015
diff changeset
  1280
    def _unpackheader(self, format):
b477afb1c81e bundle2: move unpackheader closure into the class
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21015
diff changeset
  1281
        """read given format from header
b477afb1c81e bundle2: move unpackheader closure into the class
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21015
diff changeset
  1282
b477afb1c81e bundle2: move unpackheader closure into the class
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21015
diff changeset
  1283
        This automatically compute the size of the format to read."""
b477afb1c81e bundle2: move unpackheader closure into the class
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21015
diff changeset
  1284
        data = self._fromheader(struct.calcsize(format))
b477afb1c81e bundle2: move unpackheader closure into the class
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21015
diff changeset
  1285
        return _unpack(format, data)
b477afb1c81e bundle2: move unpackheader closure into the class
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21015
diff changeset
  1286
21608
3cb96ca90c17 bundle2: introduce an ``_initparams`` method
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21607
diff changeset
  1287
    def _initparams(self, mandatoryparams, advisoryparams):
3cb96ca90c17 bundle2: introduce an ``_initparams`` method
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21607
diff changeset
  1288
        """internal function to setup all logic related parameters"""
21609
63cc2594ef8a bundle2: make sure unbundled part param are read-only
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21608
diff changeset
  1289
        # make it read only to prevent people touching it by mistake.
63cc2594ef8a bundle2: make sure unbundled part param are read-only
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21608
diff changeset
  1290
        self.mandatoryparams = tuple(mandatoryparams)
63cc2594ef8a bundle2: make sure unbundled part param are read-only
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21608
diff changeset
  1291
        self.advisoryparams  = tuple(advisoryparams)
21610
d6056805f8f4 bundle2: introduce a ``params`` dictionary on unbundled parts
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21609
diff changeset
  1292
        # user friendly UI
29591
6215b5537ba5 bundle2: use a sorted dict for holding parameters
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29294
diff changeset
  1293
        self.params = util.sortdict(self.mandatoryparams)
6215b5537ba5 bundle2: use a sorted dict for holding parameters
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29294
diff changeset
  1294
        self.params.update(self.advisoryparams)
21612
f221eb0531d9 bundle2: expose mandatory params in a mandatorykeys attribute
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21611
diff changeset
  1295
        self.mandatorykeys = frozenset(p[0] for p in mandatoryparams)
21608
3cb96ca90c17 bundle2: introduce an ``_initparams`` method
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21607
diff changeset
  1296
21019
3dc09f831a2e bundle2: lazy unbundle of part payload
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21016
diff changeset
  1297
    def _readheader(self):
21014
a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21013
diff changeset
  1298
        """read the header and setup the object"""
21016
b477afb1c81e bundle2: move unpackheader closure into the class
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21015
diff changeset
  1299
        typesize = self._unpackheader(_fparttypesize)[0]
21015
14dd49260246 bundle2: move the fromheader closure into the class itself
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21014
diff changeset
  1300
        self.type = self._fromheader(typesize)
25320
697d8953b04d bundle2: handle new line in 'indebug' function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25319
diff changeset
  1301
        indebug(self.ui, 'part type: "%s"' % self.type)
21016
b477afb1c81e bundle2: move unpackheader closure into the class
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21015
diff changeset
  1302
        self.id = self._unpackheader(_fpartid)[0]
33737
373ca5103ba8 bundle2: convert ints to strings using pycompat.bytestring()
Augie Fackler <augie@google.com>
parents: 33719
diff changeset
  1303
        indebug(self.ui, 'part id: "%s"' % pycompat.bytestr(self.id))
23585
94b25d71dd0f bundle2.unbundlepart: decouple mandatory from parttype
Eric Sumner <ericsumner@fb.com>
parents: 23439
diff changeset
  1304
        # extract mandatory bit from type
94b25d71dd0f bundle2.unbundlepart: decouple mandatory from parttype
Eric Sumner <ericsumner@fb.com>
parents: 23439
diff changeset
  1305
        self.mandatory = (self.type != self.type.lower())
94b25d71dd0f bundle2.unbundlepart: decouple mandatory from parttype
Eric Sumner <ericsumner@fb.com>
parents: 23439
diff changeset
  1306
        self.type = self.type.lower()
21014
a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21013
diff changeset
  1307
        ## reading parameters
a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21013
diff changeset
  1308
        # param count
21016
b477afb1c81e bundle2: move unpackheader closure into the class
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21015
diff changeset
  1309
        mancount, advcount = self._unpackheader(_fpartparamcount)
25320
697d8953b04d bundle2: handle new line in 'indebug' function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25319
diff changeset
  1310
        indebug(self.ui, 'part parameters: %i' % (mancount + advcount))
21014
a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21013
diff changeset
  1311
        # param size
21016
b477afb1c81e bundle2: move unpackheader closure into the class
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21015
diff changeset
  1312
        fparamsizes = _makefpartparamsizes(mancount + advcount)
b477afb1c81e bundle2: move unpackheader closure into the class
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21015
diff changeset
  1313
        paramsizes = self._unpackheader(fparamsizes)
21014
a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21013
diff changeset
  1314
        # make it a list of couple again
33700
9a323773216c bundle2: work around zip() being lazy in Python 3
Augie Fackler <augie@google.com>
parents: 33699
diff changeset
  1315
        paramsizes = list(zip(paramsizes[::2], paramsizes[1::2]))
21014
a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21013
diff changeset
  1316
        # split mandatory from advisory
a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21013
diff changeset
  1317
        mansizes = paramsizes[:mancount]
a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21013
diff changeset
  1318
        advsizes = paramsizes[mancount:]
23139
e53f6b72a0e4 spelling: fixes from proofreading of spell checker issues
Mads Kiilerich <madski@unity3d.com>
parents: 23067
diff changeset
  1319
        # retrieve param value
21014
a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21013
diff changeset
  1320
        manparams = []
a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21013
diff changeset
  1321
        for key, value in mansizes:
21015
14dd49260246 bundle2: move the fromheader closure into the class itself
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21014
diff changeset
  1322
            manparams.append((self._fromheader(key), self._fromheader(value)))
21014
a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21013
diff changeset
  1323
        advparams = []
a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21013
diff changeset
  1324
        for key, value in advsizes:
21015
14dd49260246 bundle2: move the fromheader closure into the class itself
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21014
diff changeset
  1325
            advparams.append((self._fromheader(key), self._fromheader(value)))
21608
3cb96ca90c17 bundle2: introduce an ``_initparams`` method
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21607
diff changeset
  1326
        self._initparams(manparams, advparams)
21014
a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21013
diff changeset
  1327
        ## part payload
24034
9881a1437799 bundle2.unbundlepart: raise payloadchunks from a closure to a method
Eric Sumner <ericsumner@fb.com>
parents: 24026
diff changeset
  1328
        self._payloadstream = util.chunkbuffer(self._payloadchunks())
21019
3dc09f831a2e bundle2: lazy unbundle of part payload
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21016
diff changeset
  1329
        # we read the data, tell it
3dc09f831a2e bundle2: lazy unbundle of part payload
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21016
diff changeset
  1330
        self._initialized = True
3dc09f831a2e bundle2: lazy unbundle of part payload
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21016
diff changeset
  1331
35134
8aa43ff9c12c bundle2: implement generic part payload decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35133
diff changeset
  1332
    def _payloadchunks(self):
8aa43ff9c12c bundle2: implement generic part payload decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35133
diff changeset
  1333
        """Generator of decoded chunks in the payload."""
8aa43ff9c12c bundle2: implement generic part payload decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35133
diff changeset
  1334
        return decodepayloadchunks(self.ui, self._fp)
8aa43ff9c12c bundle2: implement generic part payload decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35133
diff changeset
  1335
35135
db5038525718 bundle2: implement consume() API on unbundlepart
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35134
diff changeset
  1336
    def consume(self):
db5038525718 bundle2: implement consume() API on unbundlepart
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35134
diff changeset
  1337
        """Read the part payload until completion.
db5038525718 bundle2: implement consume() API on unbundlepart
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35134
diff changeset
  1338
db5038525718 bundle2: implement consume() API on unbundlepart
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35134
diff changeset
  1339
        By consuming the part data, the underlying stream read offset will
db5038525718 bundle2: implement consume() API on unbundlepart
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35134
diff changeset
  1340
        be advanced to the next part (or end of stream).
db5038525718 bundle2: implement consume() API on unbundlepart
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35134
diff changeset
  1341
        """
db5038525718 bundle2: implement consume() API on unbundlepart
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35134
diff changeset
  1342
        if self.consumed:
db5038525718 bundle2: implement consume() API on unbundlepart
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35134
diff changeset
  1343
            return
db5038525718 bundle2: implement consume() API on unbundlepart
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35134
diff changeset
  1344
db5038525718 bundle2: implement consume() API on unbundlepart
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35134
diff changeset
  1345
        chunk = self.read(32768)
db5038525718 bundle2: implement consume() API on unbundlepart
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35134
diff changeset
  1346
        while chunk:
db5038525718 bundle2: implement consume() API on unbundlepart
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35134
diff changeset
  1347
            self._pos += len(chunk)
db5038525718 bundle2: implement consume() API on unbundlepart
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35134
diff changeset
  1348
            chunk = self.read(32768)
db5038525718 bundle2: implement consume() API on unbundlepart
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35134
diff changeset
  1349
21019
3dc09f831a2e bundle2: lazy unbundle of part payload
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21016
diff changeset
  1350
    def read(self, size=None):
3dc09f831a2e bundle2: lazy unbundle of part payload
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21016
diff changeset
  1351
        """read payload data"""
3dc09f831a2e bundle2: lazy unbundle of part payload
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21016
diff changeset
  1352
        if not self._initialized:
3dc09f831a2e bundle2: lazy unbundle of part payload
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21016
diff changeset
  1353
            self._readheader()
3dc09f831a2e bundle2: lazy unbundle of part payload
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21016
diff changeset
  1354
        if size is None:
3dc09f831a2e bundle2: lazy unbundle of part payload
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21016
diff changeset
  1355
            data = self._payloadstream.read()
3dc09f831a2e bundle2: lazy unbundle of part payload
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21016
diff changeset
  1356
        else:
3dc09f831a2e bundle2: lazy unbundle of part payload
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21016
diff changeset
  1357
            data = self._payloadstream.read(size)
25334
76cba1ecf55f bundle2: add generic debug output regarding processed part payload
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25333
diff changeset
  1358
        self._pos += len(data)
21019
3dc09f831a2e bundle2: lazy unbundle of part payload
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21016
diff changeset
  1359
        if size is None or len(data) < size:
25334
76cba1ecf55f bundle2: add generic debug output regarding processed part payload
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25333
diff changeset
  1360
            if not self.consumed and self._pos:
76cba1ecf55f bundle2: add generic debug output regarding processed part payload
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25333
diff changeset
  1361
                self.ui.debug('bundle2-input-part: total payload size %i\n'
76cba1ecf55f bundle2: add generic debug output regarding processed part payload
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25333
diff changeset
  1362
                              % self._pos)
21019
3dc09f831a2e bundle2: lazy unbundle of part payload
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21016
diff changeset
  1363
            self.consumed = True
3dc09f831a2e bundle2: lazy unbundle of part payload
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21016
diff changeset
  1364
        return data
3dc09f831a2e bundle2: lazy unbundle of part payload
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21016
diff changeset
  1365
35133
073eec083e25 bundle2: extract logic for seeking bundle2 part into own class
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35069
diff changeset
  1366
class seekableunbundlepart(unbundlepart):
073eec083e25 bundle2: extract logic for seeking bundle2 part into own class
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35069
diff changeset
  1367
    """A bundle2 part in a bundle that is seekable.
073eec083e25 bundle2: extract logic for seeking bundle2 part into own class
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35069
diff changeset
  1368
073eec083e25 bundle2: extract logic for seeking bundle2 part into own class
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35069
diff changeset
  1369
    Regular ``unbundlepart`` instances can only be read once. This class
073eec083e25 bundle2: extract logic for seeking bundle2 part into own class
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35069
diff changeset
  1370
    extends ``unbundlepart`` to enable bi-directional seeking within the
073eec083e25 bundle2: extract logic for seeking bundle2 part into own class
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35069
diff changeset
  1371
    part.
073eec083e25 bundle2: extract logic for seeking bundle2 part into own class
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35069
diff changeset
  1372
073eec083e25 bundle2: extract logic for seeking bundle2 part into own class
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35069
diff changeset
  1373
    Bundle2 part data consists of framed chunks. Offsets when seeking
073eec083e25 bundle2: extract logic for seeking bundle2 part into own class
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35069
diff changeset
  1374
    refer to the decoded data, not the offsets in the underlying bundle2
073eec083e25 bundle2: extract logic for seeking bundle2 part into own class
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35069
diff changeset
  1375
    stream.
073eec083e25 bundle2: extract logic for seeking bundle2 part into own class
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35069
diff changeset
  1376
073eec083e25 bundle2: extract logic for seeking bundle2 part into own class
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35069
diff changeset
  1377
    To facilitate quickly seeking within the decoded data, instances of this
073eec083e25 bundle2: extract logic for seeking bundle2 part into own class
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35069
diff changeset
  1378
    class maintain a mapping between offsets in the underlying stream and
073eec083e25 bundle2: extract logic for seeking bundle2 part into own class
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35069
diff changeset
  1379
    the decoded payload. This mapping will consume memory in proportion
073eec083e25 bundle2: extract logic for seeking bundle2 part into own class
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35069
diff changeset
  1380
    to the number of chunks within the payload (which almost certainly
073eec083e25 bundle2: extract logic for seeking bundle2 part into own class
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35069
diff changeset
  1381
    increases in proportion with the size of the part).
073eec083e25 bundle2: extract logic for seeking bundle2 part into own class
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35069
diff changeset
  1382
    """
073eec083e25 bundle2: extract logic for seeking bundle2 part into own class
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35069
diff changeset
  1383
    def __init__(self, ui, header, fp):
073eec083e25 bundle2: extract logic for seeking bundle2 part into own class
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35069
diff changeset
  1384
        # (payload, file) offsets for chunk starts.
073eec083e25 bundle2: extract logic for seeking bundle2 part into own class
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35069
diff changeset
  1385
        self._chunkindex = []
073eec083e25 bundle2: extract logic for seeking bundle2 part into own class
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35069
diff changeset
  1386
073eec083e25 bundle2: extract logic for seeking bundle2 part into own class
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35069
diff changeset
  1387
        super(seekableunbundlepart, self).__init__(ui, header, fp)
073eec083e25 bundle2: extract logic for seeking bundle2 part into own class
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35069
diff changeset
  1388
073eec083e25 bundle2: extract logic for seeking bundle2 part into own class
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35069
diff changeset
  1389
    def _payloadchunks(self, chunknum=0):
073eec083e25 bundle2: extract logic for seeking bundle2 part into own class
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35069
diff changeset
  1390
        '''seek to specified chunk and start yielding data'''
073eec083e25 bundle2: extract logic for seeking bundle2 part into own class
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35069
diff changeset
  1391
        if len(self._chunkindex) == 0:
073eec083e25 bundle2: extract logic for seeking bundle2 part into own class
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35069
diff changeset
  1392
            assert chunknum == 0, 'Must start with chunk 0'
073eec083e25 bundle2: extract logic for seeking bundle2 part into own class
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35069
diff changeset
  1393
            self._chunkindex.append((0, self._tellfp()))
073eec083e25 bundle2: extract logic for seeking bundle2 part into own class
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35069
diff changeset
  1394
        else:
073eec083e25 bundle2: extract logic for seeking bundle2 part into own class
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35069
diff changeset
  1395
            assert chunknum < len(self._chunkindex), \
073eec083e25 bundle2: extract logic for seeking bundle2 part into own class
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35069
diff changeset
  1396
                   'Unknown chunk %d' % chunknum
073eec083e25 bundle2: extract logic for seeking bundle2 part into own class
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35069
diff changeset
  1397
            self._seekfp(self._chunkindex[chunknum][1])
073eec083e25 bundle2: extract logic for seeking bundle2 part into own class
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35069
diff changeset
  1398
073eec083e25 bundle2: extract logic for seeking bundle2 part into own class
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35069
diff changeset
  1399
        pos = self._chunkindex[chunknum][0]
35134
8aa43ff9c12c bundle2: implement generic part payload decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35133
diff changeset
  1400
8aa43ff9c12c bundle2: implement generic part payload decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35133
diff changeset
  1401
        for chunk in decodepayloadchunks(self.ui, self._fp):
8aa43ff9c12c bundle2: implement generic part payload decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35133
diff changeset
  1402
            chunknum += 1
8aa43ff9c12c bundle2: implement generic part payload decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35133
diff changeset
  1403
            pos += len(chunk)
8aa43ff9c12c bundle2: implement generic part payload decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35133
diff changeset
  1404
            if chunknum == len(self._chunkindex):
8aa43ff9c12c bundle2: implement generic part payload decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35133
diff changeset
  1405
                self._chunkindex.append((pos, self._tellfp()))
8aa43ff9c12c bundle2: implement generic part payload decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35133
diff changeset
  1406
8aa43ff9c12c bundle2: implement generic part payload decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35133
diff changeset
  1407
            yield chunk
35133
073eec083e25 bundle2: extract logic for seeking bundle2 part into own class
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35069
diff changeset
  1408
073eec083e25 bundle2: extract logic for seeking bundle2 part into own class
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35069
diff changeset
  1409
    def _findchunk(self, pos):
073eec083e25 bundle2: extract logic for seeking bundle2 part into own class
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35069
diff changeset
  1410
        '''for a given payload position, return a chunk number and offset'''
073eec083e25 bundle2: extract logic for seeking bundle2 part into own class
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35069
diff changeset
  1411
        for chunk, (ppos, fpos) in enumerate(self._chunkindex):
073eec083e25 bundle2: extract logic for seeking bundle2 part into own class
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35069
diff changeset
  1412
            if ppos == pos:
073eec083e25 bundle2: extract logic for seeking bundle2 part into own class
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35069
diff changeset
  1413
                return chunk, 0
073eec083e25 bundle2: extract logic for seeking bundle2 part into own class
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35069
diff changeset
  1414
            elif ppos > pos:
073eec083e25 bundle2: extract logic for seeking bundle2 part into own class
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35069
diff changeset
  1415
                return chunk - 1, pos - self._chunkindex[chunk - 1][0]
073eec083e25 bundle2: extract logic for seeking bundle2 part into own class
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35069
diff changeset
  1416
        raise ValueError('Unknown chunk')
073eec083e25 bundle2: extract logic for seeking bundle2 part into own class
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35069
diff changeset
  1417
24036
c7601086338a bundle2.unbundlepart: tell() implementation
Eric Sumner <ericsumner@fb.com>
parents: 24035
diff changeset
  1418
    def tell(self):
c7601086338a bundle2.unbundlepart: tell() implementation
Eric Sumner <ericsumner@fb.com>
parents: 24035
diff changeset
  1419
        return self._pos
c7601086338a bundle2.unbundlepart: tell() implementation
Eric Sumner <ericsumner@fb.com>
parents: 24035
diff changeset
  1420
35069
241d9caca11e bundle2: use os.SEEK_* constants
Gregory Szorc <gregory.szorc@gmail.com>
parents: 34820
diff changeset
  1421
    def seek(self, offset, whence=os.SEEK_SET):
241d9caca11e bundle2: use os.SEEK_* constants
Gregory Szorc <gregory.szorc@gmail.com>
parents: 34820
diff changeset
  1422
        if whence == os.SEEK_SET:
24037
f0b498cfc5c8 bundle2.unbundlepart: implement seek()
Eric Sumner <ericsumner@fb.com>
parents: 24036
diff changeset
  1423
            newpos = offset
35069
241d9caca11e bundle2: use os.SEEK_* constants
Gregory Szorc <gregory.szorc@gmail.com>
parents: 34820
diff changeset
  1424
        elif whence == os.SEEK_CUR:
24037
f0b498cfc5c8 bundle2.unbundlepart: implement seek()
Eric Sumner <ericsumner@fb.com>
parents: 24036
diff changeset
  1425
            newpos = self._pos + offset
35069
241d9caca11e bundle2: use os.SEEK_* constants
Gregory Szorc <gregory.szorc@gmail.com>
parents: 34820
diff changeset
  1426
        elif whence == os.SEEK_END:
24037
f0b498cfc5c8 bundle2.unbundlepart: implement seek()
Eric Sumner <ericsumner@fb.com>
parents: 24036
diff changeset
  1427
            if not self.consumed:
35141
699b2a759319 bundle2: avoid unbound read when seeking
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35140
diff changeset
  1428
                # Can't use self.consume() here because it advances self._pos.
699b2a759319 bundle2: avoid unbound read when seeking
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35140
diff changeset
  1429
                chunk = self.read(32768)
699b2a759319 bundle2: avoid unbound read when seeking
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35140
diff changeset
  1430
                while chunk:
699b2a759319 bundle2: avoid unbound read when seeking
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35140
diff changeset
  1431
                    chunk = self.read(32768)
24037
f0b498cfc5c8 bundle2.unbundlepart: implement seek()
Eric Sumner <ericsumner@fb.com>
parents: 24036
diff changeset
  1432
            newpos = self._chunkindex[-1][0] - offset
f0b498cfc5c8 bundle2.unbundlepart: implement seek()
Eric Sumner <ericsumner@fb.com>
parents: 24036
diff changeset
  1433
        else:
f0b498cfc5c8 bundle2.unbundlepart: implement seek()
Eric Sumner <ericsumner@fb.com>
parents: 24036
diff changeset
  1434
            raise ValueError('Unknown whence value: %r' % (whence,))
f0b498cfc5c8 bundle2.unbundlepart: implement seek()
Eric Sumner <ericsumner@fb.com>
parents: 24036
diff changeset
  1435
f0b498cfc5c8 bundle2.unbundlepart: implement seek()
Eric Sumner <ericsumner@fb.com>
parents: 24036
diff changeset
  1436
        if newpos > self._chunkindex[-1][0] and not self.consumed:
35141
699b2a759319 bundle2: avoid unbound read when seeking
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35140
diff changeset
  1437
            # Can't use self.consume() here because it advances self._pos.
699b2a759319 bundle2: avoid unbound read when seeking
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35140
diff changeset
  1438
            chunk = self.read(32768)
699b2a759319 bundle2: avoid unbound read when seeking
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35140
diff changeset
  1439
            while chunk:
699b2a759319 bundle2: avoid unbound read when seeking
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35140
diff changeset
  1440
                chunk = self.read(32668)
699b2a759319 bundle2: avoid unbound read when seeking
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35140
diff changeset
  1441
24037
f0b498cfc5c8 bundle2.unbundlepart: implement seek()
Eric Sumner <ericsumner@fb.com>
parents: 24036
diff changeset
  1442
        if not 0 <= newpos <= self._chunkindex[-1][0]:
f0b498cfc5c8 bundle2.unbundlepart: implement seek()
Eric Sumner <ericsumner@fb.com>
parents: 24036
diff changeset
  1443
            raise ValueError('Offset out of range')
f0b498cfc5c8 bundle2.unbundlepart: implement seek()
Eric Sumner <ericsumner@fb.com>
parents: 24036
diff changeset
  1444
f0b498cfc5c8 bundle2.unbundlepart: implement seek()
Eric Sumner <ericsumner@fb.com>
parents: 24036
diff changeset
  1445
        if self._pos != newpos:
f0b498cfc5c8 bundle2.unbundlepart: implement seek()
Eric Sumner <ericsumner@fb.com>
parents: 24036
diff changeset
  1446
            chunk, internaloffset = self._findchunk(newpos)
f0b498cfc5c8 bundle2.unbundlepart: implement seek()
Eric Sumner <ericsumner@fb.com>
parents: 24036
diff changeset
  1447
            self._payloadstream = util.chunkbuffer(self._payloadchunks(chunk))
f0b498cfc5c8 bundle2.unbundlepart: implement seek()
Eric Sumner <ericsumner@fb.com>
parents: 24036
diff changeset
  1448
            adjust = self.read(internaloffset)
f0b498cfc5c8 bundle2.unbundlepart: implement seek()
Eric Sumner <ericsumner@fb.com>
parents: 24036
diff changeset
  1449
            if len(adjust) != internaloffset:
26587
56b2bcea2529 error: get Abort from 'error' instead of 'util'
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26565
diff changeset
  1450
                raise error.Abort(_('Seek failed\n'))
24037
f0b498cfc5c8 bundle2.unbundlepart: implement seek()
Eric Sumner <ericsumner@fb.com>
parents: 24036
diff changeset
  1451
            self._pos = newpos
f0b498cfc5c8 bundle2.unbundlepart: implement seek()
Eric Sumner <ericsumner@fb.com>
parents: 24036
diff changeset
  1452
31889
a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31863
diff changeset
  1453
    def _seekfp(self, offset, whence=0):
a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31863
diff changeset
  1454
        """move the underlying file pointer
a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31863
diff changeset
  1455
a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31863
diff changeset
  1456
        This method is meant for internal usage by the bundle2 protocol only.
a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31863
diff changeset
  1457
        They directly manipulate the low level stream including bundle2 level
a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31863
diff changeset
  1458
        instruction.
a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31863
diff changeset
  1459
a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31863
diff changeset
  1460
        Do not use it to implement higher-level logic or methods."""
a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31863
diff changeset
  1461
        if self._seekable:
a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31863
diff changeset
  1462
            return self._fp.seek(offset, whence)
a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31863
diff changeset
  1463
        else:
a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31863
diff changeset
  1464
            raise NotImplementedError(_('File pointer is not seekable'))
a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31863
diff changeset
  1465
a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31863
diff changeset
  1466
    def _tellfp(self):
a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31863
diff changeset
  1467
        """return the file offset, or None if file is not seekable
a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31863
diff changeset
  1468
a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31863
diff changeset
  1469
        This method is meant for internal usage by the bundle2 protocol only.
a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31863
diff changeset
  1470
        They directly manipulate the low level stream including bundle2 level
a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31863
diff changeset
  1471
        instruction.
a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31863
diff changeset
  1472
a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31863
diff changeset
  1473
        Do not use it to implement higher-level logic or methods."""
a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31863
diff changeset
  1474
        if self._seekable:
a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31863
diff changeset
  1475
            try:
a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31863
diff changeset
  1476
                return self._fp.tell()
a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31863
diff changeset
  1477
            except IOError as e:
a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31863
diff changeset
  1478
                if e.errno == errno.ESPIPE:
a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31863
diff changeset
  1479
                    self._seekable = False
a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31863
diff changeset
  1480
                else:
a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31863
diff changeset
  1481
                    raise
a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31863
diff changeset
  1482
        return None
a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31863
diff changeset
  1483
25317
5a5b7046d00e bundle2: add an informative comment to the capability dict
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25315
diff changeset
  1484
# These are only the static capabilities.
5a5b7046d00e bundle2: add an informative comment to the capability dict
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25315
diff changeset
  1485
# Check the 'getrepocaps' function for the rest.
24686
e0e28e910fa3 bundle2: rename format, parts and config to final names
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24648
diff changeset
  1486
capabilities = {'HG20': (),
35268
ad5f2b923b0d push: include a 'check:bookmarks' part when possible
Boris Feld <boris.feld@octobus.net>
parents: 35267
diff changeset
  1487
                'bookmarks': (),
25493
d8e7b0781ad7 bundle2: convey PushkeyFailed error over the wire
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25492
diff changeset
  1488
                'error': ('abort', 'unsupportedcontent', 'pushraced',
d8e7b0781ad7 bundle2: convey PushkeyFailed error over the wire
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25492
diff changeset
  1489
                          'pushkey'),
24686
e0e28e910fa3 bundle2: rename format, parts and config to final names
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24648
diff changeset
  1490
                'listkeys': (),
e0e28e910fa3 bundle2: rename format, parts and config to final names
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24648
diff changeset
  1491
                'pushkey': (),
23029
149fc8a44184 bundle2: client side support for a part to import external bundles
Mike Hommey <mh@glandium.org>
parents: 23011
diff changeset
  1492
                'digests': tuple(sorted(util.DIGESTS.keys())),
24686
e0e28e910fa3 bundle2: rename format, parts and config to final names
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24648
diff changeset
  1493
                'remote-changegroup': ('http', 'https'),
25401
d29201352af7 bundle2: part handler for processing .hgtags fnodes mappings
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25336
diff changeset
  1494
                'hgtagsfnodes': (),
34329
10e162bb9bf5 pull: use 'phase-heads' to retrieve phase information
Boris Feld <boris.feld@octobus.net>
parents: 34328
diff changeset
  1495
                'phases': ('heads',),
35763
7eedbd5d4880 streamclone: add support for bundle2 based stream clone
Boris Feld <boris.feld@octobus.net>
parents: 35758
diff changeset
  1496
                'stream': ('v2',),
22341
2d16b39601b5 obsmarker: move bundle2caps from the localrepo class to the bundle2 module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 22340
diff changeset
  1497
               }
2d16b39601b5 obsmarker: move bundle2caps from the localrepo class to the bundle2 module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 22340
diff changeset
  1498
35783
c97639ad6874 bundle2: specify what capabilities will be used for
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35763
diff changeset
  1499
def getrepocaps(repo, allowpushback=False, role=None):
22342
262c5cc126c1 bundle2: introduce a `getrepocaps` to retrieve the bundle2 caps of a repo
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 22341
diff changeset
  1500
    """return the bundle2 capabilities for a given repo
262c5cc126c1 bundle2: introduce a `getrepocaps` to retrieve the bundle2 caps of a repo
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 22341
diff changeset
  1501
22343
76677a2c1cfd bundle2: advertise the obsmarker part in bundle2 capabilities
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 22342
diff changeset
  1502
    Exists to allow extensions (like evolution) to mutate the capabilities.
35783
c97639ad6874 bundle2: specify what capabilities will be used for
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35763
diff changeset
  1503
c97639ad6874 bundle2: specify what capabilities will be used for
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35763
diff changeset
  1504
    The returned value is used for servers advertising their capabilities as
c97639ad6874 bundle2: specify what capabilities will be used for
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35763
diff changeset
  1505
    well as clients advertising their capabilities to servers as part of
c97639ad6874 bundle2: specify what capabilities will be used for
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35763
diff changeset
  1506
    bundle2 requests. The ``role`` argument specifies which is which.
22342
262c5cc126c1 bundle2: introduce a `getrepocaps` to retrieve the bundle2 caps of a repo
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 22341
diff changeset
  1507
    """
35783
c97639ad6874 bundle2: specify what capabilities will be used for
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35763
diff changeset
  1508
    if role not in ('client', 'server'):
c97639ad6874 bundle2: specify what capabilities will be used for
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35763
diff changeset
  1509
        raise error.ProgrammingError('role argument must be client or server')
c97639ad6874 bundle2: specify what capabilities will be used for
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35763
diff changeset
  1510
22343
76677a2c1cfd bundle2: advertise the obsmarker part in bundle2 capabilities
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 22342
diff changeset
  1511
    caps = capabilities.copy()
27953
88609cfa3745 changegroup: fix pulling to treemanifest repo from flat repo (issue5066)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27879
diff changeset
  1512
    caps['changegroup'] = tuple(sorted(
88609cfa3745 changegroup: fix pulling to treemanifest repo from flat repo (issue5066)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27879
diff changeset
  1513
        changegroup.supportedincomingversions(repo)))
22953
b1d694d3975e obsolete: add exchange option
Durham Goode <durham@fb.com>
parents: 22661
diff changeset
  1514
    if obsolete.isenabled(repo, obsolete.exchangeopt):
22343
76677a2c1cfd bundle2: advertise the obsmarker part in bundle2 capabilities
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 22342
diff changeset
  1515
        supportedformat = tuple('V%i' % v for v in obsolete.formats)
24686
e0e28e910fa3 bundle2: rename format, parts and config to final names
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24648
diff changeset
  1516
        caps['obsmarkers'] = supportedformat
23439
743736fc7c41 bundle2-push: provide transaction to reply unbundler
Eric Sumner <ericsumner@fb.com>
parents: 23438
diff changeset
  1517
    if allowpushback:
24686
e0e28e910fa3 bundle2: rename format, parts and config to final names
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24648
diff changeset
  1518
        caps['pushback'] = ()
33222
593ad8df9dd2 configitems: register the 'server.concurrent-push-mode' config
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 33165
diff changeset
  1519
    cpmode = repo.ui.config('server', 'concurrent-push-mode')
32910
a7851519ea02 check-concurrency: expose the feature as 'concurrent-push-mode'
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32729
diff changeset
  1520
    if cpmode == 'check-related':
32729
16ada4cbb1a9 push: add a way to allow concurrent pushes on unrelated heads
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32549
diff changeset
  1521
        caps['checkheads'] = ('related',)
34329
10e162bb9bf5 pull: use 'phase-heads' to retrieve phase information
Boris Feld <boris.feld@octobus.net>
parents: 34328
diff changeset
  1522
    if 'phases' in repo.ui.configlist('devel', 'legacy.exchange'):
10e162bb9bf5 pull: use 'phase-heads' to retrieve phase information
Boris Feld <boris.feld@octobus.net>
parents: 34328
diff changeset
  1523
        caps.pop('phases')
35790
c1b9eb15a51c bundle2: don't advertise stream bundle2 capability when feature disabled
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35788
diff changeset
  1524
c1b9eb15a51c bundle2: don't advertise stream bundle2 capability when feature disabled
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35788
diff changeset
  1525
    # Don't advertise stream clone support in server mode if not configured.
c1b9eb15a51c bundle2: don't advertise stream bundle2 capability when feature disabled
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35788
diff changeset
  1526
    if role == 'server':
c1b9eb15a51c bundle2: don't advertise stream bundle2 capability when feature disabled
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35788
diff changeset
  1527
        streamsupported = repo.ui.configbool('server', 'uncompressed',
c1b9eb15a51c bundle2: don't advertise stream bundle2 capability when feature disabled
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35788
diff changeset
  1528
                                             untrusted=True)
c1b9eb15a51c bundle2: don't advertise stream bundle2 capability when feature disabled
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35788
diff changeset
  1529
        featuresupported = repo.ui.configbool('experimental', 'bundle2.stream')
c1b9eb15a51c bundle2: don't advertise stream bundle2 capability when feature disabled
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35788
diff changeset
  1530
c1b9eb15a51c bundle2: don't advertise stream bundle2 capability when feature disabled
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35788
diff changeset
  1531
        if not streamsupported or not featuresupported:
c1b9eb15a51c bundle2: don't advertise stream bundle2 capability when feature disabled
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35788
diff changeset
  1532
            caps.pop('stream')
35792
1d118f9f4f57 bundle2: always advertise client support for stream parts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35790
diff changeset
  1533
    # Else always advertise support on client, because payload support
1d118f9f4f57 bundle2: always advertise client support for stream parts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35790
diff changeset
  1534
    # should always be advertised.
35790
c1b9eb15a51c bundle2: don't advertise stream bundle2 capability when feature disabled
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35788
diff changeset
  1535
22343
76677a2c1cfd bundle2: advertise the obsmarker part in bundle2 capabilities
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 22342
diff changeset
  1536
    return caps
22342
262c5cc126c1 bundle2: introduce a `getrepocaps` to retrieve the bundle2 caps of a repo
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 22341
diff changeset
  1537
21644
17755dd8c509 bundle2: introduce a bundle2caps function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21628
diff changeset
  1538
def bundle2caps(remote):
23139
e53f6b72a0e4 spelling: fixes from proofreading of spell checker issues
Mads Kiilerich <madski@unity3d.com>
parents: 23067
diff changeset
  1539
    """return the bundle capabilities of a peer as dict"""
24686
e0e28e910fa3 bundle2: rename format, parts and config to final names
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24648
diff changeset
  1540
    raw = remote.capable('bundle2')
21644
17755dd8c509 bundle2: introduce a bundle2caps function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21628
diff changeset
  1541
    if not raw and raw != '':
17755dd8c509 bundle2: introduce a bundle2caps function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21628
diff changeset
  1542
        return {}
28883
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents: 28672
diff changeset
  1543
    capsblob = urlreq.unquote(remote.capable('bundle2'))
21644
17755dd8c509 bundle2: introduce a bundle2caps function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21628
diff changeset
  1544
    return decodecaps(capsblob)
21014
a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21013
diff changeset
  1545
22344
9829b7948100 bundle2: add a `obsmarkersversion` function to extract supported version
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 22343
diff changeset
  1546
def obsmarkersversion(caps):
9829b7948100 bundle2: add a `obsmarkersversion` function to extract supported version
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 22343
diff changeset
  1547
    """extract the list of supported obsmarkers versions from a bundle2caps dict
9829b7948100 bundle2: add a `obsmarkersversion` function to extract supported version
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 22343
diff changeset
  1548
    """
24686
e0e28e910fa3 bundle2: rename format, parts and config to final names
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24648
diff changeset
  1549
    obscaps = caps.get('obsmarkers', ())
22344
9829b7948100 bundle2: add a `obsmarkersversion` function to extract supported version
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 22343
diff changeset
  1550
    return [int(c[1:]) for c in obscaps if c.startswith('V')]
9829b7948100 bundle2: add a `obsmarkersversion` function to extract supported version
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 22343
diff changeset
  1551
32261
9dc36df78403 bundle: introduce an higher level function to write bundle on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32232
diff changeset
  1552
def writenewbundle(ui, repo, source, filename, bundletype, outgoing, opts,
9dc36df78403 bundle: introduce an higher level function to write bundle on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32232
diff changeset
  1553
                   vfs=None, compression=None, compopts=None):
9dc36df78403 bundle: introduce an higher level function to write bundle on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32232
diff changeset
  1554
    if bundletype.startswith('HG10'):
34116
5ede882c249c changegroup: replace getchangegroup with makechangegroup
Durham Goode <durham@fb.com>
parents: 33906
diff changeset
  1555
        cg = changegroup.makechangegroup(repo, outgoing, '01', source)
32261
9dc36df78403 bundle: introduce an higher level function to write bundle on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32232
diff changeset
  1556
        return writebundle(ui, cg, filename, bundletype, vfs=vfs,
9dc36df78403 bundle: introduce an higher level function to write bundle on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32232
diff changeset
  1557
                           compression=compression, compopts=compopts)
9dc36df78403 bundle: introduce an higher level function to write bundle on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32232
diff changeset
  1558
    elif not bundletype.startswith('HG20'):
9dc36df78403 bundle: introduce an higher level function to write bundle on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32232
diff changeset
  1559
        raise error.ProgrammingError('unknown bundle type: %s' % bundletype)
9dc36df78403 bundle: introduce an higher level function to write bundle on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32232
diff changeset
  1560
32549
37d70ba1d9d1 bundle: add an experimental knob to include obsmarkers in bundle
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32548
diff changeset
  1561
    caps = {}
37d70ba1d9d1 bundle: add an experimental knob to include obsmarkers in bundle
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32548
diff changeset
  1562
    if 'obsolescence' in opts:
37d70ba1d9d1 bundle: add an experimental knob to include obsmarkers in bundle
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32548
diff changeset
  1563
        caps['obsmarkers'] = ('V1',)
37d70ba1d9d1 bundle: add an experimental knob to include obsmarkers in bundle
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32548
diff changeset
  1564
    bundle = bundle20(ui, caps)
32261
9dc36df78403 bundle: introduce an higher level function to write bundle on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32232
diff changeset
  1565
    bundle.setcompression(compression, compopts)
9dc36df78403 bundle: introduce an higher level function to write bundle on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32232
diff changeset
  1566
    _addpartsfromopts(ui, repo, bundle, source, outgoing, opts)
9dc36df78403 bundle: introduce an higher level function to write bundle on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32232
diff changeset
  1567
    chunkiter = bundle.getchunks()
9dc36df78403 bundle: introduce an higher level function to write bundle on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32232
diff changeset
  1568
9dc36df78403 bundle: introduce an higher level function to write bundle on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32232
diff changeset
  1569
    return changegroup.writechunks(ui, chunkiter, filename, vfs=vfs)
9dc36df78403 bundle: introduce an higher level function to write bundle on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32232
diff changeset
  1570
9dc36df78403 bundle: introduce an higher level function to write bundle on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32232
diff changeset
  1571
def _addpartsfromopts(ui, repo, bundler, source, outgoing, opts):
9dc36df78403 bundle: introduce an higher level function to write bundle on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32232
diff changeset
  1572
    # We should eventually reconcile this logic with the one behind
9dc36df78403 bundle: introduce an higher level function to write bundle on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32232
diff changeset
  1573
    # 'exchange.getbundle2partsgenerator'.
9dc36df78403 bundle: introduce an higher level function to write bundle on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32232
diff changeset
  1574
    #
9dc36df78403 bundle: introduce an higher level function to write bundle on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32232
diff changeset
  1575
    # The type of input from 'getbundle' and 'writenewbundle' are a bit
9dc36df78403 bundle: introduce an higher level function to write bundle on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32232
diff changeset
  1576
    # different right now. So we keep them separated for now for the sake of
9dc36df78403 bundle: introduce an higher level function to write bundle on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32232
diff changeset
  1577
    # simplicity.
9dc36df78403 bundle: introduce an higher level function to write bundle on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32232
diff changeset
  1578
9dc36df78403 bundle: introduce an higher level function to write bundle on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32232
diff changeset
  1579
    # we always want a changegroup in such bundle
9dc36df78403 bundle: introduce an higher level function to write bundle on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32232
diff changeset
  1580
    cgversion = opts.get('cg.version')
9dc36df78403 bundle: introduce an higher level function to write bundle on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32232
diff changeset
  1581
    if cgversion is None:
9dc36df78403 bundle: introduce an higher level function to write bundle on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32232
diff changeset
  1582
        cgversion = changegroup.safeversion(repo)
34116
5ede882c249c changegroup: replace getchangegroup with makechangegroup
Durham Goode <durham@fb.com>
parents: 33906
diff changeset
  1583
    cg = changegroup.makechangegroup(repo, outgoing, cgversion, source)
32261
9dc36df78403 bundle: introduce an higher level function to write bundle on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32232
diff changeset
  1584
    part = bundler.newpart('changegroup', data=cg.getchunks())
9dc36df78403 bundle: introduce an higher level function to write bundle on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32232
diff changeset
  1585
    part.addparam('version', cg.version)
9dc36df78403 bundle: introduce an higher level function to write bundle on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32232
diff changeset
  1586
    if 'clcount' in cg.extras:
34228
8e0358024a36 bundles: turn nbchanges int into a bytestr using pycompat.bytestr
Augie Fackler <raf@durin42.com>
parents: 34164
diff changeset
  1587
        part.addparam('nbchanges', '%d' % cg.extras['clcount'],
32261
9dc36df78403 bundle: introduce an higher level function to write bundle on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32232
diff changeset
  1588
                      mandatory=False)
33408
e3867c712d51 bundle2: automatically add 'targetphase' parameter in writenewbundle
Boris Feld <boris.feld@octobus.net>
parents: 33407
diff changeset
  1589
    if opts.get('phases') and repo.revs('%ln and secret()',
e3867c712d51 bundle2: automatically add 'targetphase' parameter in writenewbundle
Boris Feld <boris.feld@octobus.net>
parents: 33407
diff changeset
  1590
                                        outgoing.missingheads):
e3867c712d51 bundle2: automatically add 'targetphase' parameter in writenewbundle
Boris Feld <boris.feld@octobus.net>
parents: 33407
diff changeset
  1591
        part.addparam('targetphase', '%d' % phases.secret, mandatory=False)
32261
9dc36df78403 bundle: introduce an higher level function to write bundle on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32232
diff changeset
  1592
32263
d7f93ebbbbdf bundle: add optional 'tagsfnodecache' data to on disk bundle (issue5543)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32262
diff changeset
  1593
    addparttagsfnodescache(repo, bundler, outgoing)
d7f93ebbbbdf bundle: add optional 'tagsfnodecache' data to on disk bundle (issue5543)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32262
diff changeset
  1594
32549
37d70ba1d9d1 bundle: add an experimental knob to include obsmarkers in bundle
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32548
diff changeset
  1595
    if opts.get('obsolescence', False):
37d70ba1d9d1 bundle: add an experimental knob to include obsmarkers in bundle
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32548
diff changeset
  1596
        obsmarkers = repo.obsstore.relevantmarkers(outgoing.missing)
37d70ba1d9d1 bundle: add an experimental knob to include obsmarkers in bundle
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32548
diff changeset
  1597
        buildobsmarkerspart(bundler, obsmarkers)
37d70ba1d9d1 bundle: add an experimental knob to include obsmarkers in bundle
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32548
diff changeset
  1598
33043
e8c8d81eb864 bundle: add config option to include phases
Martin von Zweigbergk <martinvonz@google.com>
parents: 33042
diff changeset
  1599
    if opts.get('phases', False):
e8c8d81eb864 bundle: add config option to include phases
Martin von Zweigbergk <martinvonz@google.com>
parents: 33042
diff changeset
  1600
        headsbyphase = phases.subsetphaseheads(repo, outgoing.missing)
34326
5779d096a696 phases: move binary encoding into a reusable function
Boris Feld <boris.feld@octobus.net>
parents: 34325
diff changeset
  1601
        phasedata = phases.binaryencode(headsbyphase)
5779d096a696 phases: move binary encoding into a reusable function
Boris Feld <boris.feld@octobus.net>
parents: 34325
diff changeset
  1602
        bundler.newpart('phase-heads', data=phasedata)
33043
e8c8d81eb864 bundle: add config option to include phases
Martin von Zweigbergk <martinvonz@google.com>
parents: 33042
diff changeset
  1603
32262
6068712cbf03 bundle2: move tagsfnodecache generation in a generic function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32261
diff changeset
  1604
def addparttagsfnodescache(repo, bundler, outgoing):
6068712cbf03 bundle2: move tagsfnodecache generation in a generic function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32261
diff changeset
  1605
    # we include the tags fnode cache for the bundle changeset
6068712cbf03 bundle2: move tagsfnodecache generation in a generic function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32261
diff changeset
  1606
    # (as an optional parts)
6068712cbf03 bundle2: move tagsfnodecache generation in a generic function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32261
diff changeset
  1607
    cache = tags.hgtagsfnodescache(repo.unfiltered())
6068712cbf03 bundle2: move tagsfnodecache generation in a generic function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32261
diff changeset
  1608
    chunks = []
6068712cbf03 bundle2: move tagsfnodecache generation in a generic function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32261
diff changeset
  1609
6068712cbf03 bundle2: move tagsfnodecache generation in a generic function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32261
diff changeset
  1610
    # .hgtags fnodes are only relevant for head changesets. While we could
6068712cbf03 bundle2: move tagsfnodecache generation in a generic function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32261
diff changeset
  1611
    # transfer values for all known nodes, there will likely be little to
6068712cbf03 bundle2: move tagsfnodecache generation in a generic function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32261
diff changeset
  1612
    # no benefit.
6068712cbf03 bundle2: move tagsfnodecache generation in a generic function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32261
diff changeset
  1613
    #
6068712cbf03 bundle2: move tagsfnodecache generation in a generic function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32261
diff changeset
  1614
    # We don't bother using a generator to produce output data because
6068712cbf03 bundle2: move tagsfnodecache generation in a generic function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32261
diff changeset
  1615
    # a) we only have 40 bytes per head and even esoteric numbers of heads
6068712cbf03 bundle2: move tagsfnodecache generation in a generic function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32261
diff changeset
  1616
    # consume little memory (1M heads is 40MB) b) we don't want to send the
6068712cbf03 bundle2: move tagsfnodecache generation in a generic function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32261
diff changeset
  1617
    # part if we don't have entries and knowing if we have entries requires
6068712cbf03 bundle2: move tagsfnodecache generation in a generic function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32261
diff changeset
  1618
    # cache lookups.
6068712cbf03 bundle2: move tagsfnodecache generation in a generic function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32261
diff changeset
  1619
    for node in outgoing.missingheads:
6068712cbf03 bundle2: move tagsfnodecache generation in a generic function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32261
diff changeset
  1620
        # Don't compute missing, as this may slow down serving.
6068712cbf03 bundle2: move tagsfnodecache generation in a generic function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32261
diff changeset
  1621
        fnode = cache.getfnode(node, computemissing=False)
6068712cbf03 bundle2: move tagsfnodecache generation in a generic function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32261
diff changeset
  1622
        if fnode is not None:
6068712cbf03 bundle2: move tagsfnodecache generation in a generic function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32261
diff changeset
  1623
            chunks.extend([node, fnode])
6068712cbf03 bundle2: move tagsfnodecache generation in a generic function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32261
diff changeset
  1624
6068712cbf03 bundle2: move tagsfnodecache generation in a generic function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32261
diff changeset
  1625
    if chunks:
6068712cbf03 bundle2: move tagsfnodecache generation in a generic function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32261
diff changeset
  1626
        bundler.newpart('hgtagsfnodes', data=''.join(chunks))
6068712cbf03 bundle2: move tagsfnodecache generation in a generic function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32261
diff changeset
  1627
36970
79b73be4dda5 rev-branch-cache: add a function to generate a part
Boris Feld <boris.feld@octobus.net>
parents: 36969
diff changeset
  1628
def addpartrevbranchcache(repo, bundler, outgoing):
79b73be4dda5 rev-branch-cache: add a function to generate a part
Boris Feld <boris.feld@octobus.net>
parents: 36969
diff changeset
  1629
    # we include the rev branch cache for the bundle changeset
79b73be4dda5 rev-branch-cache: add a function to generate a part
Boris Feld <boris.feld@octobus.net>
parents: 36969
diff changeset
  1630
    # (as an optional parts)
79b73be4dda5 rev-branch-cache: add a function to generate a part
Boris Feld <boris.feld@octobus.net>
parents: 36969
diff changeset
  1631
    cache = repo.revbranchcache()
79b73be4dda5 rev-branch-cache: add a function to generate a part
Boris Feld <boris.feld@octobus.net>
parents: 36969
diff changeset
  1632
    cl = repo.unfiltered().changelog
79b73be4dda5 rev-branch-cache: add a function to generate a part
Boris Feld <boris.feld@octobus.net>
parents: 36969
diff changeset
  1633
    branchesdata = collections.defaultdict(lambda: (set(), set()))
79b73be4dda5 rev-branch-cache: add a function to generate a part
Boris Feld <boris.feld@octobus.net>
parents: 36969
diff changeset
  1634
    for node in outgoing.missing:
79b73be4dda5 rev-branch-cache: add a function to generate a part
Boris Feld <boris.feld@octobus.net>
parents: 36969
diff changeset
  1635
        branch, close = cache.branchinfo(cl.rev(node))
79b73be4dda5 rev-branch-cache: add a function to generate a part
Boris Feld <boris.feld@octobus.net>
parents: 36969
diff changeset
  1636
        branchesdata[branch][close].add(node)
79b73be4dda5 rev-branch-cache: add a function to generate a part
Boris Feld <boris.feld@octobus.net>
parents: 36969
diff changeset
  1637
79b73be4dda5 rev-branch-cache: add a function to generate a part
Boris Feld <boris.feld@octobus.net>
parents: 36969
diff changeset
  1638
    def generate():
79b73be4dda5 rev-branch-cache: add a function to generate a part
Boris Feld <boris.feld@octobus.net>
parents: 36969
diff changeset
  1639
        for branch, (nodes, closed) in sorted(branchesdata.items()):
79b73be4dda5 rev-branch-cache: add a function to generate a part
Boris Feld <boris.feld@octobus.net>
parents: 36969
diff changeset
  1640
            utf8branch = encoding.fromlocal(branch)
79b73be4dda5 rev-branch-cache: add a function to generate a part
Boris Feld <boris.feld@octobus.net>
parents: 36969
diff changeset
  1641
            yield rbcstruct.pack(len(utf8branch), len(nodes), len(closed))
79b73be4dda5 rev-branch-cache: add a function to generate a part
Boris Feld <boris.feld@octobus.net>
parents: 36969
diff changeset
  1642
            yield utf8branch
79b73be4dda5 rev-branch-cache: add a function to generate a part
Boris Feld <boris.feld@octobus.net>
parents: 36969
diff changeset
  1643
            for n in sorted(nodes):
79b73be4dda5 rev-branch-cache: add a function to generate a part
Boris Feld <boris.feld@octobus.net>
parents: 36969
diff changeset
  1644
                yield n
79b73be4dda5 rev-branch-cache: add a function to generate a part
Boris Feld <boris.feld@octobus.net>
parents: 36969
diff changeset
  1645
            for n in sorted(closed):
79b73be4dda5 rev-branch-cache: add a function to generate a part
Boris Feld <boris.feld@octobus.net>
parents: 36969
diff changeset
  1646
                yield n
79b73be4dda5 rev-branch-cache: add a function to generate a part
Boris Feld <boris.feld@octobus.net>
parents: 36969
diff changeset
  1647
79b73be4dda5 rev-branch-cache: add a function to generate a part
Boris Feld <boris.feld@octobus.net>
parents: 36969
diff changeset
  1648
    bundler.newpart('cache:rev-branch-cache', data=generate())
79b73be4dda5 rev-branch-cache: add a function to generate a part
Boris Feld <boris.feld@octobus.net>
parents: 36969
diff changeset
  1649
32548
e70d6dbde713 bundle2: move function building obsmarker-part in the bundle2 module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32425
diff changeset
  1650
def buildobsmarkerspart(bundler, markers):
e70d6dbde713 bundle2: move function building obsmarker-part in the bundle2 module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32425
diff changeset
  1651
    """add an obsmarker part to the bundler with <markers>
e70d6dbde713 bundle2: move function building obsmarker-part in the bundle2 module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32425
diff changeset
  1652
e70d6dbde713 bundle2: move function building obsmarker-part in the bundle2 module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32425
diff changeset
  1653
    No part is created if markers is empty.
e70d6dbde713 bundle2: move function building obsmarker-part in the bundle2 module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32425
diff changeset
  1654
    Raises ValueError if the bundler doesn't support any known obsmarker format.
e70d6dbde713 bundle2: move function building obsmarker-part in the bundle2 module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32425
diff changeset
  1655
    """
e70d6dbde713 bundle2: move function building obsmarker-part in the bundle2 module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32425
diff changeset
  1656
    if not markers:
e70d6dbde713 bundle2: move function building obsmarker-part in the bundle2 module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32425
diff changeset
  1657
        return None
e70d6dbde713 bundle2: move function building obsmarker-part in the bundle2 module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32425
diff changeset
  1658
e70d6dbde713 bundle2: move function building obsmarker-part in the bundle2 module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32425
diff changeset
  1659
    remoteversions = obsmarkersversion(bundler.capabilities)
e70d6dbde713 bundle2: move function building obsmarker-part in the bundle2 module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32425
diff changeset
  1660
    version = obsolete.commonversion(remoteversions)
e70d6dbde713 bundle2: move function building obsmarker-part in the bundle2 module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32425
diff changeset
  1661
    if version is None:
e70d6dbde713 bundle2: move function building obsmarker-part in the bundle2 module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32425
diff changeset
  1662
        raise ValueError('bundler does not support common obsmarker format')
e70d6dbde713 bundle2: move function building obsmarker-part in the bundle2 module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32425
diff changeset
  1663
    stream = obsolete.encodemarkers(markers, True, version=version)
e70d6dbde713 bundle2: move function building obsmarker-part in the bundle2 module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32425
diff changeset
  1664
    return bundler.newpart('obsmarkers', data=stream)
e70d6dbde713 bundle2: move function building obsmarker-part in the bundle2 module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32425
diff changeset
  1665
30757
511a4bf52754 bundle2: allow compression options to be passed to compressor
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30448
diff changeset
  1666
def writebundle(ui, cg, filename, bundletype, vfs=None, compression=None,
511a4bf52754 bundle2: allow compression options to be passed to compressor
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30448
diff changeset
  1667
                compopts=None):
28666
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27953
diff changeset
  1668
    """Write a bundle file and return its filename.
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27953
diff changeset
  1669
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27953
diff changeset
  1670
    Existing files will not be overwritten.
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27953
diff changeset
  1671
    If no filename is specified, a temporary file is created.
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27953
diff changeset
  1672
    bz2 compression can be turned off.
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27953
diff changeset
  1673
    The bundle file will be deleted in case of errors.
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27953
diff changeset
  1674
    """
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27953
diff changeset
  1675
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27953
diff changeset
  1676
    if bundletype == "HG20":
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27953
diff changeset
  1677
        bundle = bundle20(ui)
30757
511a4bf52754 bundle2: allow compression options to be passed to compressor
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30448
diff changeset
  1678
        bundle.setcompression(compression, compopts)
28666
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27953
diff changeset
  1679
        part = bundle.newpart('changegroup', data=cg.getchunks())
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27953
diff changeset
  1680
        part.addparam('version', cg.version)
29593
953839de96ab bundle2: store changeset count when creating file bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29591
diff changeset
  1681
        if 'clcount' in cg.extras:
34228
8e0358024a36 bundles: turn nbchanges int into a bytestr using pycompat.bytestr
Augie Fackler <raf@durin42.com>
parents: 34164
diff changeset
  1682
            part.addparam('nbchanges', '%d' % cg.extras['clcount'],
29593
953839de96ab bundle2: store changeset count when creating file bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29591
diff changeset
  1683
                          mandatory=False)
28666
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27953
diff changeset
  1684
        chunkiter = bundle.getchunks()
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27953
diff changeset
  1685
    else:
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27953
diff changeset
  1686
        # compression argument is only for the bundle2 case
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27953
diff changeset
  1687
        assert compression is None
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27953
diff changeset
  1688
        if cg.version != '01':
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27953
diff changeset
  1689
            raise error.Abort(_('old bundle types only supports v1 '
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27953
diff changeset
  1690
                                'changegroups'))
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27953
diff changeset
  1691
        header, comp = bundletypes[bundletype]
30361
f81002f736d7 bundle2: use new compression engine API for compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30177
diff changeset
  1692
        if comp not in util.compengines.supportedbundletypes:
28666
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27953
diff changeset
  1693
            raise error.Abort(_('unknown stream compression type: %s')
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27953
diff changeset
  1694
                              % comp)
30361
f81002f736d7 bundle2: use new compression engine API for compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30177
diff changeset
  1695
        compengine = util.compengines.forbundletype(comp)
28666
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27953
diff changeset
  1696
        def chunkiter():
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27953
diff changeset
  1697
            yield header
30757
511a4bf52754 bundle2: allow compression options to be passed to compressor
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30448
diff changeset
  1698
            for chunk in compengine.compressstream(cg.getchunks(), compopts):
30367
5925bda42dbd bundle2: use compressstream compression engine API
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30363
diff changeset
  1699
                yield chunk
28666
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27953
diff changeset
  1700
        chunkiter = chunkiter()
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27953
diff changeset
  1701
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27953
diff changeset
  1702
    # parse the changegroup data, otherwise we will block
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27953
diff changeset
  1703
    # in case of sshrepo because we don't know the end of the stream
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27953
diff changeset
  1704
    return changegroup.writechunks(ui, chunkiter, filename, vfs=vfs)
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27953
diff changeset
  1705
33049
d765ad56081f bundle: make combinechangegroupresults() take a bundleoperation
Martin von Zweigbergk <martinvonz@google.com>
parents: 33048
diff changeset
  1706
def combinechangegroupresults(op):
33048
52c7060b707a bundle: move combineresults() from changegroup to bundle2
Martin von Zweigbergk <martinvonz@google.com>
parents: 33047
diff changeset
  1707
    """logic to combine 0 or more addchangegroup results into one"""
33049
d765ad56081f bundle: make combinechangegroupresults() take a bundleoperation
Martin von Zweigbergk <martinvonz@google.com>
parents: 33048
diff changeset
  1708
    results = [r.get('return', 0)
d765ad56081f bundle: make combinechangegroupresults() take a bundleoperation
Martin von Zweigbergk <martinvonz@google.com>
parents: 33048
diff changeset
  1709
               for r in op.records['changegroup']]
33048
52c7060b707a bundle: move combineresults() from changegroup to bundle2
Martin von Zweigbergk <martinvonz@google.com>
parents: 33047
diff changeset
  1710
    changedheads = 0
52c7060b707a bundle: move combineresults() from changegroup to bundle2
Martin von Zweigbergk <martinvonz@google.com>
parents: 33047
diff changeset
  1711
    result = 1
52c7060b707a bundle: move combineresults() from changegroup to bundle2
Martin von Zweigbergk <martinvonz@google.com>
parents: 33047
diff changeset
  1712
    for ret in results:
52c7060b707a bundle: move combineresults() from changegroup to bundle2
Martin von Zweigbergk <martinvonz@google.com>
parents: 33047
diff changeset
  1713
        # If any changegroup result is 0, return 0
52c7060b707a bundle: move combineresults() from changegroup to bundle2
Martin von Zweigbergk <martinvonz@google.com>
parents: 33047
diff changeset
  1714
        if ret == 0:
52c7060b707a bundle: move combineresults() from changegroup to bundle2
Martin von Zweigbergk <martinvonz@google.com>
parents: 33047
diff changeset
  1715
            result = 0
52c7060b707a bundle: move combineresults() from changegroup to bundle2
Martin von Zweigbergk <martinvonz@google.com>
parents: 33047
diff changeset
  1716
            break
52c7060b707a bundle: move combineresults() from changegroup to bundle2
Martin von Zweigbergk <martinvonz@google.com>
parents: 33047
diff changeset
  1717
        if ret < -1:
52c7060b707a bundle: move combineresults() from changegroup to bundle2
Martin von Zweigbergk <martinvonz@google.com>
parents: 33047
diff changeset
  1718
            changedheads += ret + 1
52c7060b707a bundle: move combineresults() from changegroup to bundle2
Martin von Zweigbergk <martinvonz@google.com>
parents: 33047
diff changeset
  1719
        elif ret > 1:
52c7060b707a bundle: move combineresults() from changegroup to bundle2
Martin von Zweigbergk <martinvonz@google.com>
parents: 33047
diff changeset
  1720
            changedheads += ret - 1
52c7060b707a bundle: move combineresults() from changegroup to bundle2
Martin von Zweigbergk <martinvonz@google.com>
parents: 33047
diff changeset
  1721
    if changedheads > 0:
52c7060b707a bundle: move combineresults() from changegroup to bundle2
Martin von Zweigbergk <martinvonz@google.com>
parents: 33047
diff changeset
  1722
        result = 1 + changedheads
52c7060b707a bundle: move combineresults() from changegroup to bundle2
Martin von Zweigbergk <martinvonz@google.com>
parents: 33047
diff changeset
  1723
    elif changedheads < 0:
52c7060b707a bundle: move combineresults() from changegroup to bundle2
Martin von Zweigbergk <martinvonz@google.com>
parents: 33047
diff changeset
  1724
        result = -1 + changedheads
52c7060b707a bundle: move combineresults() from changegroup to bundle2
Martin von Zweigbergk <martinvonz@google.com>
parents: 33047
diff changeset
  1725
    return result
52c7060b707a bundle: move combineresults() from changegroup to bundle2
Martin von Zweigbergk <martinvonz@google.com>
parents: 33047
diff changeset
  1726
33407
39d4e5a66f5a bundle2: support the 'targetphase' parameter for the changegroup part
Boris Feld <boris.feld@octobus.net>
parents: 33252
diff changeset
  1727
@parthandler('changegroup', ('version', 'nbchanges', 'treemanifest',
39d4e5a66f5a bundle2: support the 'targetphase' parameter for the changegroup part
Boris Feld <boris.feld@octobus.net>
parents: 33252
diff changeset
  1728
                             'targetphase'))
20998
93a3c5b58635 bundle2: use reply part to return result of addchangegroup
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20997
diff changeset
  1729
def handlechangegroup(op, inpart):
20950
c7ceae0faf69 bundle2: first crude version of bundling changeset with bundle2
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20949
diff changeset
  1730
    """apply a changegroup part on the repo
c7ceae0faf69 bundle2: first crude version of bundling changeset with bundle2
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20949
diff changeset
  1731
c7ceae0faf69 bundle2: first crude version of bundling changeset with bundle2
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20949
diff changeset
  1732
    This is a very early implementation that will massive rework before being
c7ceae0faf69 bundle2: first crude version of bundling changeset with bundle2
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20949
diff changeset
  1733
    inflicted to any end-user.
c7ceae0faf69 bundle2: first crude version of bundling changeset with bundle2
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20949
diff changeset
  1734
    """
32948
af31d531dda0 changegroup: let callers pass in transaction to apply() (API)
Martin von Zweigbergk <martinvonz@google.com>
parents: 32910
diff changeset
  1735
    tr = op.gettransaction()
23170
02e8f9b60052 bundle2: support a "version" argument in `changegroup` part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23169
diff changeset
  1736
    unpackerversion = inpart.params.get('version', '01')
02e8f9b60052 bundle2: support a "version" argument in `changegroup` part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23169
diff changeset
  1737
    # We should raise an appropriate exception here
27751
a40e2f7fe49d changegroup: hide packermap behind methods
Martin von Zweigbergk <martinvonz@google.com>
parents: 27734
diff changeset
  1738
    cg = changegroup.getunbundler(unpackerversion, inpart, None)
23001
4df9b5e62f70 bundle2: add a comment about addchangegroup source and url
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 22953
diff changeset
  1739
    # the source and url passed here are overwritten by the one contained in
4df9b5e62f70 bundle2: add a comment about addchangegroup source and url
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 22953
diff changeset
  1740
    # the transaction.hookargs argument. So 'bundle2' is a placeholder
25518
ca656f3dffd7 bundle2: provide number of changesets information to 'addchangegroup'
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25507
diff changeset
  1741
    nbchangesets = None
ca656f3dffd7 bundle2: provide number of changesets information to 'addchangegroup'
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25507
diff changeset
  1742
    if 'nbchanges' in inpart.params:
ca656f3dffd7 bundle2: provide number of changesets information to 'addchangegroup'
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25507
diff changeset
  1743
        nbchangesets = int(inpart.params.get('nbchanges'))
27734
5c0fd878779c treemanifests: set bundle2 part parameter indicating treemanifest
Martin von Zweigbergk <martinvonz@google.com>
parents: 26829
diff changeset
  1744
    if ('treemanifest' in inpart.params and
5c0fd878779c treemanifests: set bundle2 part parameter indicating treemanifest
Martin von Zweigbergk <martinvonz@google.com>
parents: 26829
diff changeset
  1745
        'treemanifest' not in op.repo.requirements):
5c0fd878779c treemanifests: set bundle2 part parameter indicating treemanifest
Martin von Zweigbergk <martinvonz@google.com>
parents: 26829
diff changeset
  1746
        if len(op.repo.changelog) != 0:
5c0fd878779c treemanifests: set bundle2 part parameter indicating treemanifest
Martin von Zweigbergk <martinvonz@google.com>
parents: 26829
diff changeset
  1747
            raise error.Abort(_(
5c0fd878779c treemanifests: set bundle2 part parameter indicating treemanifest
Martin von Zweigbergk <martinvonz@google.com>
parents: 26829
diff changeset
  1748
                "bundle contains tree manifests, but local repo is "
5c0fd878779c treemanifests: set bundle2 part parameter indicating treemanifest
Martin von Zweigbergk <martinvonz@google.com>
parents: 26829
diff changeset
  1749
                "non-empty and does not use tree manifests"))
5c0fd878779c treemanifests: set bundle2 part parameter indicating treemanifest
Martin von Zweigbergk <martinvonz@google.com>
parents: 26829
diff changeset
  1750
        op.repo.requirements.add('treemanifest')
5c0fd878779c treemanifests: set bundle2 part parameter indicating treemanifest
Martin von Zweigbergk <martinvonz@google.com>
parents: 26829
diff changeset
  1751
        op.repo._applyopenerreqs()
5c0fd878779c treemanifests: set bundle2 part parameter indicating treemanifest
Martin von Zweigbergk <martinvonz@google.com>
parents: 26829
diff changeset
  1752
        op.repo._writerequirements()
33407
39d4e5a66f5a bundle2: support the 'targetphase' parameter for the changegroup part
Boris Feld <boris.feld@octobus.net>
parents: 33252
diff changeset
  1753
    extrakwargs = {}
39d4e5a66f5a bundle2: support the 'targetphase' parameter for the changegroup part
Boris Feld <boris.feld@octobus.net>
parents: 33252
diff changeset
  1754
    targetphase = inpart.params.get('targetphase')
39d4e5a66f5a bundle2: support the 'targetphase' parameter for the changegroup part
Boris Feld <boris.feld@octobus.net>
parents: 33252
diff changeset
  1755
    if targetphase is not None:
36154
b587a889b97e py3: use raw string for key in **kwargs
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35813
diff changeset
  1756
        extrakwargs[r'targetphase'] = int(targetphase)
33050
f0efd2bffe1e bundle: extract _processchangegroup() method
Martin von Zweigbergk <martinvonz@google.com>
parents: 33049
diff changeset
  1757
    ret = _processchangegroup(op, cg, tr, 'bundle2', 'bundle2',
33407
39d4e5a66f5a bundle2: support the 'targetphase' parameter for the changegroup part
Boris Feld <boris.feld@octobus.net>
parents: 33252
diff changeset
  1758
                              expectedtotal=nbchangesets, **extrakwargs)
20998
93a3c5b58635 bundle2: use reply part to return result of addchangegroup
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20997
diff changeset
  1759
    if op.reply is not None:
23139
e53f6b72a0e4 spelling: fixes from proofreading of spell checker issues
Mads Kiilerich <madski@unity3d.com>
parents: 23067
diff changeset
  1760
        # This is definitely not the final form of this
20998
93a3c5b58635 bundle2: use reply part to return result of addchangegroup
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20997
diff changeset
  1761
        # return. But one need to start somewhere.
24686
e0e28e910fa3 bundle2: rename format, parts and config to final names
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24648
diff changeset
  1762
        part = op.reply.newpart('reply:changegroup', mandatory=False)
33701
24b7c75c6441 bundle2: use bytestr() instead of str() to convert part id to bytes
Augie Fackler <augie@google.com>
parents: 33700
diff changeset
  1763
        part.addparam(
24b7c75c6441 bundle2: use bytestr() instead of str() to convert part id to bytes
Augie Fackler <augie@google.com>
parents: 33700
diff changeset
  1764
            'in-reply-to', pycompat.bytestr(inpart.id), mandatory=False)
21606
e55888447958 bundle2: update part creators to ``addparam`` when relevant
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21605
diff changeset
  1765
        part.addparam('return', '%i' % ret, mandatory=False)
21019
3dc09f831a2e bundle2: lazy unbundle of part payload
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21016
diff changeset
  1766
    assert not inpart.read()
20950
c7ceae0faf69 bundle2: first crude version of bundling changeset with bundle2
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20949
diff changeset
  1767
23029
149fc8a44184 bundle2: client side support for a part to import external bundles
Mike Hommey <mh@glandium.org>
parents: 23011
diff changeset
  1768
_remotechangegroupparams = tuple(['url', 'size', 'digests'] +
149fc8a44184 bundle2: client side support for a part to import external bundles
Mike Hommey <mh@glandium.org>
parents: 23011
diff changeset
  1769
    ['digest:%s' % k for k in util.DIGESTS.keys()])
24686
e0e28e910fa3 bundle2: rename format, parts and config to final names
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24648
diff changeset
  1770
@parthandler('remote-changegroup', _remotechangegroupparams)
23029
149fc8a44184 bundle2: client side support for a part to import external bundles
Mike Hommey <mh@glandium.org>
parents: 23011
diff changeset
  1771
def handleremotechangegroup(op, inpart):
149fc8a44184 bundle2: client side support for a part to import external bundles
Mike Hommey <mh@glandium.org>
parents: 23011
diff changeset
  1772
    """apply a bundle10 on the repo, given an url and validation information
149fc8a44184 bundle2: client side support for a part to import external bundles
Mike Hommey <mh@glandium.org>
parents: 23011
diff changeset
  1773
149fc8a44184 bundle2: client side support for a part to import external bundles
Mike Hommey <mh@glandium.org>
parents: 23011
diff changeset
  1774
    All the information about the remote bundle to import are given as
149fc8a44184 bundle2: client side support for a part to import external bundles
Mike Hommey <mh@glandium.org>
parents: 23011
diff changeset
  1775
    parameters. The parameters include:
149fc8a44184 bundle2: client side support for a part to import external bundles
Mike Hommey <mh@glandium.org>
parents: 23011
diff changeset
  1776
      - url: the url to the bundle10.
149fc8a44184 bundle2: client side support for a part to import external bundles
Mike Hommey <mh@glandium.org>
parents: 23011
diff changeset
  1777
      - size: the bundle10 file size. It is used to validate what was
149fc8a44184 bundle2: client side support for a part to import external bundles
Mike Hommey <mh@glandium.org>
parents: 23011
diff changeset
  1778
        retrieved by the client matches the server knowledge about the bundle.
149fc8a44184 bundle2: client side support for a part to import external bundles
Mike Hommey <mh@glandium.org>
parents: 23011
diff changeset
  1779
      - digests: a space separated list of the digest types provided as
149fc8a44184 bundle2: client side support for a part to import external bundles
Mike Hommey <mh@glandium.org>
parents: 23011
diff changeset
  1780
        parameters.
149fc8a44184 bundle2: client side support for a part to import external bundles
Mike Hommey <mh@glandium.org>
parents: 23011
diff changeset
  1781
      - digest:<digest-type>: the hexadecimal representation of the digest with
149fc8a44184 bundle2: client side support for a part to import external bundles
Mike Hommey <mh@glandium.org>
parents: 23011
diff changeset
  1782
        that name. Like the size, it is used to validate what was retrieved by
149fc8a44184 bundle2: client side support for a part to import external bundles
Mike Hommey <mh@glandium.org>
parents: 23011
diff changeset
  1783
        the client matches what the server knows about the bundle.
149fc8a44184 bundle2: client side support for a part to import external bundles
Mike Hommey <mh@glandium.org>
parents: 23011
diff changeset
  1784
149fc8a44184 bundle2: client side support for a part to import external bundles
Mike Hommey <mh@glandium.org>
parents: 23011
diff changeset
  1785
    When multiple digest types are given, all of them are checked.
149fc8a44184 bundle2: client side support for a part to import external bundles
Mike Hommey <mh@glandium.org>
parents: 23011
diff changeset
  1786
    """
149fc8a44184 bundle2: client side support for a part to import external bundles
Mike Hommey <mh@glandium.org>
parents: 23011
diff changeset
  1787
    try:
149fc8a44184 bundle2: client side support for a part to import external bundles
Mike Hommey <mh@glandium.org>
parents: 23011
diff changeset
  1788
        raw_url = inpart.params['url']
149fc8a44184 bundle2: client side support for a part to import external bundles
Mike Hommey <mh@glandium.org>
parents: 23011
diff changeset
  1789
    except KeyError:
26587
56b2bcea2529 error: get Abort from 'error' instead of 'util'
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26565
diff changeset
  1790
        raise error.Abort(_('remote-changegroup: missing "%s" param') % 'url')
23029
149fc8a44184 bundle2: client side support for a part to import external bundles
Mike Hommey <mh@glandium.org>
parents: 23011
diff changeset
  1791
    parsed_url = util.url(raw_url)
24686
e0e28e910fa3 bundle2: rename format, parts and config to final names
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24648
diff changeset
  1792
    if parsed_url.scheme not in capabilities['remote-changegroup']:
26587
56b2bcea2529 error: get Abort from 'error' instead of 'util'
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26565
diff changeset
  1793
        raise error.Abort(_('remote-changegroup does not support %s urls') %
23029
149fc8a44184 bundle2: client side support for a part to import external bundles
Mike Hommey <mh@glandium.org>
parents: 23011
diff changeset
  1794
            parsed_url.scheme)
149fc8a44184 bundle2: client side support for a part to import external bundles
Mike Hommey <mh@glandium.org>
parents: 23011
diff changeset
  1795
149fc8a44184 bundle2: client side support for a part to import external bundles
Mike Hommey <mh@glandium.org>
parents: 23011
diff changeset
  1796
    try:
149fc8a44184 bundle2: client side support for a part to import external bundles
Mike Hommey <mh@glandium.org>
parents: 23011
diff changeset
  1797
        size = int(inpart.params['size'])
149fc8a44184 bundle2: client side support for a part to import external bundles
Mike Hommey <mh@glandium.org>
parents: 23011
diff changeset
  1798
    except ValueError:
26587
56b2bcea2529 error: get Abort from 'error' instead of 'util'
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26565
diff changeset
  1799
        raise error.Abort(_('remote-changegroup: invalid value for param "%s"')
23029
149fc8a44184 bundle2: client side support for a part to import external bundles
Mike Hommey <mh@glandium.org>
parents: 23011
diff changeset
  1800
            % 'size')
149fc8a44184 bundle2: client side support for a part to import external bundles
Mike Hommey <mh@glandium.org>
parents: 23011
diff changeset
  1801
    except KeyError:
26587
56b2bcea2529 error: get Abort from 'error' instead of 'util'
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26565
diff changeset
  1802
        raise error.Abort(_('remote-changegroup: missing "%s" param') % 'size')
23029
149fc8a44184 bundle2: client side support for a part to import external bundles
Mike Hommey <mh@glandium.org>
parents: 23011
diff changeset
  1803
149fc8a44184 bundle2: client side support for a part to import external bundles
Mike Hommey <mh@glandium.org>
parents: 23011
diff changeset
  1804
    digests = {}
149fc8a44184 bundle2: client side support for a part to import external bundles
Mike Hommey <mh@glandium.org>
parents: 23011
diff changeset
  1805
    for typ in inpart.params.get('digests', '').split():
149fc8a44184 bundle2: client side support for a part to import external bundles
Mike Hommey <mh@glandium.org>
parents: 23011
diff changeset
  1806
        param = 'digest:%s' % typ
149fc8a44184 bundle2: client side support for a part to import external bundles
Mike Hommey <mh@glandium.org>
parents: 23011
diff changeset
  1807
        try:
149fc8a44184 bundle2: client side support for a part to import external bundles
Mike Hommey <mh@glandium.org>
parents: 23011
diff changeset
  1808
            value = inpart.params[param]
149fc8a44184 bundle2: client side support for a part to import external bundles
Mike Hommey <mh@glandium.org>
parents: 23011
diff changeset
  1809
        except KeyError:
26587
56b2bcea2529 error: get Abort from 'error' instead of 'util'
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26565
diff changeset
  1810
            raise error.Abort(_('remote-changegroup: missing "%s" param') %
23029
149fc8a44184 bundle2: client side support for a part to import external bundles
Mike Hommey <mh@glandium.org>
parents: 23011
diff changeset
  1811
                param)
149fc8a44184 bundle2: client side support for a part to import external bundles
Mike Hommey <mh@glandium.org>
parents: 23011
diff changeset
  1812
        digests[typ] = value
149fc8a44184 bundle2: client side support for a part to import external bundles
Mike Hommey <mh@glandium.org>
parents: 23011
diff changeset
  1813
149fc8a44184 bundle2: client side support for a part to import external bundles
Mike Hommey <mh@glandium.org>
parents: 23011
diff changeset
  1814
    real_part = util.digestchecker(url.open(op.ui, raw_url), size, digests)
149fc8a44184 bundle2: client side support for a part to import external bundles
Mike Hommey <mh@glandium.org>
parents: 23011
diff changeset
  1815
32948
af31d531dda0 changegroup: let callers pass in transaction to apply() (API)
Martin von Zweigbergk <martinvonz@google.com>
parents: 32910
diff changeset
  1816
    tr = op.gettransaction()
25919
8221fefaea08 bundle2: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25660
diff changeset
  1817
    from . import exchange
23029
149fc8a44184 bundle2: client side support for a part to import external bundles
Mike Hommey <mh@glandium.org>
parents: 23011
diff changeset
  1818
    cg = exchange.readbundle(op.repo.ui, real_part, raw_url)
149fc8a44184 bundle2: client side support for a part to import external bundles
Mike Hommey <mh@glandium.org>
parents: 23011
diff changeset
  1819
    if not isinstance(cg, changegroup.cg1unpacker):
26587
56b2bcea2529 error: get Abort from 'error' instead of 'util'
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26565
diff changeset
  1820
        raise error.Abort(_('%s: not a bundle version 1.0') %
23029
149fc8a44184 bundle2: client side support for a part to import external bundles
Mike Hommey <mh@glandium.org>
parents: 23011
diff changeset
  1821
            util.hidepassword(raw_url))
33050
f0efd2bffe1e bundle: extract _processchangegroup() method
Martin von Zweigbergk <martinvonz@google.com>
parents: 33049
diff changeset
  1822
    ret = _processchangegroup(op, cg, tr, 'bundle2', 'bundle2')
23029
149fc8a44184 bundle2: client side support for a part to import external bundles
Mike Hommey <mh@glandium.org>
parents: 23011
diff changeset
  1823
    if op.reply is not None:
23139
e53f6b72a0e4 spelling: fixes from proofreading of spell checker issues
Mads Kiilerich <madski@unity3d.com>
parents: 23067
diff changeset
  1824
        # This is definitely not the final form of this
23029
149fc8a44184 bundle2: client side support for a part to import external bundles
Mike Hommey <mh@glandium.org>
parents: 23011
diff changeset
  1825
        # return. But one need to start somewhere.
24686
e0e28e910fa3 bundle2: rename format, parts and config to final names
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24648
diff changeset
  1826
        part = op.reply.newpart('reply:changegroup')
33701
24b7c75c6441 bundle2: use bytestr() instead of str() to convert part id to bytes
Augie Fackler <augie@google.com>
parents: 33700
diff changeset
  1827
        part.addparam(
24b7c75c6441 bundle2: use bytestr() instead of str() to convert part id to bytes
Augie Fackler <augie@google.com>
parents: 33700
diff changeset
  1828
            'in-reply-to', pycompat.bytestr(inpart.id), mandatory=False)
23029
149fc8a44184 bundle2: client side support for a part to import external bundles
Mike Hommey <mh@glandium.org>
parents: 23011
diff changeset
  1829
        part.addparam('return', '%i' % ret, mandatory=False)
149fc8a44184 bundle2: client side support for a part to import external bundles
Mike Hommey <mh@glandium.org>
parents: 23011
diff changeset
  1830
    try:
149fc8a44184 bundle2: client side support for a part to import external bundles
Mike Hommey <mh@glandium.org>
parents: 23011
diff changeset
  1831
        real_part.validate()
26587
56b2bcea2529 error: get Abort from 'error' instead of 'util'
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26565
diff changeset
  1832
    except error.Abort as e:
56b2bcea2529 error: get Abort from 'error' instead of 'util'
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26565
diff changeset
  1833
        raise error.Abort(_('bundle at %s is corrupted:\n%s') %
23029
149fc8a44184 bundle2: client side support for a part to import external bundles
Mike Hommey <mh@glandium.org>
parents: 23011
diff changeset
  1834
            (util.hidepassword(raw_url), str(e)))
149fc8a44184 bundle2: client side support for a part to import external bundles
Mike Hommey <mh@glandium.org>
parents: 23011
diff changeset
  1835
    assert not inpart.read()
149fc8a44184 bundle2: client side support for a part to import external bundles
Mike Hommey <mh@glandium.org>
parents: 23011
diff changeset
  1836
24686
e0e28e910fa3 bundle2: rename format, parts and config to final names
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24648
diff changeset
  1837
@parthandler('reply:changegroup', ('return', 'in-reply-to'))
22548
8a1ae133770a bundle2: rename functions that have the same name
Mike Hommey <mh@glandium.org>
parents: 22390
diff changeset
  1838
def handlereplychangegroup(op, inpart):
21611
71b7b3f79a3c bundle2: use the new ``part.params`` dictionary
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21610
diff changeset
  1839
    ret = int(inpart.params['return'])
71b7b3f79a3c bundle2: use the new ``part.params`` dictionary
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21610
diff changeset
  1840
    replyto = int(inpart.params['in-reply-to'])
71b7b3f79a3c bundle2: use the new ``part.params`` dictionary
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21610
diff changeset
  1841
    op.records.add('changegroup', {'return': ret}, replyto)
20950
c7ceae0faf69 bundle2: first crude version of bundling changeset with bundle2
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20949
diff changeset
  1842
35267
dbf868623daf bookmark: add a 'check:bookmarks' bundle2 part
Boris Feld <boris.feld@octobus.net>
parents: 35141
diff changeset
  1843
@parthandler('check:bookmarks')
dbf868623daf bookmark: add a 'check:bookmarks' bundle2 part
Boris Feld <boris.feld@octobus.net>
parents: 35141
diff changeset
  1844
def handlecheckbookmarks(op, inpart):
dbf868623daf bookmark: add a 'check:bookmarks' bundle2 part
Boris Feld <boris.feld@octobus.net>
parents: 35141
diff changeset
  1845
    """check location of bookmarks
dbf868623daf bookmark: add a 'check:bookmarks' bundle2 part
Boris Feld <boris.feld@octobus.net>
parents: 35141
diff changeset
  1846
dbf868623daf bookmark: add a 'check:bookmarks' bundle2 part
Boris Feld <boris.feld@octobus.net>
parents: 35141
diff changeset
  1847
    This part is to be used to detect push race regarding bookmark, it
dbf868623daf bookmark: add a 'check:bookmarks' bundle2 part
Boris Feld <boris.feld@octobus.net>
parents: 35141
diff changeset
  1848
    contains binary encoded (bookmark, node) tuple. If the local state does
dbf868623daf bookmark: add a 'check:bookmarks' bundle2 part
Boris Feld <boris.feld@octobus.net>
parents: 35141
diff changeset
  1849
    not marks the one in the part, a PushRaced exception is raised
dbf868623daf bookmark: add a 'check:bookmarks' bundle2 part
Boris Feld <boris.feld@octobus.net>
parents: 35141
diff changeset
  1850
    """
dbf868623daf bookmark: add a 'check:bookmarks' bundle2 part
Boris Feld <boris.feld@octobus.net>
parents: 35141
diff changeset
  1851
    bookdata = bookmarks.binarydecode(inpart)
dbf868623daf bookmark: add a 'check:bookmarks' bundle2 part
Boris Feld <boris.feld@octobus.net>
parents: 35141
diff changeset
  1852
dbf868623daf bookmark: add a 'check:bookmarks' bundle2 part
Boris Feld <boris.feld@octobus.net>
parents: 35141
diff changeset
  1853
    msgstandard = ('repository changed while pushing - please try again '
dbf868623daf bookmark: add a 'check:bookmarks' bundle2 part
Boris Feld <boris.feld@octobus.net>
parents: 35141
diff changeset
  1854
                   '(bookmark "%s" move from %s to %s)')
dbf868623daf bookmark: add a 'check:bookmarks' bundle2 part
Boris Feld <boris.feld@octobus.net>
parents: 35141
diff changeset
  1855
    msgmissing = ('repository changed while pushing - please try again '
dbf868623daf bookmark: add a 'check:bookmarks' bundle2 part
Boris Feld <boris.feld@octobus.net>
parents: 35141
diff changeset
  1856
                  '(bookmark "%s" is missing, expected %s)')
dbf868623daf bookmark: add a 'check:bookmarks' bundle2 part
Boris Feld <boris.feld@octobus.net>
parents: 35141
diff changeset
  1857
    msgexist = ('repository changed while pushing - please try again '
dbf868623daf bookmark: add a 'check:bookmarks' bundle2 part
Boris Feld <boris.feld@octobus.net>
parents: 35141
diff changeset
  1858
                '(bookmark "%s" set on %s, expected missing)')
dbf868623daf bookmark: add a 'check:bookmarks' bundle2 part
Boris Feld <boris.feld@octobus.net>
parents: 35141
diff changeset
  1859
    for book, node in bookdata:
dbf868623daf bookmark: add a 'check:bookmarks' bundle2 part
Boris Feld <boris.feld@octobus.net>
parents: 35141
diff changeset
  1860
        currentnode = op.repo._bookmarks.get(book)
dbf868623daf bookmark: add a 'check:bookmarks' bundle2 part
Boris Feld <boris.feld@octobus.net>
parents: 35141
diff changeset
  1861
        if currentnode != node:
dbf868623daf bookmark: add a 'check:bookmarks' bundle2 part
Boris Feld <boris.feld@octobus.net>
parents: 35141
diff changeset
  1862
            if node is None:
dbf868623daf bookmark: add a 'check:bookmarks' bundle2 part
Boris Feld <boris.feld@octobus.net>
parents: 35141
diff changeset
  1863
                finalmsg = msgexist % (book, nodemod.short(currentnode))
dbf868623daf bookmark: add a 'check:bookmarks' bundle2 part
Boris Feld <boris.feld@octobus.net>
parents: 35141
diff changeset
  1864
            elif currentnode is None:
dbf868623daf bookmark: add a 'check:bookmarks' bundle2 part
Boris Feld <boris.feld@octobus.net>
parents: 35141
diff changeset
  1865
                finalmsg = msgmissing % (book, nodemod.short(node))
dbf868623daf bookmark: add a 'check:bookmarks' bundle2 part
Boris Feld <boris.feld@octobus.net>
parents: 35141
diff changeset
  1866
            else:
dbf868623daf bookmark: add a 'check:bookmarks' bundle2 part
Boris Feld <boris.feld@octobus.net>
parents: 35141
diff changeset
  1867
                finalmsg = msgstandard % (book, nodemod.short(node),
dbf868623daf bookmark: add a 'check:bookmarks' bundle2 part
Boris Feld <boris.feld@octobus.net>
parents: 35141
diff changeset
  1868
                                          nodemod.short(currentnode))
dbf868623daf bookmark: add a 'check:bookmarks' bundle2 part
Boris Feld <boris.feld@octobus.net>
parents: 35141
diff changeset
  1869
            raise error.PushRaced(finalmsg)
dbf868623daf bookmark: add a 'check:bookmarks' bundle2 part
Boris Feld <boris.feld@octobus.net>
parents: 35141
diff changeset
  1870
24686
e0e28e910fa3 bundle2: rename format, parts and config to final names
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24648
diff changeset
  1871
@parthandler('check:heads')
22548
8a1ae133770a bundle2: rename functions that have the same name
Mike Hommey <mh@glandium.org>
parents: 22390
diff changeset
  1872
def handlecheckheads(op, inpart):
21060
0bea9db7543b bundle2: add a "check:heads" handler
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21024
diff changeset
  1873
    """check that head of the repo did not change
0bea9db7543b bundle2: add a "check:heads" handler
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21024
diff changeset
  1874
0bea9db7543b bundle2: add a "check:heads" handler
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21024
diff changeset
  1875
    This is used to detect a push race when using unbundle.
0bea9db7543b bundle2: add a "check:heads" handler
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21024
diff changeset
  1876
    This replaces the "heads" argument of unbundle."""
0bea9db7543b bundle2: add a "check:heads" handler
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21024
diff changeset
  1877
    h = inpart.read(20)
0bea9db7543b bundle2: add a "check:heads" handler
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21024
diff changeset
  1878
    heads = []
0bea9db7543b bundle2: add a "check:heads" handler
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21024
diff changeset
  1879
    while len(h) == 20:
0bea9db7543b bundle2: add a "check:heads" handler
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21024
diff changeset
  1880
        heads.append(h)
0bea9db7543b bundle2: add a "check:heads" handler
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21024
diff changeset
  1881
        h = inpart.read(20)
0bea9db7543b bundle2: add a "check:heads" handler
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21024
diff changeset
  1882
    assert not h
26565
ee1bcb9aa6e4 bundle2: add op.gettransaction() to handlers that need the lock
Durham Goode <durham@fb.com>
parents: 26542
diff changeset
  1883
    # Trigger a transaction so that we are guaranteed to have the lock now.
ee1bcb9aa6e4 bundle2: add op.gettransaction() to handlers that need the lock
Durham Goode <durham@fb.com>
parents: 26542
diff changeset
  1884
    if op.ui.configbool('experimental', 'bundle2lazylocking'):
ee1bcb9aa6e4 bundle2: add op.gettransaction() to handlers that need the lock
Durham Goode <durham@fb.com>
parents: 26542
diff changeset
  1885
        op.gettransaction()
29294
077d0535f51f bundle2: don't assume ordering of heads checked after push
Mads Kiilerich <madski@unity3d.com>
parents: 28883
diff changeset
  1886
    if sorted(heads) != sorted(op.repo.heads()):
21185
5b3717e1a3ea bundle2: add an error message to push race error
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21184
diff changeset
  1887
        raise error.PushRaced('repository changed while pushing - '
5b3717e1a3ea bundle2: add an error message to push race error
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21184
diff changeset
  1888
                              'please try again')
21130
1ff06386217f bundle2: introduce `replycaps` part for on-demand reply
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21129
diff changeset
  1889
32729
16ada4cbb1a9 push: add a way to allow concurrent pushes on unrelated heads
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32549
diff changeset
  1890
@parthandler('check:updated-heads')
16ada4cbb1a9 push: add a way to allow concurrent pushes on unrelated heads
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32549
diff changeset
  1891
def handlecheckupdatedheads(op, inpart):
16ada4cbb1a9 push: add a way to allow concurrent pushes on unrelated heads
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32549
diff changeset
  1892
    """check for race on the heads touched by a push
16ada4cbb1a9 push: add a way to allow concurrent pushes on unrelated heads
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32549
diff changeset
  1893
16ada4cbb1a9 push: add a way to allow concurrent pushes on unrelated heads
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32549
diff changeset
  1894
    This is similar to 'check:heads' but focus on the heads actually updated
16ada4cbb1a9 push: add a way to allow concurrent pushes on unrelated heads
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32549
diff changeset
  1895
    during the push. If other activities happen on unrelated heads, it is
16ada4cbb1a9 push: add a way to allow concurrent pushes on unrelated heads
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32549
diff changeset
  1896
    ignored.
16ada4cbb1a9 push: add a way to allow concurrent pushes on unrelated heads
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32549
diff changeset
  1897
16ada4cbb1a9 push: add a way to allow concurrent pushes on unrelated heads
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32549
diff changeset
  1898
    This allow server with high traffic to avoid push contention as long as
16ada4cbb1a9 push: add a way to allow concurrent pushes on unrelated heads
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32549
diff changeset
  1899
    unrelated parts of the graph are involved."""
16ada4cbb1a9 push: add a way to allow concurrent pushes on unrelated heads
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32549
diff changeset
  1900
    h = inpart.read(20)
16ada4cbb1a9 push: add a way to allow concurrent pushes on unrelated heads
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32549
diff changeset
  1901
    heads = []
16ada4cbb1a9 push: add a way to allow concurrent pushes on unrelated heads
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32549
diff changeset
  1902
    while len(h) == 20:
16ada4cbb1a9 push: add a way to allow concurrent pushes on unrelated heads
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32549
diff changeset
  1903
        heads.append(h)
16ada4cbb1a9 push: add a way to allow concurrent pushes on unrelated heads
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32549
diff changeset
  1904
        h = inpart.read(20)
16ada4cbb1a9 push: add a way to allow concurrent pushes on unrelated heads
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32549
diff changeset
  1905
    assert not h
16ada4cbb1a9 push: add a way to allow concurrent pushes on unrelated heads
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32549
diff changeset
  1906
    # trigger a transaction so that we are guaranteed to have the lock now.
16ada4cbb1a9 push: add a way to allow concurrent pushes on unrelated heads
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32549
diff changeset
  1907
    if op.ui.configbool('experimental', 'bundle2lazylocking'):
16ada4cbb1a9 push: add a way to allow concurrent pushes on unrelated heads
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32549
diff changeset
  1908
        op.gettransaction()
16ada4cbb1a9 push: add a way to allow concurrent pushes on unrelated heads
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32549
diff changeset
  1909
16ada4cbb1a9 push: add a way to allow concurrent pushes on unrelated heads
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32549
diff changeset
  1910
    currentheads = set()
16ada4cbb1a9 push: add a way to allow concurrent pushes on unrelated heads
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32549
diff changeset
  1911
    for ls in op.repo.branchmap().itervalues():
16ada4cbb1a9 push: add a way to allow concurrent pushes on unrelated heads
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32549
diff changeset
  1912
        currentheads.update(ls)
16ada4cbb1a9 push: add a way to allow concurrent pushes on unrelated heads
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32549
diff changeset
  1913
16ada4cbb1a9 push: add a way to allow concurrent pushes on unrelated heads
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32549
diff changeset
  1914
    for h in heads:
16ada4cbb1a9 push: add a way to allow concurrent pushes on unrelated heads
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32549
diff changeset
  1915
        if h not in currentheads:
16ada4cbb1a9 push: add a way to allow concurrent pushes on unrelated heads
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32549
diff changeset
  1916
            raise error.PushRaced('repository changed while pushing - '
16ada4cbb1a9 push: add a way to allow concurrent pushes on unrelated heads
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32549
diff changeset
  1917
                                  'please try again')
16ada4cbb1a9 push: add a way to allow concurrent pushes on unrelated heads
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32549
diff changeset
  1918
34820
a95067b1dca6 phase: introduce a new 'check:phases' part
Boris Feld <boris.feld@octobus.net>
parents: 34637
diff changeset
  1919
@parthandler('check:phases')
a95067b1dca6 phase: introduce a new 'check:phases' part
Boris Feld <boris.feld@octobus.net>
parents: 34637
diff changeset
  1920
def handlecheckphases(op, inpart):
a95067b1dca6 phase: introduce a new 'check:phases' part
Boris Feld <boris.feld@octobus.net>
parents: 34637
diff changeset
  1921
    """check that phase boundaries of the repository did not change
a95067b1dca6 phase: introduce a new 'check:phases' part
Boris Feld <boris.feld@octobus.net>
parents: 34637
diff changeset
  1922
a95067b1dca6 phase: introduce a new 'check:phases' part
Boris Feld <boris.feld@octobus.net>
parents: 34637
diff changeset
  1923
    This is used to detect a push race.
a95067b1dca6 phase: introduce a new 'check:phases' part
Boris Feld <boris.feld@octobus.net>
parents: 34637
diff changeset
  1924
    """
a95067b1dca6 phase: introduce a new 'check:phases' part
Boris Feld <boris.feld@octobus.net>
parents: 34637
diff changeset
  1925
    phasetonodes = phases.binarydecode(inpart)
a95067b1dca6 phase: introduce a new 'check:phases' part
Boris Feld <boris.feld@octobus.net>
parents: 34637
diff changeset
  1926
    unfi = op.repo.unfiltered()
a95067b1dca6 phase: introduce a new 'check:phases' part
Boris Feld <boris.feld@octobus.net>
parents: 34637
diff changeset
  1927
    cl = unfi.changelog
a95067b1dca6 phase: introduce a new 'check:phases' part
Boris Feld <boris.feld@octobus.net>
parents: 34637
diff changeset
  1928
    phasecache = unfi._phasecache
a95067b1dca6 phase: introduce a new 'check:phases' part
Boris Feld <boris.feld@octobus.net>
parents: 34637
diff changeset
  1929
    msg = ('repository changed while pushing - please try again '
a95067b1dca6 phase: introduce a new 'check:phases' part
Boris Feld <boris.feld@octobus.net>
parents: 34637
diff changeset
  1930
           '(%s is %s expected %s)')
a95067b1dca6 phase: introduce a new 'check:phases' part
Boris Feld <boris.feld@octobus.net>
parents: 34637
diff changeset
  1931
    for expectedphase, nodes in enumerate(phasetonodes):
a95067b1dca6 phase: introduce a new 'check:phases' part
Boris Feld <boris.feld@octobus.net>
parents: 34637
diff changeset
  1932
        for n in nodes:
a95067b1dca6 phase: introduce a new 'check:phases' part
Boris Feld <boris.feld@octobus.net>
parents: 34637
diff changeset
  1933
            actualphase = phasecache.phase(unfi, cl.rev(n))
a95067b1dca6 phase: introduce a new 'check:phases' part
Boris Feld <boris.feld@octobus.net>
parents: 34637
diff changeset
  1934
            if actualphase != expectedphase:
a95067b1dca6 phase: introduce a new 'check:phases' part
Boris Feld <boris.feld@octobus.net>
parents: 34637
diff changeset
  1935
                finalmsg = msg % (nodemod.short(n),
a95067b1dca6 phase: introduce a new 'check:phases' part
Boris Feld <boris.feld@octobus.net>
parents: 34637
diff changeset
  1936
                                  phases.phasenames[actualphase],
a95067b1dca6 phase: introduce a new 'check:phases' part
Boris Feld <boris.feld@octobus.net>
parents: 34637
diff changeset
  1937
                                  phases.phasenames[expectedphase])
a95067b1dca6 phase: introduce a new 'check:phases' part
Boris Feld <boris.feld@octobus.net>
parents: 34637
diff changeset
  1938
                raise error.PushRaced(finalmsg)
a95067b1dca6 phase: introduce a new 'check:phases' part
Boris Feld <boris.feld@octobus.net>
parents: 34637
diff changeset
  1939
24686
e0e28e910fa3 bundle2: rename format, parts and config to final names
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24648
diff changeset
  1940
@parthandler('output')
21131
b7435117d951 bundle2: capture remote stdout while unbundling
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21130
diff changeset
  1941
def handleoutput(op, inpart):
b7435117d951 bundle2: capture remote stdout while unbundling
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21130
diff changeset
  1942
    """forward output captured on the server to the client"""
b7435117d951 bundle2: capture remote stdout while unbundling
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21130
diff changeset
  1943
    for line in inpart.read().splitlines():
29664
ccd436f7db6d bundle2: localize handleoutput remote prompts
Akihiko Odaki <akihiko.odaki.4i@stu.hosei.ac.jp>
parents: 29662
diff changeset
  1944
        op.ui.status(_('remote: %s\n') % line)
21131
b7435117d951 bundle2: capture remote stdout while unbundling
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21130
diff changeset
  1945
24686
e0e28e910fa3 bundle2: rename format, parts and config to final names
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24648
diff changeset
  1946
@parthandler('replycaps')
21130
1ff06386217f bundle2: introduce `replycaps` part for on-demand reply
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21129
diff changeset
  1947
def handlereplycaps(op, inpart):
1ff06386217f bundle2: introduce `replycaps` part for on-demand reply
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21129
diff changeset
  1948
    """Notify that a reply bundle should be created
1ff06386217f bundle2: introduce `replycaps` part for on-demand reply
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21129
diff changeset
  1949
21138
f469879d27ec bundle2: extract capabilities decoding
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21137
diff changeset
  1950
    The payload contains the capabilities information for the reply"""
f469879d27ec bundle2: extract capabilities decoding
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21137
diff changeset
  1951
    caps = decodecaps(inpart.read())
21130
1ff06386217f bundle2: introduce `replycaps` part for on-demand reply
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21129
diff changeset
  1952
    if op.reply is None:
21135
98fbf3adfd83 bundle2: add capabilities support in `replycaps` part
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21134
diff changeset
  1953
        op.reply = bundle20(op.ui, caps)
21131
b7435117d951 bundle2: capture remote stdout while unbundling
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21130
diff changeset
  1954
26829
58f1645f72c3 bundle2: attribute remote failures to remote (issue4788)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26802
diff changeset
  1955
class AbortFromPart(error.Abort):
58f1645f72c3 bundle2: attribute remote failures to remote (issue4788)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26802
diff changeset
  1956
    """Sub-class of Abort that denotes an error from a bundle2 part."""
58f1645f72c3 bundle2: attribute remote failures to remote (issue4788)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26802
diff changeset
  1957
24686
e0e28e910fa3 bundle2: rename format, parts and config to final names
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24648
diff changeset
  1958
@parthandler('error:abort', ('message', 'hint'))
24741
bb67e52362d6 bundle2: fix names for error part handler
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24733
diff changeset
  1959
def handleerrorabort(op, inpart):
21177
952af771bc17 bundle2: gracefully handle abort during unbundle
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21176
diff changeset
  1960
    """Used to transmit abort error over the wire"""
26829
58f1645f72c3 bundle2: attribute remote failures to remote (issue4788)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26802
diff changeset
  1961
    raise AbortFromPart(inpart.params['message'],
58f1645f72c3 bundle2: attribute remote failures to remote (issue4788)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26802
diff changeset
  1962
                        hint=inpart.params.get('hint'))
21183
4345274adc4b bundle2: gracefully handle UnknownPartError during unbundle
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21179
diff changeset
  1963
25493
d8e7b0781ad7 bundle2: convey PushkeyFailed error over the wire
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25492
diff changeset
  1964
@parthandler('error:pushkey', ('namespace', 'key', 'new', 'old', 'ret',
d8e7b0781ad7 bundle2: convey PushkeyFailed error over the wire
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25492
diff changeset
  1965
                               'in-reply-to'))
d8e7b0781ad7 bundle2: convey PushkeyFailed error over the wire
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25492
diff changeset
  1966
def handleerrorpushkey(op, inpart):
d8e7b0781ad7 bundle2: convey PushkeyFailed error over the wire
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25492
diff changeset
  1967
    """Used to transmit failure of a mandatory pushkey over the wire"""
d8e7b0781ad7 bundle2: convey PushkeyFailed error over the wire
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25492
diff changeset
  1968
    kwargs = {}
d8e7b0781ad7 bundle2: convey PushkeyFailed error over the wire
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25492
diff changeset
  1969
    for name in ('namespace', 'key', 'new', 'old', 'ret'):
d8e7b0781ad7 bundle2: convey PushkeyFailed error over the wire
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25492
diff changeset
  1970
        value = inpart.params.get(name)
d8e7b0781ad7 bundle2: convey PushkeyFailed error over the wire
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25492
diff changeset
  1971
        if value is not None:
d8e7b0781ad7 bundle2: convey PushkeyFailed error over the wire
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25492
diff changeset
  1972
            kwargs[name] = value
36457
247b473f408e bundle2: **strkwargs love on various kwargs constructions
Augie Fackler <augie@google.com>
parents: 36306
diff changeset
  1973
    raise error.PushkeyFailed(inpart.params['in-reply-to'],
247b473f408e bundle2: **strkwargs love on various kwargs constructions
Augie Fackler <augie@google.com>
parents: 36306
diff changeset
  1974
                              **pycompat.strkwargs(kwargs))
25493
d8e7b0781ad7 bundle2: convey PushkeyFailed error over the wire
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25492
diff changeset
  1975
24686
e0e28e910fa3 bundle2: rename format, parts and config to final names
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24648
diff changeset
  1976
@parthandler('error:unsupportedcontent', ('parttype', 'params'))
24741
bb67e52362d6 bundle2: fix names for error part handler
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24733
diff changeset
  1977
def handleerrorunsupportedcontent(op, inpart):
21619
292331e906d7 bundle2: rename b2x:error:unknownpart to b2x:error:unsupportedcontent
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21618
diff changeset
  1978
    """Used to transmit unknown content error over the wire"""
21622
457492741007 bundle2: support transmission of params error over the wire
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21620
diff changeset
  1979
    kwargs = {}
21627
3e8bcc90f07c bundle2: support None parttype in BundleValueError
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21626
diff changeset
  1980
    parttype = inpart.params.get('parttype')
3e8bcc90f07c bundle2: support None parttype in BundleValueError
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21626
diff changeset
  1981
    if parttype is not None:
3e8bcc90f07c bundle2: support None parttype in BundleValueError
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21626
diff changeset
  1982
        kwargs['parttype'] = parttype
21622
457492741007 bundle2: support transmission of params error over the wire
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21620
diff changeset
  1983
    params = inpart.params.get('params')
457492741007 bundle2: support transmission of params error over the wire
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21620
diff changeset
  1984
    if params is not None:
457492741007 bundle2: support transmission of params error over the wire
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21620
diff changeset
  1985
        kwargs['params'] = params.split('\0')
457492741007 bundle2: support transmission of params error over the wire
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21620
diff changeset
  1986
36457
247b473f408e bundle2: **strkwargs love on various kwargs constructions
Augie Fackler <augie@google.com>
parents: 36306
diff changeset
  1987
    raise error.BundleUnknownFeatureError(**pycompat.strkwargs(kwargs))
21186
9f3652e851f8 bundle2: gracefully handle PushRaced error during unbundle
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21185
diff changeset
  1988
24686
e0e28e910fa3 bundle2: rename format, parts and config to final names
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24648
diff changeset
  1989
@parthandler('error:pushraced', ('message',))
24741
bb67e52362d6 bundle2: fix names for error part handler
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24733
diff changeset
  1990
def handleerrorpushraced(op, inpart):
21186
9f3652e851f8 bundle2: gracefully handle PushRaced error during unbundle
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21185
diff changeset
  1991
    """Used to transmit push race error over the wire"""
21611
71b7b3f79a3c bundle2: use the new ``part.params`` dictionary
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21610
diff changeset
  1992
    raise error.ResponseError(_('push failed:'), inpart.params['message'])
21655
35095f332846 bundle: introduce a listkey handler
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21644
diff changeset
  1993
24686
e0e28e910fa3 bundle2: rename format, parts and config to final names
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24648
diff changeset
  1994
@parthandler('listkeys', ('namespace',))
21655
35095f332846 bundle: introduce a listkey handler
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21644
diff changeset
  1995
def handlelistkeys(op, inpart):
35095f332846 bundle: introduce a listkey handler
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21644
diff changeset
  1996
    """retrieve pushkey namespace content stored in a bundle2"""
35095f332846 bundle: introduce a listkey handler
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21644
diff changeset
  1997
    namespace = inpart.params['namespace']
35095f332846 bundle: introduce a listkey handler
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21644
diff changeset
  1998
    r = pushkey.decodekeys(inpart.read())
35095f332846 bundle: introduce a listkey handler
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21644
diff changeset
  1999
    op.records.add('listkeys', (namespace, r))
21660
e87d2a12d41b bundle2: add ``pushkey`` support
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21655
diff changeset
  2000
24686
e0e28e910fa3 bundle2: rename format, parts and config to final names
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24648
diff changeset
  2001
@parthandler('pushkey', ('namespace', 'key', 'old', 'new'))
21660
e87d2a12d41b bundle2: add ``pushkey`` support
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21655
diff changeset
  2002
def handlepushkey(op, inpart):
e87d2a12d41b bundle2: add ``pushkey`` support
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21655
diff changeset
  2003
    """process a pushkey request"""
e87d2a12d41b bundle2: add ``pushkey`` support
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21655
diff changeset
  2004
    dec = pushkey.decode
e87d2a12d41b bundle2: add ``pushkey`` support
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21655
diff changeset
  2005
    namespace = dec(inpart.params['namespace'])
e87d2a12d41b bundle2: add ``pushkey`` support
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21655
diff changeset
  2006
    key = dec(inpart.params['key'])
e87d2a12d41b bundle2: add ``pushkey`` support
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21655
diff changeset
  2007
    old = dec(inpart.params['old'])
e87d2a12d41b bundle2: add ``pushkey`` support
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21655
diff changeset
  2008
    new = dec(inpart.params['new'])
26565
ee1bcb9aa6e4 bundle2: add op.gettransaction() to handlers that need the lock
Durham Goode <durham@fb.com>
parents: 26542
diff changeset
  2009
    # Grab the transaction to ensure that we have the lock before performing the
ee1bcb9aa6e4 bundle2: add op.gettransaction() to handlers that need the lock
Durham Goode <durham@fb.com>
parents: 26542
diff changeset
  2010
    # pushkey.
ee1bcb9aa6e4 bundle2: add op.gettransaction() to handlers that need the lock
Durham Goode <durham@fb.com>
parents: 26542
diff changeset
  2011
    if op.ui.configbool('experimental', 'bundle2lazylocking'):
ee1bcb9aa6e4 bundle2: add op.gettransaction() to handlers that need the lock
Durham Goode <durham@fb.com>
parents: 26542
diff changeset
  2012
        op.gettransaction()
21660
e87d2a12d41b bundle2: add ``pushkey`` support
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21655
diff changeset
  2013
    ret = op.repo.pushkey(namespace, key, old, new)
e87d2a12d41b bundle2: add ``pushkey`` support
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21655
diff changeset
  2014
    record = {'namespace': namespace,
e87d2a12d41b bundle2: add ``pushkey`` support
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21655
diff changeset
  2015
              'key': key,
e87d2a12d41b bundle2: add ``pushkey`` support
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21655
diff changeset
  2016
              'old': old,
e87d2a12d41b bundle2: add ``pushkey`` support
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21655
diff changeset
  2017
              'new': new}
e87d2a12d41b bundle2: add ``pushkey`` support
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21655
diff changeset
  2018
    op.records.add('pushkey', record)
e87d2a12d41b bundle2: add ``pushkey`` support
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21655
diff changeset
  2019
    if op.reply is not None:
24686
e0e28e910fa3 bundle2: rename format, parts and config to final names
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24648
diff changeset
  2020
        rpart = op.reply.newpart('reply:pushkey')
33701
24b7c75c6441 bundle2: use bytestr() instead of str() to convert part id to bytes
Augie Fackler <augie@google.com>
parents: 33700
diff changeset
  2021
        rpart.addparam(
24b7c75c6441 bundle2: use bytestr() instead of str() to convert part id to bytes
Augie Fackler <augie@google.com>
parents: 33700
diff changeset
  2022
            'in-reply-to', pycompat.bytestr(inpart.id), mandatory=False)
21660
e87d2a12d41b bundle2: add ``pushkey`` support
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21655
diff changeset
  2023
        rpart.addparam('return', '%i' % ret, mandatory=False)
25481
6de96cb31176 bundle2: abort when a mandatory pushkey part fails
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25401
diff changeset
  2024
    if inpart.mandatory and not ret:
25484
a5192774e925 bundle2: introduce a PushkeyFail error to abort unbundle on pushkey error
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25481
diff changeset
  2025
        kwargs = {}
a5192774e925 bundle2: introduce a PushkeyFail error to abort unbundle on pushkey error
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25481
diff changeset
  2026
        for key in ('namespace', 'key', 'new', 'old', 'ret'):
a5192774e925 bundle2: introduce a PushkeyFail error to abort unbundle on pushkey error
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25481
diff changeset
  2027
            if key in inpart.params:
a5192774e925 bundle2: introduce a PushkeyFail error to abort unbundle on pushkey error
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25481
diff changeset
  2028
                kwargs[key] = inpart.params[key]
36458
0b697c650b04 bundle2: part id is an int, use %d to make it bytes
Augie Fackler <augie@google.com>
parents: 36457
diff changeset
  2029
        raise error.PushkeyFailed(partid='%d' % inpart.id,
36457
247b473f408e bundle2: **strkwargs love on various kwargs constructions
Augie Fackler <augie@google.com>
parents: 36306
diff changeset
  2030
                                  **pycompat.strkwargs(kwargs))
21660
e87d2a12d41b bundle2: add ``pushkey`` support
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21655
diff changeset
  2031
35269
af5507203d01 bookmark: introduce a 'bookmarks' part
Boris Feld <boris.feld@octobus.net>
parents: 35268
diff changeset
  2032
@parthandler('bookmarks')
af5507203d01 bookmark: introduce a 'bookmarks' part
Boris Feld <boris.feld@octobus.net>
parents: 35268
diff changeset
  2033
def handlebookmark(op, inpart):
af5507203d01 bookmark: introduce a 'bookmarks' part
Boris Feld <boris.feld@octobus.net>
parents: 35268
diff changeset
  2034
    """transmit bookmark information
af5507203d01 bookmark: introduce a 'bookmarks' part
Boris Feld <boris.feld@octobus.net>
parents: 35268
diff changeset
  2035
35275
496154e41968 bundle2: support a 'records' mode for the 'bookmarks' part
Boris Feld <boris.feld@octobus.net>
parents: 35274
diff changeset
  2036
    The part contains binary encoded bookmark information.
496154e41968 bundle2: support a 'records' mode for the 'bookmarks' part
Boris Feld <boris.feld@octobus.net>
parents: 35274
diff changeset
  2037
496154e41968 bundle2: support a 'records' mode for the 'bookmarks' part
Boris Feld <boris.feld@octobus.net>
parents: 35274
diff changeset
  2038
    The exact behavior of this part can be controlled by the 'bookmarks' mode
496154e41968 bundle2: support a 'records' mode for the 'bookmarks' part
Boris Feld <boris.feld@octobus.net>
parents: 35274
diff changeset
  2039
    on the bundle operation.
35269
af5507203d01 bookmark: introduce a 'bookmarks' part
Boris Feld <boris.feld@octobus.net>
parents: 35268
diff changeset
  2040
35275
496154e41968 bundle2: support a 'records' mode for the 'bookmarks' part
Boris Feld <boris.feld@octobus.net>
parents: 35274
diff changeset
  2041
    When mode is 'apply' (the default) the bookmark information is applied as
496154e41968 bundle2: support a 'records' mode for the 'bookmarks' part
Boris Feld <boris.feld@octobus.net>
parents: 35274
diff changeset
  2042
    is to the unbundling repository. Make sure a 'check:bookmarks' part is
496154e41968 bundle2: support a 'records' mode for the 'bookmarks' part
Boris Feld <boris.feld@octobus.net>
parents: 35274
diff changeset
  2043
    issued earlier to check for push races in such update. This behavior is
496154e41968 bundle2: support a 'records' mode for the 'bookmarks' part
Boris Feld <boris.feld@octobus.net>
parents: 35274
diff changeset
  2044
    suitable for pushing.
496154e41968 bundle2: support a 'records' mode for the 'bookmarks' part
Boris Feld <boris.feld@octobus.net>
parents: 35274
diff changeset
  2045
496154e41968 bundle2: support a 'records' mode for the 'bookmarks' part
Boris Feld <boris.feld@octobus.net>
parents: 35274
diff changeset
  2046
    When mode is 'records', the information is recorded into the 'bookmarks'
496154e41968 bundle2: support a 'records' mode for the 'bookmarks' part
Boris Feld <boris.feld@octobus.net>
parents: 35274
diff changeset
  2047
    records of the bundle operation. This behavior is suitable for pulling.
35269
af5507203d01 bookmark: introduce a 'bookmarks' part
Boris Feld <boris.feld@octobus.net>
parents: 35268
diff changeset
  2048
    """
af5507203d01 bookmark: introduce a 'bookmarks' part
Boris Feld <boris.feld@octobus.net>
parents: 35268
diff changeset
  2049
    changes = bookmarks.binarydecode(inpart)
35270
f392066d127c bookmark: add pushkey hook compatiblity to the bundle2 part
Boris Feld <boris.feld@octobus.net>
parents: 35269
diff changeset
  2050
35275
496154e41968 bundle2: support a 'records' mode for the 'bookmarks' part
Boris Feld <boris.feld@octobus.net>
parents: 35274
diff changeset
  2051
    pushkeycompat = op.repo.ui.configbool('server', 'bookmarks-pushkey-compat')
496154e41968 bundle2: support a 'records' mode for the 'bookmarks' part
Boris Feld <boris.feld@octobus.net>
parents: 35274
diff changeset
  2052
    bookmarksmode = op.modes.get('bookmarks', 'apply')
35270
f392066d127c bookmark: add pushkey hook compatiblity to the bundle2 part
Boris Feld <boris.feld@octobus.net>
parents: 35269
diff changeset
  2053
35275
496154e41968 bundle2: support a 'records' mode for the 'bookmarks' part
Boris Feld <boris.feld@octobus.net>
parents: 35274
diff changeset
  2054
    if bookmarksmode == 'apply':
496154e41968 bundle2: support a 'records' mode for the 'bookmarks' part
Boris Feld <boris.feld@octobus.net>
parents: 35274
diff changeset
  2055
        tr = op.gettransaction()
496154e41968 bundle2: support a 'records' mode for the 'bookmarks' part
Boris Feld <boris.feld@octobus.net>
parents: 35274
diff changeset
  2056
        bookstore = op.repo._bookmarks
496154e41968 bundle2: support a 'records' mode for the 'bookmarks' part
Boris Feld <boris.feld@octobus.net>
parents: 35274
diff changeset
  2057
        if pushkeycompat:
496154e41968 bundle2: support a 'records' mode for the 'bookmarks' part
Boris Feld <boris.feld@octobus.net>
parents: 35274
diff changeset
  2058
            allhooks = []
496154e41968 bundle2: support a 'records' mode for the 'bookmarks' part
Boris Feld <boris.feld@octobus.net>
parents: 35274
diff changeset
  2059
            for book, node in changes:
496154e41968 bundle2: support a 'records' mode for the 'bookmarks' part
Boris Feld <boris.feld@octobus.net>
parents: 35274
diff changeset
  2060
                hookargs = tr.hookargs.copy()
496154e41968 bundle2: support a 'records' mode for the 'bookmarks' part
Boris Feld <boris.feld@octobus.net>
parents: 35274
diff changeset
  2061
                hookargs['pushkeycompat'] = '1'
35811
e35320ce8043 bookmarks: fix pushkey compatibility mode (issue5777)
Boris Feld <boris.feld@octobus.net>
parents: 35804
diff changeset
  2062
                hookargs['namespace'] = 'bookmarks'
35275
496154e41968 bundle2: support a 'records' mode for the 'bookmarks' part
Boris Feld <boris.feld@octobus.net>
parents: 35274
diff changeset
  2063
                hookargs['key'] = book
496154e41968 bundle2: support a 'records' mode for the 'bookmarks' part
Boris Feld <boris.feld@octobus.net>
parents: 35274
diff changeset
  2064
                hookargs['old'] = nodemod.hex(bookstore.get(book, ''))
496154e41968 bundle2: support a 'records' mode for the 'bookmarks' part
Boris Feld <boris.feld@octobus.net>
parents: 35274
diff changeset
  2065
                hookargs['new'] = nodemod.hex(node if node is not None else '')
496154e41968 bundle2: support a 'records' mode for the 'bookmarks' part
Boris Feld <boris.feld@octobus.net>
parents: 35274
diff changeset
  2066
                allhooks.append(hookargs)
496154e41968 bundle2: support a 'records' mode for the 'bookmarks' part
Boris Feld <boris.feld@octobus.net>
parents: 35274
diff changeset
  2067
496154e41968 bundle2: support a 'records' mode for the 'bookmarks' part
Boris Feld <boris.feld@octobus.net>
parents: 35274
diff changeset
  2068
            for hookargs in allhooks:
36306
1ec6fd26f841 bundle2: use pycompat.strkwargs as needed
Augie Fackler <augie@google.com>
parents: 36154
diff changeset
  2069
                op.repo.hook('prepushkey', throw=True,
1ec6fd26f841 bundle2: use pycompat.strkwargs as needed
Augie Fackler <augie@google.com>
parents: 36154
diff changeset
  2070
                             **pycompat.strkwargs(hookargs))
35275
496154e41968 bundle2: support a 'records' mode for the 'bookmarks' part
Boris Feld <boris.feld@octobus.net>
parents: 35274
diff changeset
  2071
496154e41968 bundle2: support a 'records' mode for the 'bookmarks' part
Boris Feld <boris.feld@octobus.net>
parents: 35274
diff changeset
  2072
        bookstore.applychanges(op.repo, op.gettransaction(), changes)
496154e41968 bundle2: support a 'records' mode for the 'bookmarks' part
Boris Feld <boris.feld@octobus.net>
parents: 35274
diff changeset
  2073
496154e41968 bundle2: support a 'records' mode for the 'bookmarks' part
Boris Feld <boris.feld@octobus.net>
parents: 35274
diff changeset
  2074
        if pushkeycompat:
496154e41968 bundle2: support a 'records' mode for the 'bookmarks' part
Boris Feld <boris.feld@octobus.net>
parents: 35274
diff changeset
  2075
            def runhook():
496154e41968 bundle2: support a 'records' mode for the 'bookmarks' part
Boris Feld <boris.feld@octobus.net>
parents: 35274
diff changeset
  2076
                for hookargs in allhooks:
36306
1ec6fd26f841 bundle2: use pycompat.strkwargs as needed
Augie Fackler <augie@google.com>
parents: 36154
diff changeset
  2077
                    op.repo.hook('pushkey', **pycompat.strkwargs(hookargs))
35275
496154e41968 bundle2: support a 'records' mode for the 'bookmarks' part
Boris Feld <boris.feld@octobus.net>
parents: 35274
diff changeset
  2078
            op.repo._afterlock(runhook)
496154e41968 bundle2: support a 'records' mode for the 'bookmarks' part
Boris Feld <boris.feld@octobus.net>
parents: 35274
diff changeset
  2079
496154e41968 bundle2: support a 'records' mode for the 'bookmarks' part
Boris Feld <boris.feld@octobus.net>
parents: 35274
diff changeset
  2080
    elif bookmarksmode == 'records':
35270
f392066d127c bookmark: add pushkey hook compatiblity to the bundle2 part
Boris Feld <boris.feld@octobus.net>
parents: 35269
diff changeset
  2081
        for book, node in changes:
35275
496154e41968 bundle2: support a 'records' mode for the 'bookmarks' part
Boris Feld <boris.feld@octobus.net>
parents: 35274
diff changeset
  2082
            record = {'bookmark': book, 'node': node}
496154e41968 bundle2: support a 'records' mode for the 'bookmarks' part
Boris Feld <boris.feld@octobus.net>
parents: 35274
diff changeset
  2083
            op.records.add('bookmarks', record)
496154e41968 bundle2: support a 'records' mode for the 'bookmarks' part
Boris Feld <boris.feld@octobus.net>
parents: 35274
diff changeset
  2084
    else:
496154e41968 bundle2: support a 'records' mode for the 'bookmarks' part
Boris Feld <boris.feld@octobus.net>
parents: 35274
diff changeset
  2085
        raise error.ProgrammingError('unkown bookmark mode: %s' % bookmarksmode)
35269
af5507203d01 bookmark: introduce a 'bookmarks' part
Boris Feld <boris.feld@octobus.net>
parents: 35268
diff changeset
  2086
33043
e8c8d81eb864 bundle: add config option to include phases
Martin von Zweigbergk <martinvonz@google.com>
parents: 33042
diff changeset
  2087
@parthandler('phase-heads')
e8c8d81eb864 bundle: add config option to include phases
Martin von Zweigbergk <martinvonz@google.com>
parents: 33042
diff changeset
  2088
def handlephases(op, inpart):
e8c8d81eb864 bundle: add config option to include phases
Martin von Zweigbergk <martinvonz@google.com>
parents: 33042
diff changeset
  2089
    """apply phases from bundle part to repo"""
34327
12c42bcd4133 phases: move the binary decoding function in the phases module
Boris Feld <boris.feld@octobus.net>
parents: 34326
diff changeset
  2090
    headsbyphase = phases.binarydecode(inpart)
34328
4ef472b975ff bundle2: only grab a transaction when 'phase-heads' affect the repository
Boris Feld <boris.feld@octobus.net>
parents: 34327
diff changeset
  2091
    phases.updatephases(op.repo.unfiltered(), op.gettransaction, headsbyphase)
33043
e8c8d81eb864 bundle: add config option to include phases
Martin von Zweigbergk <martinvonz@google.com>
parents: 33042
diff changeset
  2092
24686
e0e28e910fa3 bundle2: rename format, parts and config to final names
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24648
diff changeset
  2093
@parthandler('reply:pushkey', ('return', 'in-reply-to'))
21660
e87d2a12d41b bundle2: add ``pushkey`` support
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21655
diff changeset
  2094
def handlepushkeyreply(op, inpart):
e87d2a12d41b bundle2: add ``pushkey`` support
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21655
diff changeset
  2095
    """retrieve the result of a pushkey request"""
e87d2a12d41b bundle2: add ``pushkey`` support
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21655
diff changeset
  2096
    ret = int(inpart.params['return'])
e87d2a12d41b bundle2: add ``pushkey`` support
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21655
diff changeset
  2097
    partid = int(inpart.params['in-reply-to'])
e87d2a12d41b bundle2: add ``pushkey`` support
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21655
diff changeset
  2098
    op.records.add('pushkey', {'return': ret}, partid)
22336
60786c8a2f70 bundle2: add an obsmarkers part handler
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21900
diff changeset
  2099
24686
e0e28e910fa3 bundle2: rename format, parts and config to final names
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24648
diff changeset
  2100
@parthandler('obsmarkers')
22336
60786c8a2f70 bundle2: add an obsmarkers part handler
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21900
diff changeset
  2101
def handleobsmarker(op, inpart):
60786c8a2f70 bundle2: add an obsmarkers part handler
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21900
diff changeset
  2102
    """add a stream of obsmarkers to the repo"""
60786c8a2f70 bundle2: add an obsmarkers part handler
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21900
diff changeset
  2103
    tr = op.gettransaction()
24733
c00e4338fa4b obsolete: experimental flag to get debug about obsmarkers exchange
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24686
diff changeset
  2104
    markerdata = inpart.read()
33499
0407a51b9d8c codemod: register core configitems using a script
Jun Wu <quark@fb.com>
parents: 33461
diff changeset
  2105
    if op.ui.config('experimental', 'obsmarkers-exchange-debug'):
24733
c00e4338fa4b obsolete: experimental flag to get debug about obsmarkers exchange
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24686
diff changeset
  2106
        op.ui.write(('obsmarker-exchange: %i bytes received\n')
c00e4338fa4b obsolete: experimental flag to get debug about obsmarkers exchange
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24686
diff changeset
  2107
                    % len(markerdata))
26685
ea390d889d3a bundle2: gracefully skip 'obsmarkers' part if evolution is disabled
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26587
diff changeset
  2108
    # The mergemarkers call will crash if marker creation is not enabled.
ea390d889d3a bundle2: gracefully skip 'obsmarkers' part if evolution is disabled
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26587
diff changeset
  2109
    # we want to avoid this if the part is advisory.
ea390d889d3a bundle2: gracefully skip 'obsmarkers' part if evolution is disabled
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26587
diff changeset
  2110
    if not inpart.mandatory and op.repo.obsstore.readonly:
35470
7906354cbc68 debug: add newlines at the end of three locations that appear to need it
Kyle Lippincott <spectral@google.com>
parents: 35275
diff changeset
  2111
        op.repo.ui.debug('ignoring obsolescence markers, feature not enabled\n')
26685
ea390d889d3a bundle2: gracefully skip 'obsmarkers' part if evolution is disabled
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26587
diff changeset
  2112
        return
24733
c00e4338fa4b obsolete: experimental flag to get debug about obsmarkers exchange
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24686
diff changeset
  2113
    new = op.repo.obsstore.mergemarkers(tr, markerdata)
32124
99515353c72a obsolete: invalidate "volatile" set cache after merging marker
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 32024
diff changeset
  2114
    op.repo.invalidatevolatilesets()
22337
c380fe290290 obsmarker: write a message with the number of markers added through bundle2
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 22336
diff changeset
  2115
    if new:
c380fe290290 obsmarker: write a message with the number of markers added through bundle2
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 22336
diff changeset
  2116
        op.repo.ui.status(_('%i new obsolescence markers\n') % new)
22338
8fcd56095d3b obssmarker: add a bundle2 record with the number of markers added
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 22337
diff changeset
  2117
    op.records.add('obsmarkers', {'new': new})
22340
394a17de6a2d obsmarker: produce a reply part for markers received through bundle2
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 22338
diff changeset
  2118
    if op.reply is not None:
24686
e0e28e910fa3 bundle2: rename format, parts and config to final names
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24648
diff changeset
  2119
        rpart = op.reply.newpart('reply:obsmarkers')
33701
24b7c75c6441 bundle2: use bytestr() instead of str() to convert part id to bytes
Augie Fackler <augie@google.com>
parents: 33700
diff changeset
  2120
        rpart.addparam(
24b7c75c6441 bundle2: use bytestr() instead of str() to convert part id to bytes
Augie Fackler <augie@google.com>
parents: 33700
diff changeset
  2121
            'in-reply-to', pycompat.bytestr(inpart.id), mandatory=False)
22340
394a17de6a2d obsmarker: produce a reply part for markers received through bundle2
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 22338
diff changeset
  2122
        rpart.addparam('new', '%i' % new, mandatory=False)
394a17de6a2d obsmarker: produce a reply part for markers received through bundle2
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 22338
diff changeset
  2123
394a17de6a2d obsmarker: produce a reply part for markers received through bundle2
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 22338
diff changeset
  2124
24686
e0e28e910fa3 bundle2: rename format, parts and config to final names
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 24648
diff changeset
  2125
@parthandler('reply:obsmarkers', ('new', 'in-reply-to'))
25506
86472038dfd2 bundle2: rename duplicate handlepushkeyreply to handleobsmarkerreply
Martin von Zweigbergk <martinvonz@google.com>
parents: 25493
diff changeset
  2126
def handleobsmarkerreply(op, inpart):
22340
394a17de6a2d obsmarker: produce a reply part for markers received through bundle2
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 22338
diff changeset
  2127
    """retrieve the result of a pushkey request"""
394a17de6a2d obsmarker: produce a reply part for markers received through bundle2
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 22338
diff changeset
  2128
    ret = int(inpart.params['new'])
394a17de6a2d obsmarker: produce a reply part for markers received through bundle2
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 22338
diff changeset
  2129
    partid = int(inpart.params['in-reply-to'])
394a17de6a2d obsmarker: produce a reply part for markers received through bundle2
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 22338
diff changeset
  2130
    op.records.add('obsmarkers', {'new': ret}, partid)
25401
d29201352af7 bundle2: part handler for processing .hgtags fnodes mappings
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25336
diff changeset
  2131
d29201352af7 bundle2: part handler for processing .hgtags fnodes mappings
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25336
diff changeset
  2132
@parthandler('hgtagsfnodes')
d29201352af7 bundle2: part handler for processing .hgtags fnodes mappings
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25336
diff changeset
  2133
def handlehgtagsfnodes(op, inpart):
d29201352af7 bundle2: part handler for processing .hgtags fnodes mappings
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25336
diff changeset
  2134
    """Applies .hgtags fnodes cache entries to the local repo.
d29201352af7 bundle2: part handler for processing .hgtags fnodes mappings
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25336
diff changeset
  2135
d29201352af7 bundle2: part handler for processing .hgtags fnodes mappings
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25336
diff changeset
  2136
    Payload is pairs of 20 byte changeset nodes and filenodes.
d29201352af7 bundle2: part handler for processing .hgtags fnodes mappings
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25336
diff changeset
  2137
    """
26565
ee1bcb9aa6e4 bundle2: add op.gettransaction() to handlers that need the lock
Durham Goode <durham@fb.com>
parents: 26542
diff changeset
  2138
    # Grab the transaction so we ensure that we have the lock at this point.
ee1bcb9aa6e4 bundle2: add op.gettransaction() to handlers that need the lock
Durham Goode <durham@fb.com>
parents: 26542
diff changeset
  2139
    if op.ui.configbool('experimental', 'bundle2lazylocking'):
ee1bcb9aa6e4 bundle2: add op.gettransaction() to handlers that need the lock
Durham Goode <durham@fb.com>
parents: 26542
diff changeset
  2140
        op.gettransaction()
25401
d29201352af7 bundle2: part handler for processing .hgtags fnodes mappings
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25336
diff changeset
  2141
    cache = tags.hgtagsfnodescache(op.repo.unfiltered())
d29201352af7 bundle2: part handler for processing .hgtags fnodes mappings
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25336
diff changeset
  2142
d29201352af7 bundle2: part handler for processing .hgtags fnodes mappings
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25336
diff changeset
  2143
    count = 0
d29201352af7 bundle2: part handler for processing .hgtags fnodes mappings
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25336
diff changeset
  2144
    while True:
d29201352af7 bundle2: part handler for processing .hgtags fnodes mappings
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25336
diff changeset
  2145
        node = inpart.read(20)
d29201352af7 bundle2: part handler for processing .hgtags fnodes mappings
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25336
diff changeset
  2146
        fnode = inpart.read(20)
d29201352af7 bundle2: part handler for processing .hgtags fnodes mappings
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25336
diff changeset
  2147
        if len(node) < 20 or len(fnode) < 20:
25641
c0bdfe87b245 bundle2: reword debug message for invalid .hgtags data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25640
diff changeset
  2148
            op.ui.debug('ignoring incomplete received .hgtags fnodes data\n')
25401
d29201352af7 bundle2: part handler for processing .hgtags fnodes mappings
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25336
diff changeset
  2149
            break
d29201352af7 bundle2: part handler for processing .hgtags fnodes mappings
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25336
diff changeset
  2150
        cache.setfnode(node, fnode)
d29201352af7 bundle2: part handler for processing .hgtags fnodes mappings
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25336
diff changeset
  2151
        count += 1
d29201352af7 bundle2: part handler for processing .hgtags fnodes mappings
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25336
diff changeset
  2152
d29201352af7 bundle2: part handler for processing .hgtags fnodes mappings
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25336
diff changeset
  2153
    cache.write()
d29201352af7 bundle2: part handler for processing .hgtags fnodes mappings
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25336
diff changeset
  2154
    op.ui.debug('applied %i hgtags fnodes cache entries\n' % count)
33719
db3dc11356ed pushvars: move fb extension pushvars to core
Pulkit Goyal <7895pulkit@gmail.com>
parents: 33708
diff changeset
  2155
36969
9988fc10f49e revbranchcache: add a bundle2 handler for a rbc part
Boris Feld <boris.feld@octobus.net>
parents: 36458
diff changeset
  2156
rbcstruct = struct.Struct('>III')
9988fc10f49e revbranchcache: add a bundle2 handler for a rbc part
Boris Feld <boris.feld@octobus.net>
parents: 36458
diff changeset
  2157
9988fc10f49e revbranchcache: add a bundle2 handler for a rbc part
Boris Feld <boris.feld@octobus.net>
parents: 36458
diff changeset
  2158
@parthandler('cache:rev-branch-cache')
9988fc10f49e revbranchcache: add a bundle2 handler for a rbc part
Boris Feld <boris.feld@octobus.net>
parents: 36458
diff changeset
  2159
def handlerbc(op, inpart):
9988fc10f49e revbranchcache: add a bundle2 handler for a rbc part
Boris Feld <boris.feld@octobus.net>
parents: 36458
diff changeset
  2160
    """receive a rev-branch-cache payload and update the local cache
9988fc10f49e revbranchcache: add a bundle2 handler for a rbc part
Boris Feld <boris.feld@octobus.net>
parents: 36458
diff changeset
  2161
9988fc10f49e revbranchcache: add a bundle2 handler for a rbc part
Boris Feld <boris.feld@octobus.net>
parents: 36458
diff changeset
  2162
    The payload is a series of data related to each branch
9988fc10f49e revbranchcache: add a bundle2 handler for a rbc part
Boris Feld <boris.feld@octobus.net>
parents: 36458
diff changeset
  2163
9988fc10f49e revbranchcache: add a bundle2 handler for a rbc part
Boris Feld <boris.feld@octobus.net>
parents: 36458
diff changeset
  2164
    1) branch name length
9988fc10f49e revbranchcache: add a bundle2 handler for a rbc part
Boris Feld <boris.feld@octobus.net>
parents: 36458
diff changeset
  2165
    2) number of open heads
9988fc10f49e revbranchcache: add a bundle2 handler for a rbc part
Boris Feld <boris.feld@octobus.net>
parents: 36458
diff changeset
  2166
    3) number of closed heads
9988fc10f49e revbranchcache: add a bundle2 handler for a rbc part
Boris Feld <boris.feld@octobus.net>
parents: 36458
diff changeset
  2167
    4) open heads nodes
9988fc10f49e revbranchcache: add a bundle2 handler for a rbc part
Boris Feld <boris.feld@octobus.net>
parents: 36458
diff changeset
  2168
    5) closed heads nodes
9988fc10f49e revbranchcache: add a bundle2 handler for a rbc part
Boris Feld <boris.feld@octobus.net>
parents: 36458
diff changeset
  2169
    """
9988fc10f49e revbranchcache: add a bundle2 handler for a rbc part
Boris Feld <boris.feld@octobus.net>
parents: 36458
diff changeset
  2170
    total = 0
9988fc10f49e revbranchcache: add a bundle2 handler for a rbc part
Boris Feld <boris.feld@octobus.net>
parents: 36458
diff changeset
  2171
    rawheader = inpart.read(rbcstruct.size)
9988fc10f49e revbranchcache: add a bundle2 handler for a rbc part
Boris Feld <boris.feld@octobus.net>
parents: 36458
diff changeset
  2172
    cache = op.repo.revbranchcache()
9988fc10f49e revbranchcache: add a bundle2 handler for a rbc part
Boris Feld <boris.feld@octobus.net>
parents: 36458
diff changeset
  2173
    cl = op.repo.unfiltered().changelog
9988fc10f49e revbranchcache: add a bundle2 handler for a rbc part
Boris Feld <boris.feld@octobus.net>
parents: 36458
diff changeset
  2174
    while rawheader:
9988fc10f49e revbranchcache: add a bundle2 handler for a rbc part
Boris Feld <boris.feld@octobus.net>
parents: 36458
diff changeset
  2175
        header = rbcstruct.unpack(rawheader)
9988fc10f49e revbranchcache: add a bundle2 handler for a rbc part
Boris Feld <boris.feld@octobus.net>
parents: 36458
diff changeset
  2176
        total += header[1] + header[2]
9988fc10f49e revbranchcache: add a bundle2 handler for a rbc part
Boris Feld <boris.feld@octobus.net>
parents: 36458
diff changeset
  2177
        utf8branch = inpart.read(header[0])
9988fc10f49e revbranchcache: add a bundle2 handler for a rbc part
Boris Feld <boris.feld@octobus.net>
parents: 36458
diff changeset
  2178
        branch = encoding.tolocal(utf8branch)
9988fc10f49e revbranchcache: add a bundle2 handler for a rbc part
Boris Feld <boris.feld@octobus.net>
parents: 36458
diff changeset
  2179
        for x in xrange(header[1]):
9988fc10f49e revbranchcache: add a bundle2 handler for a rbc part
Boris Feld <boris.feld@octobus.net>
parents: 36458
diff changeset
  2180
            node = inpart.read(20)
9988fc10f49e revbranchcache: add a bundle2 handler for a rbc part
Boris Feld <boris.feld@octobus.net>
parents: 36458
diff changeset
  2181
            rev = cl.rev(node)
9988fc10f49e revbranchcache: add a bundle2 handler for a rbc part
Boris Feld <boris.feld@octobus.net>
parents: 36458
diff changeset
  2182
            cache.setdata(branch, rev, node, False)
9988fc10f49e revbranchcache: add a bundle2 handler for a rbc part
Boris Feld <boris.feld@octobus.net>
parents: 36458
diff changeset
  2183
        for x in xrange(header[2]):
9988fc10f49e revbranchcache: add a bundle2 handler for a rbc part
Boris Feld <boris.feld@octobus.net>
parents: 36458
diff changeset
  2184
            node = inpart.read(20)
9988fc10f49e revbranchcache: add a bundle2 handler for a rbc part
Boris Feld <boris.feld@octobus.net>
parents: 36458
diff changeset
  2185
            rev = cl.rev(node)
9988fc10f49e revbranchcache: add a bundle2 handler for a rbc part
Boris Feld <boris.feld@octobus.net>
parents: 36458
diff changeset
  2186
            cache.setdata(branch, rev, node, True)
9988fc10f49e revbranchcache: add a bundle2 handler for a rbc part
Boris Feld <boris.feld@octobus.net>
parents: 36458
diff changeset
  2187
        rawheader = inpart.read(rbcstruct.size)
9988fc10f49e revbranchcache: add a bundle2 handler for a rbc part
Boris Feld <boris.feld@octobus.net>
parents: 36458
diff changeset
  2188
    cache.write()
9988fc10f49e revbranchcache: add a bundle2 handler for a rbc part
Boris Feld <boris.feld@octobus.net>
parents: 36458
diff changeset
  2189
33719
db3dc11356ed pushvars: move fb extension pushvars to core
Pulkit Goyal <7895pulkit@gmail.com>
parents: 33708
diff changeset
  2190
@parthandler('pushvars')
db3dc11356ed pushvars: move fb extension pushvars to core
Pulkit Goyal <7895pulkit@gmail.com>
parents: 33708
diff changeset
  2191
def bundle2getvars(op, part):
db3dc11356ed pushvars: move fb extension pushvars to core
Pulkit Goyal <7895pulkit@gmail.com>
parents: 33708
diff changeset
  2192
    '''unbundle a bundle2 containing shellvars on the server'''
db3dc11356ed pushvars: move fb extension pushvars to core
Pulkit Goyal <7895pulkit@gmail.com>
parents: 33708
diff changeset
  2193
    # An option to disable unbundling on server-side for security reasons
33856
057d31ceace3 pushvars: add a coreconfigitem for push.pushvars.server
Pulkit Goyal <7895pulkit@gmail.com>
parents: 33831
diff changeset
  2194
    if op.ui.configbool('push', 'pushvars.server'):
33719
db3dc11356ed pushvars: move fb extension pushvars to core
Pulkit Goyal <7895pulkit@gmail.com>
parents: 33708
diff changeset
  2195
        hookargs = {}
db3dc11356ed pushvars: move fb extension pushvars to core
Pulkit Goyal <7895pulkit@gmail.com>
parents: 33708
diff changeset
  2196
        for key, value in part.advisoryparams:
db3dc11356ed pushvars: move fb extension pushvars to core
Pulkit Goyal <7895pulkit@gmail.com>
parents: 33708
diff changeset
  2197
            key = key.upper()
db3dc11356ed pushvars: move fb extension pushvars to core
Pulkit Goyal <7895pulkit@gmail.com>
parents: 33708
diff changeset
  2198
            # We want pushed variables to have USERVAR_ prepended so we know
db3dc11356ed pushvars: move fb extension pushvars to core
Pulkit Goyal <7895pulkit@gmail.com>
parents: 33708
diff changeset
  2199
            # they came from the --pushvar flag.
db3dc11356ed pushvars: move fb extension pushvars to core
Pulkit Goyal <7895pulkit@gmail.com>
parents: 33708
diff changeset
  2200
            key = "USERVAR_" + key
db3dc11356ed pushvars: move fb extension pushvars to core
Pulkit Goyal <7895pulkit@gmail.com>
parents: 33708
diff changeset
  2201
            hookargs[key] = value
db3dc11356ed pushvars: move fb extension pushvars to core
Pulkit Goyal <7895pulkit@gmail.com>
parents: 33708
diff changeset
  2202
        op.addhookargs(hookargs)
35758
b996ddf5963d bundle2: add a 'stream' part handler for stream cloning
Boris Feld <boris.feld@octobus.net>
parents: 35655
diff changeset
  2203
35788
b116a66bcc44 bundle2: move version of stream clone into part name
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35787
diff changeset
  2204
@parthandler('stream2', ('requirements', 'filecount', 'bytecount'))
b116a66bcc44 bundle2: move version of stream clone into part name
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35787
diff changeset
  2205
def handlestreamv2bundle(op, part):
35758
b996ddf5963d bundle2: add a 'stream' part handler for stream cloning
Boris Feld <boris.feld@octobus.net>
parents: 35655
diff changeset
  2206
35813
768326377e4d bundle2: fix the formatting of the stream part requirements
Boris Feld <boris.feld@octobus.net>
parents: 35811
diff changeset
  2207
    requirements = urlreq.unquote(part.params['requirements']).split(',')
35758
b996ddf5963d bundle2: add a 'stream' part handler for stream cloning
Boris Feld <boris.feld@octobus.net>
parents: 35655
diff changeset
  2208
    filecount = int(part.params['filecount'])
b996ddf5963d bundle2: add a 'stream' part handler for stream cloning
Boris Feld <boris.feld@octobus.net>
parents: 35655
diff changeset
  2209
    bytecount = int(part.params['bytecount'])
b996ddf5963d bundle2: add a 'stream' part handler for stream cloning
Boris Feld <boris.feld@octobus.net>
parents: 35655
diff changeset
  2210
b996ddf5963d bundle2: add a 'stream' part handler for stream cloning
Boris Feld <boris.feld@octobus.net>
parents: 35655
diff changeset
  2211
    repo = op.repo
b996ddf5963d bundle2: add a 'stream' part handler for stream cloning
Boris Feld <boris.feld@octobus.net>
parents: 35655
diff changeset
  2212
    if len(repo):
b996ddf5963d bundle2: add a 'stream' part handler for stream cloning
Boris Feld <boris.feld@octobus.net>
parents: 35655
diff changeset
  2213
        msg = _('cannot apply stream clone to non empty repository')
b996ddf5963d bundle2: add a 'stream' part handler for stream cloning
Boris Feld <boris.feld@octobus.net>
parents: 35655
diff changeset
  2214
        raise error.Abort(msg)
b996ddf5963d bundle2: add a 'stream' part handler for stream cloning
Boris Feld <boris.feld@octobus.net>
parents: 35655
diff changeset
  2215
b996ddf5963d bundle2: add a 'stream' part handler for stream cloning
Boris Feld <boris.feld@octobus.net>
parents: 35655
diff changeset
  2216
    repo.ui.debug('applying stream bundle\n')
b996ddf5963d bundle2: add a 'stream' part handler for stream cloning
Boris Feld <boris.feld@octobus.net>
parents: 35655
diff changeset
  2217
    streamclone.applybundlev2(repo, part, filecount, bytecount,
b996ddf5963d bundle2: add a 'stream' part handler for stream cloning
Boris Feld <boris.feld@octobus.net>
parents: 35655
diff changeset
  2218
                              requirements)