hgext/clonebundles.py
author Pierre-Yves David <pierre-yves.david@octobus.net>
Thu, 02 Feb 2023 17:35:09 +0100
changeset 50597 aea55f2a48b3
parent 50438 3973b1dc3ee3
child 50666 60f9602b413e
permissions -rw-r--r--
safehasattr: pass attribute name as string instead of bytes This is a step toward replacing `util.safehasattr` usage with plain `hasattr`. The builtin function behave poorly in Python2 but this was fixed in Python3. These change are done one by one as they tend to have a small odd to trigger puzzling breackage.
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
26623
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
     1
# This software may be used and distributed according to the terms of the
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
     2
# GNU General Public License version 2 or any later version.
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
     3
27738
a0e783d26e81 exchange: make clone bundles non-experimental and enabled by default
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27737
diff changeset
     4
"""advertise pre-generated bundles to seed clones
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
     5
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
     6
"clonebundles" is a server-side extension used to advertise the existence
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
     7
of pre-generated, externally hosted bundle files to clients that are
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
     8
cloning so that cloning can be faster, more reliable, and require less
37498
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
     9
resources on the server. "pullbundles" is a related feature for sending
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
    10
pre-generated bundle files to clients as part of pull operations.
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    11
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    12
Cloning can be a CPU and I/O intensive operation on servers. Traditionally,
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    13
the server, in response to a client's request to clone, dynamically generates
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    14
a bundle containing the entire repository content and sends it to the client.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    15
There is no caching on the server and the server will have to redundantly
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    16
generate the same outgoing bundle in response to each clone request. For
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    17
servers with large repositories or with high clone volume, the load from
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    18
clones can make scaling the server challenging and costly.
26623
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    19
37498
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
    20
This extension provides server operators the ability to offload
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
    21
potentially expensive clone load to an external service. Pre-generated
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
    22
bundles also allow using more CPU intensive compression, reducing the
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
    23
effective bandwidth requirements.
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
    24
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
    25
Here's how clone bundles work:
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    26
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    27
1. A server operator establishes a mechanism for making bundle files available
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    28
   on a hosting service where Mercurial clients can fetch them.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    29
2. A manifest file listing available bundle URLs and some optional metadata
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    30
   is added to the Mercurial repository on the server.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    31
3. A client initiates a clone against a clone bundles aware server.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    32
4. The client sees the server is advertising clone bundles and fetches the
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    33
   manifest listing available bundles.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    34
5. The client filters and sorts the available bundles based on what it
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    35
   supports and prefers.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    36
6. The client downloads and applies an available bundle from the
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    37
   server-specified URL.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    38
7. The client reconnects to the original server and performs the equivalent
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    39
   of :hg:`pull` to retrieve all repository data not in the bundle. (The
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    40
   repository could have been updated between when the bundle was created
37498
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
    41
   and when the client started the clone.) This may use "pullbundles".
26623
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    42
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    43
Instead of the server generating full repository bundles for every clone
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    44
request, it generates full bundles once and they are subsequently reused to
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    45
bootstrap new clones. The server may still transfer data at clone time.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    46
However, this is only data that has been added/changed since the bundle was
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    47
created. For large, established repositories, this can reduce server load for
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    48
clones to less than 1% of original.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    49
37498
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
    50
Here's how pullbundles work:
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
    51
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
    52
1. A manifest file listing available bundles and describing the revisions
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
    53
   is added to the Mercurial repository on the server.
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
    54
2. A new-enough client informs the server that it supports partial pulls
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
    55
   and initiates a pull.
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
    56
3. If the server has pull bundles enabled and sees the client advertising
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
    57
   partial pulls, it checks for a matching pull bundle in the manifest.
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
    58
   A bundle matches if the format is supported by the client, the client
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
    59
   has the required revisions already and needs something from the bundle.
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
    60
4. If there is at least one matching bundle, the server sends it to the client.
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
    61
5. The client applies the bundle and notices that the server reply was
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
    62
   incomplete. It initiates another pull.
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
    63
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    64
To work, this extension requires the following of server operators:
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    65
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    66
* Generating bundle files of repository content (typically periodically,
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    67
  such as once per day).
37498
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
    68
* Clone bundles: A file server that clients have network access to and that
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
    69
  Python knows how to talk to through its normal URL handling facility
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
    70
  (typically an HTTP/HTTPS server).
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    71
* A process for keeping the bundles manifest in sync with available bundle
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    72
  files.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    73
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    74
Strictly speaking, using a static file hosting server isn't required: a server
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    75
operator could use a dynamic service for retrieving bundle data. However,
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    76
static file hosting services are simple and scalable and should be sufficient
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    77
for most needs.
26623
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    78
26884
762bf510b42c clonebundles: fix typo s/comand/command/
Javi Merino <merino.jav@gmail.com>
parents: 26857
diff changeset
    79
Bundle files can be generated with the :hg:`bundle` command. Typically
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    80
:hg:`bundle --all` is used to produce a bundle of the entire repository.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    81
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    82
:hg:`debugcreatestreamclonebundle` can be used to produce a special
37498
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
    83
*streaming clonebundle*. These are bundle files that are extremely efficient
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    84
to produce and consume (read: fast). However, they are larger than
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    85
traditional bundle formats and require that clients support the exact set
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    86
of repository data store formats in use by the repository that created them.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    87
Typically, a newer server can serve data that is compatible with older clients.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    88
However, *streaming clone bundles* don't have this guarantee. **Server
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    89
operators need to be aware that newer versions of Mercurial may produce
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    90
streaming clone bundles incompatible with older Mercurial versions.**
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    91
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    92
A server operator is responsible for creating a ``.hg/clonebundles.manifest``
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    93
file containing the list of available bundle files suitable for seeding
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    94
clones. If this file does not exist, the repository will not advertise the
37498
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
    95
existence of clone bundles when clients connect. For pull bundles,
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
    96
``.hg/pullbundles.manifest`` is used.
26623
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    97
32454
702af1ad3b18 clonebundles: fix missing newline character
Matt Harbison <matt_harbison@yahoo.com>
parents: 31146
diff changeset
    98
The manifest file contains a newline (\\n) delimited list of entries.
26623
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    99
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   100
Each line in this file defines an available bundle. Lines have the format:
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   101
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   102
    <URL> [<key>=<value>[ <key>=<value>]]
26623
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   103
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   104
That is, a URL followed by an optional, space-delimited list of key=value
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   105
pairs describing additional properties of this bundle. Both keys and values
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   106
are URI encoded.
26623
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   107
37498
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
   108
For pull bundles, the URL is a path under the ``.hg`` directory of the
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
   109
repository.
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
   110
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   111
Keys in UPPERCASE are reserved for use by Mercurial and are defined below.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   112
All non-uppercase keys can be used by site installations. An example use
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   113
for custom properties is to use the *datacenter* attribute to define which
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   114
data center a file is hosted in. Clients could then prefer a server in the
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   115
data center closest to them.
26623
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   116
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   117
The following reserved keys are currently defined:
26623
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   118
26644
74de1c59f71c clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26623
diff changeset
   119
BUNDLESPEC
74de1c59f71c clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26623
diff changeset
   120
   A "bundle specification" string that describes the type of the bundle.
74de1c59f71c clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26623
diff changeset
   121
74de1c59f71c clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26623
diff changeset
   122
   These are string values that are accepted by the "--type" argument of
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   123
   :hg:`bundle`.
26644
74de1c59f71c clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26623
diff changeset
   124
74de1c59f71c clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26623
diff changeset
   125
   The values are parsed in strict mode, which means they must be of the
74de1c59f71c clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26623
diff changeset
   126
   "<compression>-<type>" form. See
74de1c59f71c clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26623
diff changeset
   127
   mercurial.exchange.parsebundlespec() for more details.
74de1c59f71c clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26623
diff changeset
   128
27886
0288e63ea3be clonebundles: improve BUNDLESPEC documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27738
diff changeset
   129
   :hg:`debugbundle --spec` can be used to print the bundle specification
0288e63ea3be clonebundles: improve BUNDLESPEC documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27738
diff changeset
   130
   string for a bundle file. The output of this command can be used verbatim
0288e63ea3be clonebundles: improve BUNDLESPEC documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27738
diff changeset
   131
   for the value of ``BUNDLESPEC`` (it is already escaped).
0288e63ea3be clonebundles: improve BUNDLESPEC documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27738
diff changeset
   132
26644
74de1c59f71c clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26623
diff changeset
   133
   Clients will automatically filter out specifications that are unknown or
74de1c59f71c clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26623
diff changeset
   134
   unsupported so they won't attempt to download something that likely won't
74de1c59f71c clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26623
diff changeset
   135
   apply.
74de1c59f71c clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26623
diff changeset
   136
74de1c59f71c clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26623
diff changeset
   137
   The actual value doesn't impact client behavior beyond filtering:
74de1c59f71c clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26623
diff changeset
   138
   clients will still sniff the bundle type from the header of downloaded
74de1c59f71c clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26623
diff changeset
   139
   files.
26645
2faa7671a4b3 clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26644
diff changeset
   140
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   141
   **Use of this key is highly recommended**, as it allows clients to
27886
0288e63ea3be clonebundles: improve BUNDLESPEC documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27738
diff changeset
   142
   easily skip unsupported bundles. If this key is not defined, an old
0288e63ea3be clonebundles: improve BUNDLESPEC documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27738
diff changeset
   143
   client may attempt to apply a bundle that it is incapable of reading.
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   144
26645
2faa7671a4b3 clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26644
diff changeset
   145
REQUIRESNI
2faa7671a4b3 clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26644
diff changeset
   146
   Whether Server Name Indication (SNI) is required to connect to the URL.
2faa7671a4b3 clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26644
diff changeset
   147
   SNI allows servers to use multiple certificates on the same IP. It is
2faa7671a4b3 clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26644
diff changeset
   148
   somewhat common in CDNs and other hosting providers. Older Python
2faa7671a4b3 clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26644
diff changeset
   149
   versions do not support SNI. Defining this attribute enables clients
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   150
   with older Python versions to filter this entry without experiencing
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   151
   an opaque SSL failure at connection time.
26645
2faa7671a4b3 clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26644
diff changeset
   152
2faa7671a4b3 clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26644
diff changeset
   153
   If this is defined, it is important to advertise a non-SNI fallback
2faa7671a4b3 clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26644
diff changeset
   154
   URL or clients running old Python releases may not be able to clone
2faa7671a4b3 clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26644
diff changeset
   155
   with the clonebundles facility.
2faa7671a4b3 clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26644
diff changeset
   156
2faa7671a4b3 clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26644
diff changeset
   157
   Value should be "true".
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   158
45252
20d110e6eea6 clonebundles: document REQUIREDRAM key
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43077
diff changeset
   159
REQUIREDRAM
20d110e6eea6 clonebundles: document REQUIREDRAM key
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43077
diff changeset
   160
   Value specifies expected memory requirements to decode the payload.
20d110e6eea6 clonebundles: document REQUIREDRAM key
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43077
diff changeset
   161
   Values can have suffixes for common bytes sizes. e.g. "64MB".
20d110e6eea6 clonebundles: document REQUIREDRAM key
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43077
diff changeset
   162
20d110e6eea6 clonebundles: document REQUIREDRAM key
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43077
diff changeset
   163
   This key is often used with zstd-compressed bundles using a high
20d110e6eea6 clonebundles: document REQUIREDRAM key
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43077
diff changeset
   164
   compression level / window size, which can require 100+ MB of memory
20d110e6eea6 clonebundles: document REQUIREDRAM key
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43077
diff changeset
   165
   to decode.
20d110e6eea6 clonebundles: document REQUIREDRAM key
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43077
diff changeset
   166
37498
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
   167
heads
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
   168
   Used for pull bundles. This contains the ``;`` separated changeset
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
   169
   hashes of the heads of the bundle content.
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
   170
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
   171
bases
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
   172
   Used for pull bundles. This contains the ``;`` separated changeset
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
   173
   hashes of the roots of the bundle content. This can be skipped if
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
   174
   the bundle was created without ``--base``.
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
   175
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   176
Manifests can contain multiple entries. Assuming metadata is defined, clients
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   177
will filter entries from the manifest that they don't support. The remaining
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   178
entries are optionally sorted by client preferences
32773
d25802b0eef5 clonebundles: reference correct config option
Gregory Szorc <gregory.szorc@gmail.com>
parents: 32454
diff changeset
   179
(``ui.clonebundleprefers`` config option). The client then attempts
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   180
to fetch the bundle at the first URL in the remaining list.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   181
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   182
**Errors when downloading a bundle will fail the entire clone operation:
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   183
clients do not automatically fall back to a traditional clone.** The reason
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   184
for this is that if a server is using clone bundles, it is probably doing so
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   185
because the feature is necessary to help it scale. In other words, there
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   186
is an assumption that clone load will be offloaded to another service and
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   187
that the Mercurial server isn't responsible for serving this clone load.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   188
If that other service experiences issues and clients start mass falling back to
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   189
the original Mercurial server, the added clone load could overwhelm the server
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   190
due to unexpected load and effectively take it offline. Not having clients
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   191
automatically fall back to cloning from the original server mitigates this
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   192
scenario.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   193
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   194
Because there is no automatic Mercurial server fallback on failure of the
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   195
bundle hosting service, it is important for server operators to view the bundle
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   196
hosting service as an extension of the Mercurial server in terms of
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   197
availability and service level agreements: if the bundle hosting service goes
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   198
down, so does the ability for clients to clone. Note: clients will see a
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   199
message informing them how to bypass the clone bundles facility when a failure
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   200
occurs. So server operators should prepare for some people to follow these
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   201
instructions when a failure occurs, thus driving more load to the original
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   202
Mercurial server when the bundle hosting service fails.
50430
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   203
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   204
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   205
auto-generation of clone bundles
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   206
--------------------------------
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   207
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   208
It is possible to set Mercurial to automatically re-generate clone bundles when
50432
5b70b9f5a2f9 clone-bundles: only regenerate the clone bundle when cached ration is low
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50431
diff changeset
   209
enough new content is available.
50430
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   210
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   211
Mercurial will take care of the process asynchronously. The defined list of
50431
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   212
bundle-type will be generated, uploaded, and advertised. Older bundles will get
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   213
decommissioned as newer ones replace them.
50430
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   214
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   215
Bundles Generation:
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   216
...................
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   217
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   218
The extension can generate multiple variants of the clone bundle. Each
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   219
different variant will be defined by the "bundle-spec" they use::
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   220
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   221
    [clone-bundles]
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   222
    auto-generate.formats= zstd-v2, gzip-v2
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   223
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   224
See `hg help bundlespec` for details about available options.
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   225
50434
65fb4cedd5ea clone-bundles: add a configuration to control auto-generation on changes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50433
diff changeset
   226
By default, new bundles are generated when 5% of the repository contents or at
65fb4cedd5ea clone-bundles: add a configuration to control auto-generation on changes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50433
diff changeset
   227
least 1000 revisions are not contained in the cached bundles. This option can
65fb4cedd5ea clone-bundles: add a configuration to control auto-generation on changes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50433
diff changeset
   228
be controlled by the `clone-bundles.trigger.below-bundled-ratio` option
65fb4cedd5ea clone-bundles: add a configuration to control auto-generation on changes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50433
diff changeset
   229
(default 0.95) and the `clone-bundles.trigger.revs` option (default 1000)::
50433
d611805e7374 clone-bundles: also control automation based on absolute number of revisions
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50432
diff changeset
   230
d611805e7374 clone-bundles: also control automation based on absolute number of revisions
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50432
diff changeset
   231
    [clone-bundles]
d611805e7374 clone-bundles: also control automation based on absolute number of revisions
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50432
diff changeset
   232
    trigger.below-bundled-ratio=0.95
d611805e7374 clone-bundles: also control automation based on absolute number of revisions
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50432
diff changeset
   233
    trigger.revs=1000
50432
5b70b9f5a2f9 clone-bundles: only regenerate the clone bundle when cached ration is low
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50431
diff changeset
   234
50435
23db5f15cc0c clone-bundles: introduce a command to refresh bundle
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50434
diff changeset
   235
This logic can be manually triggered using the `admin::clone-bundles-refresh`
23db5f15cc0c clone-bundles: introduce a command to refresh bundle
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50434
diff changeset
   236
command, or automatically on each repository change if
50434
65fb4cedd5ea clone-bundles: add a configuration to control auto-generation on changes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50433
diff changeset
   237
`clone-bundles.auto-generate.on-change` is set to `yes`.
65fb4cedd5ea clone-bundles: add a configuration to control auto-generation on changes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50433
diff changeset
   238
65fb4cedd5ea clone-bundles: add a configuration to control auto-generation on changes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50433
diff changeset
   239
    [clone-bundles]
65fb4cedd5ea clone-bundles: add a configuration to control auto-generation on changes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50433
diff changeset
   240
    auto-generate.on-change=yes
65fb4cedd5ea clone-bundles: add a configuration to control auto-generation on changes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50433
diff changeset
   241
    auto-generate.formats= zstd-v2, gzip-v2
65fb4cedd5ea clone-bundles: add a configuration to control auto-generation on changes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50433
diff changeset
   242
50430
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   243
Bundles Upload and Serving:
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   244
...........................
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   245
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   246
The generated bundles need to be made available to users through a "public" URL.
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   247
This should be donne through `clone-bundles.upload-command` configuration. The
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   248
value of this command should be a shell command. It will have access to the
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   249
bundle file path through the `$HGCB_BUNDLE_PATH` variable. And the expected
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   250
basename in the "public" URL is accessible at::
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   251
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   252
  [clone-bundles]
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   253
  upload-command=sftp put $HGCB_BUNDLE_PATH \
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   254
      sftp://bundles.host/clone-bundles/$HGCB_BUNDLE_BASENAME
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   255
50431
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   256
If the file was already uploaded, the command must still succeed.
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   257
50430
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   258
After upload, the file should be available at an url defined by
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   259
`clone-bundles.url-template`.
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   260
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   261
  [clone-bundles]
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   262
  url-template=https://bundles.host/cache/clone-bundles/{basename}
50431
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   263
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   264
Old bundles cleanup:
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   265
....................
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   266
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   267
When new bundles are generated, the older ones are no longer necessary and can
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   268
be removed from storage. This is done through the `clone-bundles.delete-command`
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   269
configuration. The command is given the url of the artifact to delete through
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   270
the `$HGCB_BUNDLE_URL` environment variable.
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   271
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   272
  [clone-bundles]
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   273
  delete-command=sftp rm sftp://bundles.host/clone-bundles/$HGCB_BUNDLE_BASENAME
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   274
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   275
If the file was already deleted, the command must still succeed.
26623
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   276
"""
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   277
28095
7fa139eaebb4 clonebundles: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27886
diff changeset
   278
50430
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   279
import os
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   280
import weakref
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   281
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   282
from mercurial.i18n import _
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   283
26623
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   284
from mercurial import (
45785
80f32ec8653a clonebundle: move the manifest filename to a constant
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 45252
diff changeset
   285
    bundlecaches,
50430
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   286
    commands,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   287
    error,
26623
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   288
    extensions,
50430
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   289
    localrepo,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   290
    lock,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   291
    node,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   292
    registrar,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   293
    util,
37785
b4d85bc122bd wireproto: rename wireproto to wireprotov1server (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37498
diff changeset
   294
    wireprotov1server,
26623
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   295
)
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   296
50430
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   297
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   298
from mercurial.utils import (
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   299
    procutil,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   300
)
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   301
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   302
testedwith = b'ships-with-hg-core'
26623
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   303
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 37785
diff changeset
   304
26623
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   305
def capabilities(orig, repo, proto):
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   306
    caps = orig(repo, proto)
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   307
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   308
    # Only advertise if a manifest exists. This does add some I/O to requests.
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   309
    # But this should be cheaper than a wasted network round trip due to
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   310
    # missing file.
45785
80f32ec8653a clonebundle: move the manifest filename to a constant
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 45252
diff changeset
   311
    if repo.vfs.exists(bundlecaches.CB_MANIFEST_FILE):
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   312
        caps.append(b'clonebundles')
26623
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   313
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   314
    return caps
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   315
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 37785
diff changeset
   316
26623
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   317
def extsetup(ui):
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   318
    extensions.wrapfunction(wireprotov1server, b'_capabilities', capabilities)
50430
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   319
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   320
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   321
# logic for bundle auto-generation
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   322
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   323
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   324
configtable = {}
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   325
configitem = registrar.configitem(configtable)
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   326
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   327
cmdtable = {}
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   328
command = registrar.command(cmdtable)
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   329
50434
65fb4cedd5ea clone-bundles: add a configuration to control auto-generation on changes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50433
diff changeset
   330
configitem(b'clone-bundles', b'auto-generate.on-change', default=False)
50430
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   331
configitem(b'clone-bundles', b'auto-generate.formats', default=list)
50432
5b70b9f5a2f9 clone-bundles: only regenerate the clone bundle when cached ration is low
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50431
diff changeset
   332
configitem(b'clone-bundles', b'trigger.below-bundled-ratio', default=0.95)
50433
d611805e7374 clone-bundles: also control automation based on absolute number of revisions
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50432
diff changeset
   333
configitem(b'clone-bundles', b'trigger.revs', default=1000)
50430
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   334
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   335
configitem(b'clone-bundles', b'upload-command', default=None)
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   336
50431
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   337
configitem(b'clone-bundles', b'delete-command', default=None)
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   338
50430
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   339
configitem(b'clone-bundles', b'url-template', default=None)
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   340
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   341
configitem(b'devel', b'debug.clonebundles', default=False)
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   342
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   343
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   344
# category for the post-close transaction hooks
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   345
CAT_POSTCLOSE = b"clonebundles-autobundles"
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   346
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   347
# template for bundle file names
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   348
BUNDLE_MASK = (
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   349
    b"full-%(bundle_type)s-%(revs)d_revs-%(tip_short)s_tip-%(op_id)s.hg"
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   350
)
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   351
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   352
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   353
# file in .hg/ use to track clonebundles being auto-generated
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   354
AUTO_GEN_FILE = b'clonebundles.auto-gen'
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   355
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   356
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   357
class BundleBase(object):
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   358
    """represents the core of properties that matters for us in a bundle
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   359
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   360
    :bundle_type: the bundlespec (see hg help bundlespec)
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   361
    :revs:        the number of revisions in the repo at bundle creation time
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   362
    :tip_rev:     the rev-num of the tip revision
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   363
    :tip_node:    the node id of the tip-most revision in the bundle
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   364
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   365
    :ready:       True if the bundle is ready to be served
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   366
    """
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   367
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   368
    ready = False
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   369
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   370
    def __init__(self, bundle_type, revs, tip_rev, tip_node):
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   371
        self.bundle_type = bundle_type
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   372
        self.revs = revs
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   373
        self.tip_rev = tip_rev
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   374
        self.tip_node = tip_node
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   375
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   376
    def valid_for(self, repo):
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   377
        """is this bundle applicable to the current repository
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   378
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   379
        This is useful for detecting bundles made irrelevant by stripping.
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   380
        """
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   381
        tip_node = node.bin(self.tip_node)
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   382
        return repo.changelog.index.get_rev(tip_node) == self.tip_rev
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   383
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   384
    def __eq__(self, other):
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   385
        left = (self.ready, self.bundle_type, self.tip_rev, self.tip_node)
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   386
        right = (other.ready, other.bundle_type, other.tip_rev, other.tip_node)
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   387
        return left == right
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   388
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   389
    def __neq__(self, other):
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   390
        return not self == other
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   391
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   392
    def __cmp__(self, other):
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   393
        if self == other:
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   394
            return 0
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   395
        return -1
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   396
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   397
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   398
class RequestedBundle(BundleBase):
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   399
    """A bundle that should be generated.
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   400
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   401
    Additional attributes compared to BundleBase
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   402
    :heads:       list of head revisions (as rev-num)
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   403
    :op_id:       a "unique" identifier for the operation triggering the change
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   404
    """
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   405
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   406
    def __init__(self, bundle_type, revs, tip_rev, tip_node, head_revs, op_id):
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   407
        self.head_revs = head_revs
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   408
        self.op_id = op_id
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   409
        super(RequestedBundle, self).__init__(
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   410
            bundle_type,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   411
            revs,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   412
            tip_rev,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   413
            tip_node,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   414
        )
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   415
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   416
    @property
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   417
    def suggested_filename(self):
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   418
        """A filename that can be used for the generated bundle"""
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   419
        data = {
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   420
            b'bundle_type': self.bundle_type,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   421
            b'revs': self.revs,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   422
            b'heads': self.head_revs,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   423
            b'tip_rev': self.tip_rev,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   424
            b'tip_node': self.tip_node,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   425
            b'tip_short': self.tip_node[:12],
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   426
            b'op_id': self.op_id,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   427
        }
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   428
        return BUNDLE_MASK % data
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   429
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   430
    def generate_bundle(self, repo, file_path):
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   431
        """generate the bundle at `filepath`"""
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   432
        commands.bundle(
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   433
            repo.ui,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   434
            repo,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   435
            file_path,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   436
            base=[b"null"],
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   437
            rev=self.head_revs,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   438
            type=self.bundle_type,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   439
            quiet=True,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   440
        )
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   441
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   442
    def generating(self, file_path, hostname=None, pid=None):
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   443
        """return a GeneratingBundle object from this object"""
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   444
        if pid is None:
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   445
            pid = os.getpid()
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   446
        if hostname is None:
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   447
            hostname = lock._getlockprefix()
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   448
        return GeneratingBundle(
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   449
            self.bundle_type,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   450
            self.revs,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   451
            self.tip_rev,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   452
            self.tip_node,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   453
            hostname,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   454
            pid,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   455
            file_path,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   456
        )
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   457
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   458
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   459
class GeneratingBundle(BundleBase):
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   460
    """A bundle being generated
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   461
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   462
    extra attributes compared to BundleBase:
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   463
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   464
    :hostname: the hostname of the machine generating the bundle
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   465
    :pid:      the pid of the process generating the bundle
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   466
    :filepath: the target filename of the bundle
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   467
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   468
    These attributes exist to help detect stalled generation processes.
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   469
    """
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   470
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   471
    ready = False
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   472
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   473
    def __init__(
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   474
        self, bundle_type, revs, tip_rev, tip_node, hostname, pid, filepath
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   475
    ):
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   476
        self.hostname = hostname
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   477
        self.pid = pid
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   478
        self.filepath = filepath
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   479
        super(GeneratingBundle, self).__init__(
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   480
            bundle_type, revs, tip_rev, tip_node
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   481
        )
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   482
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   483
    @classmethod
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   484
    def from_line(cls, line):
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   485
        """create an object by deserializing a line from AUTO_GEN_FILE"""
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   486
        assert line.startswith(b'PENDING-v1 ')
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   487
        (
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   488
            __,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   489
            bundle_type,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   490
            revs,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   491
            tip_rev,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   492
            tip_node,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   493
            hostname,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   494
            pid,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   495
            filepath,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   496
        ) = line.split()
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   497
        hostname = util.urlreq.unquote(hostname)
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   498
        filepath = util.urlreq.unquote(filepath)
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   499
        revs = int(revs)
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   500
        tip_rev = int(tip_rev)
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   501
        pid = int(pid)
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   502
        return cls(
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   503
            bundle_type, revs, tip_rev, tip_node, hostname, pid, filepath
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   504
        )
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   505
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   506
    def to_line(self):
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   507
        """serialize the object to include as a line in AUTO_GEN_FILE"""
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   508
        templ = b"PENDING-v1 %s %d %d %s %s %d %s"
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   509
        data = (
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   510
            self.bundle_type,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   511
            self.revs,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   512
            self.tip_rev,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   513
            self.tip_node,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   514
            util.urlreq.quote(self.hostname),
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   515
            self.pid,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   516
            util.urlreq.quote(self.filepath),
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   517
        )
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   518
        return templ % data
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   519
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   520
    def __eq__(self, other):
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   521
        if not super(GeneratingBundle, self).__eq__(other):
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   522
            return False
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   523
        left = (self.hostname, self.pid, self.filepath)
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   524
        right = (other.hostname, other.pid, other.filepath)
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   525
        return left == right
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   526
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   527
    def uploaded(self, url, basename):
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   528
        """return a GeneratedBundle from this object"""
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   529
        return GeneratedBundle(
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   530
            self.bundle_type,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   531
            self.revs,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   532
            self.tip_rev,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   533
            self.tip_node,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   534
            url,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   535
            basename,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   536
        )
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   537
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   538
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   539
class GeneratedBundle(BundleBase):
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   540
    """A bundle that is done being generated and can be served
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   541
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   542
    extra attributes compared to BundleBase:
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   543
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   544
    :file_url: the url where the bundle is available.
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   545
    :basename: the "basename" used to upload (useful for deletion)
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   546
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   547
    These attributes exist to generate a bundle manifest
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   548
    (.hg/pullbundles.manifest)
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   549
    """
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   550
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   551
    ready = True
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   552
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   553
    def __init__(
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   554
        self, bundle_type, revs, tip_rev, tip_node, file_url, basename
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   555
    ):
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   556
        self.file_url = file_url
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   557
        self.basename = basename
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   558
        super(GeneratedBundle, self).__init__(
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   559
            bundle_type, revs, tip_rev, tip_node
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   560
        )
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   561
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   562
    @classmethod
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   563
    def from_line(cls, line):
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   564
        """create an object by deserializing a line from AUTO_GEN_FILE"""
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   565
        assert line.startswith(b'DONE-v1 ')
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   566
        (
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   567
            __,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   568
            bundle_type,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   569
            revs,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   570
            tip_rev,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   571
            tip_node,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   572
            file_url,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   573
            basename,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   574
        ) = line.split()
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   575
        revs = int(revs)
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   576
        tip_rev = int(tip_rev)
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   577
        file_url = util.urlreq.unquote(file_url)
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   578
        return cls(bundle_type, revs, tip_rev, tip_node, file_url, basename)
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   579
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   580
    def to_line(self):
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   581
        """serialize the object to include as a line in AUTO_GEN_FILE"""
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   582
        templ = b"DONE-v1 %s %d %d %s %s %s"
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   583
        data = (
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   584
            self.bundle_type,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   585
            self.revs,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   586
            self.tip_rev,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   587
            self.tip_node,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   588
            util.urlreq.quote(self.file_url),
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   589
            self.basename,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   590
        )
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   591
        return templ % data
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   592
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   593
    def manifest_line(self):
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   594
        """serialize the object to include as a line in pullbundles.manifest"""
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   595
        templ = b"%s BUNDLESPEC=%s REQUIRESNI=true"
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   596
        return templ % (self.file_url, self.bundle_type)
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   597
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   598
    def __eq__(self, other):
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   599
        if not super(GeneratedBundle, self).__eq__(other):
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   600
            return False
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   601
        return self.file_url == other.file_url
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   602
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   603
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   604
def parse_auto_gen(content):
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   605
    """parse the AUTO_GEN_FILE to return a list of Bundle object"""
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   606
    bundles = []
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   607
    for line in content.splitlines():
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   608
        if line.startswith(b'PENDING-v1 '):
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   609
            bundles.append(GeneratingBundle.from_line(line))
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   610
        elif line.startswith(b'DONE-v1 '):
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   611
            bundles.append(GeneratedBundle.from_line(line))
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   612
    return bundles
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   613
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   614
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   615
def dumps_auto_gen(bundles):
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   616
    """serialize a list of Bundle as a AUTO_GEN_FILE content"""
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   617
    lines = []
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   618
    for b in bundles:
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   619
        lines.append(b"%s\n" % b.to_line())
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   620
    lines.sort()
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   621
    return b"".join(lines)
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   622
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   623
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   624
def read_auto_gen(repo):
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   625
    """read the AUTO_GEN_FILE for the <repo> a list of Bundle object"""
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   626
    data = repo.vfs.tryread(AUTO_GEN_FILE)
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   627
    if not data:
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   628
        return []
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   629
    return parse_auto_gen(data)
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   630
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   631
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   632
def write_auto_gen(repo, bundles):
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   633
    """write a list of Bundle objects into the repo's AUTO_GEN_FILE"""
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   634
    assert repo._cb_lock_ref is not None
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   635
    data = dumps_auto_gen(bundles)
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   636
    with repo.vfs(AUTO_GEN_FILE, mode=b'wb', atomictemp=True) as f:
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   637
        f.write(data)
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   638
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   639
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   640
def generate_manifest(bundles):
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   641
    """write a list of Bundle objects into the repo's AUTO_GEN_FILE"""
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   642
    bundles = list(bundles)
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   643
    bundles.sort(key=lambda b: b.bundle_type)
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   644
    lines = []
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   645
    for b in bundles:
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   646
        lines.append(b"%s\n" % b.manifest_line())
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   647
    return b"".join(lines)
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   648
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   649
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   650
def update_ondisk_manifest(repo):
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   651
    """update the clonebundle manifest with latest url"""
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   652
    with repo.clonebundles_lock():
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   653
        bundles = read_auto_gen(repo)
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   654
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   655
        per_types = {}
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   656
        for b in bundles:
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   657
            if not (b.ready and b.valid_for(repo)):
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   658
                continue
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   659
            current = per_types.get(b.bundle_type)
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   660
            if current is not None and current.revs >= b.revs:
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   661
                continue
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   662
            per_types[b.bundle_type] = b
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   663
        manifest = generate_manifest(per_types.values())
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   664
        with repo.vfs(
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   665
            bundlecaches.CB_MANIFEST_FILE, mode=b"wb", atomictemp=True
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   666
        ) as f:
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   667
            f.write(manifest)
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   668
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   669
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   670
def update_bundle_list(repo, new_bundles=(), del_bundles=()):
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   671
    """modify the repo's AUTO_GEN_FILE
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   672
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   673
    This method also regenerates the clone bundle manifest when needed"""
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   674
    with repo.clonebundles_lock():
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   675
        bundles = read_auto_gen(repo)
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   676
        if del_bundles:
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   677
            bundles = [b for b in bundles if b not in del_bundles]
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   678
        new_bundles = [b for b in new_bundles if b not in bundles]
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   679
        bundles.extend(new_bundles)
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   680
        write_auto_gen(repo, bundles)
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   681
        all_changed = []
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   682
        all_changed.extend(new_bundles)
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   683
        all_changed.extend(del_bundles)
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   684
        if any(b.ready for b in all_changed):
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   685
            update_ondisk_manifest(repo)
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   686
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   687
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   688
def cleanup_tmp_bundle(repo, target):
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   689
    """remove a GeneratingBundle file and entry"""
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   690
    assert not target.ready
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   691
    with repo.clonebundles_lock():
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   692
        repo.vfs.tryunlink(target.filepath)
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   693
        update_bundle_list(repo, del_bundles=[target])
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   694
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   695
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   696
def finalize_one_bundle(repo, target):
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   697
    """upload a generated bundle and advertise it in the clonebundles.manifest"""
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   698
    with repo.clonebundles_lock():
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   699
        bundles = read_auto_gen(repo)
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   700
        if target in bundles and target.valid_for(repo):
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   701
            result = upload_bundle(repo, target)
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   702
            update_bundle_list(repo, new_bundles=[result])
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   703
    cleanup_tmp_bundle(repo, target)
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   704
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   705
50431
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   706
def find_outdated_bundles(repo, bundles):
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   707
    """finds outdated bundles"""
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   708
    olds = []
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   709
    per_types = {}
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   710
    for b in bundles:
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   711
        if not b.valid_for(repo):
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   712
            olds.append(b)
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   713
            continue
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   714
        l = per_types.setdefault(b.bundle_type, [])
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   715
        l.append(b)
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   716
    for key in sorted(per_types):
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   717
        all = per_types[key]
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   718
        if len(all) > 1:
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   719
            all.sort(key=lambda b: b.revs, reverse=True)
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   720
            olds.extend(all[1:])
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   721
    return olds
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   722
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   723
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   724
def collect_garbage(repo):
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   725
    """finds outdated bundles and get them deleted"""
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   726
    with repo.clonebundles_lock():
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   727
        bundles = read_auto_gen(repo)
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   728
        olds = find_outdated_bundles(repo, bundles)
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   729
        for o in olds:
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   730
            delete_bundle(repo, o)
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   731
        update_bundle_list(repo, del_bundles=olds)
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   732
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   733
50430
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   734
def upload_bundle(repo, bundle):
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   735
    """upload the result of a GeneratingBundle and return a GeneratedBundle
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   736
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   737
    The upload is done using the `clone-bundles.upload-command`
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   738
    """
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   739
    cmd = repo.ui.config(b'clone-bundles', b'upload-command')
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   740
    url = repo.ui.config(b'clone-bundles', b'url-template')
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   741
    basename = repo.vfs.basename(bundle.filepath)
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   742
    filepath = procutil.shellquote(bundle.filepath)
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   743
    variables = {
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   744
        b'HGCB_BUNDLE_PATH': filepath,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   745
        b'HGCB_BUNDLE_BASENAME': basename,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   746
    }
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   747
    env = procutil.shellenviron(environ=variables)
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   748
    ret = repo.ui.system(cmd, environ=env)
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   749
    if ret:
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   750
        raise error.Abort(b"command returned status %d: %s" % (ret, cmd))
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   751
    url = (
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   752
        url.decode('utf8')
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   753
        .format(basename=basename.decode('utf8'))
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   754
        .encode('utf8')
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   755
    )
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   756
    return bundle.uploaded(url, basename)
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   757
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   758
50431
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   759
def delete_bundle(repo, bundle):
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   760
    """delete a bundle from storage"""
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   761
    assert bundle.ready
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   762
    msg = b'clone-bundles: deleting bundle %s\n'
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   763
    msg %= bundle.basename
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   764
    if repo.ui.configbool(b'devel', b'debug.clonebundles'):
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   765
        repo.ui.write(msg)
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   766
    else:
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   767
        repo.ui.debug(msg)
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   768
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   769
    cmd = repo.ui.config(b'clone-bundles', b'delete-command')
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   770
    variables = {
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   771
        b'HGCB_BUNDLE_URL': bundle.file_url,
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   772
        b'HGCB_BASENAME': bundle.basename,
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   773
    }
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   774
    env = procutil.shellenviron(environ=variables)
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   775
    ret = repo.ui.system(cmd, environ=env)
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   776
    if ret:
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   777
        raise error.Abort(b"command returned status %d: %s" % (ret, cmd))
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   778
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   779
50430
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   780
def auto_bundle_needed_actions(repo, bundles, op_id):
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   781
    """find the list of bundles that need action
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   782
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   783
    returns a list of RequestedBundle objects that need to be generated and
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   784
    uploaded."""
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   785
    create_bundles = []
50431
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   786
    delete_bundles = []
50430
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   787
    repo = repo.filtered(b"immutable")
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   788
    targets = repo.ui.configlist(b'clone-bundles', b'auto-generate.formats')
50432
5b70b9f5a2f9 clone-bundles: only regenerate the clone bundle when cached ration is low
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50431
diff changeset
   789
    ratio = float(
5b70b9f5a2f9 clone-bundles: only regenerate the clone bundle when cached ration is low
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50431
diff changeset
   790
        repo.ui.config(b'clone-bundles', b'trigger.below-bundled-ratio')
5b70b9f5a2f9 clone-bundles: only regenerate the clone bundle when cached ration is low
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50431
diff changeset
   791
    )
50433
d611805e7374 clone-bundles: also control automation based on absolute number of revisions
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50432
diff changeset
   792
    abs_revs = repo.ui.configint(b'clone-bundles', b'trigger.revs')
50430
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   793
    revs = len(repo.changelog)
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   794
    generic_data = {
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   795
        'revs': revs,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   796
        'head_revs': repo.changelog.headrevs(),
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   797
        'tip_rev': repo.changelog.tiprev(),
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   798
        'tip_node': node.hex(repo.changelog.tip()),
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   799
        'op_id': op_id,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   800
    }
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   801
    for t in targets:
50433
d611805e7374 clone-bundles: also control automation based on absolute number of revisions
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50432
diff changeset
   802
        if new_bundle_needed(repo, bundles, ratio, abs_revs, t, revs):
50432
5b70b9f5a2f9 clone-bundles: only regenerate the clone bundle when cached ration is low
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50431
diff changeset
   803
            data = generic_data.copy()
5b70b9f5a2f9 clone-bundles: only regenerate the clone bundle when cached ration is low
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50431
diff changeset
   804
            data['bundle_type'] = t
5b70b9f5a2f9 clone-bundles: only regenerate the clone bundle when cached ration is low
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50431
diff changeset
   805
            b = RequestedBundle(**data)
5b70b9f5a2f9 clone-bundles: only regenerate the clone bundle when cached ration is low
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50431
diff changeset
   806
            create_bundles.append(b)
50431
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   807
    delete_bundles.extend(find_outdated_bundles(repo, bundles))
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   808
    return create_bundles, delete_bundles
50430
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   809
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   810
50433
d611805e7374 clone-bundles: also control automation based on absolute number of revisions
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50432
diff changeset
   811
def new_bundle_needed(repo, bundles, ratio, abs_revs, bundle_type, revs):
50432
5b70b9f5a2f9 clone-bundles: only regenerate the clone bundle when cached ration is low
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50431
diff changeset
   812
    """consider the current cached content and trigger new bundles if needed"""
50433
d611805e7374 clone-bundles: also control automation based on absolute number of revisions
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50432
diff changeset
   813
    threshold = max((revs * ratio), (revs - abs_revs))
50432
5b70b9f5a2f9 clone-bundles: only regenerate the clone bundle when cached ration is low
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50431
diff changeset
   814
    for b in bundles:
5b70b9f5a2f9 clone-bundles: only regenerate the clone bundle when cached ration is low
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50431
diff changeset
   815
        if not b.valid_for(repo) or b.bundle_type != bundle_type:
5b70b9f5a2f9 clone-bundles: only regenerate the clone bundle when cached ration is low
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50431
diff changeset
   816
            continue
5b70b9f5a2f9 clone-bundles: only regenerate the clone bundle when cached ration is low
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50431
diff changeset
   817
        if b.revs > threshold:
5b70b9f5a2f9 clone-bundles: only regenerate the clone bundle when cached ration is low
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50431
diff changeset
   818
            return False
5b70b9f5a2f9 clone-bundles: only regenerate the clone bundle when cached ration is low
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50431
diff changeset
   819
    return True
5b70b9f5a2f9 clone-bundles: only regenerate the clone bundle when cached ration is low
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50431
diff changeset
   820
5b70b9f5a2f9 clone-bundles: only regenerate the clone bundle when cached ration is low
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50431
diff changeset
   821
50430
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   822
def start_one_bundle(repo, bundle):
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   823
    """start the generation of a single bundle file
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   824
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   825
    the `bundle` argument should be a RequestedBundle object.
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   826
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   827
    This data is passed to the `debugmakeclonebundles` "as is".
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   828
    """
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   829
    data = util.pickle.dumps(bundle)
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   830
    cmd = [procutil.hgexecutable(), b'--cwd', repo.path, INTERNAL_CMD]
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   831
    env = procutil.shellenviron()
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   832
    msg = b'clone-bundles: starting bundle generation: %s\n'
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   833
    stdout = None
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   834
    stderr = None
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   835
    waits = []
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   836
    record_wait = None
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   837
    if repo.ui.configbool(b'devel', b'debug.clonebundles'):
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   838
        stdout = procutil.stdout
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   839
        stderr = procutil.stderr
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   840
        repo.ui.write(msg % bundle.bundle_type)
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   841
        record_wait = waits.append
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   842
    else:
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   843
        repo.ui.debug(msg % bundle.bundle_type)
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   844
    bg = procutil.runbgcommand
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   845
    bg(
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   846
        cmd,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   847
        env,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   848
        stdin_bytes=data,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   849
        stdout=stdout,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   850
        stderr=stderr,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   851
        record_wait=record_wait,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   852
    )
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   853
    for f in waits:
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   854
        f()
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   855
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   856
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   857
INTERNAL_CMD = b'debug::internal-make-clone-bundles'
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   858
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   859
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   860
@command(INTERNAL_CMD, [], b'')
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   861
def debugmakeclonebundles(ui, repo):
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   862
    """Internal command to auto-generate debug bundles"""
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   863
    requested_bundle = util.pickle.load(procutil.stdin)
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   864
    procutil.stdin.close()
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   865
50431
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   866
    collect_garbage(repo)
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   867
50430
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   868
    fname = requested_bundle.suggested_filename
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   869
    fpath = repo.vfs.makedirs(b'tmp-bundles')
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   870
    fpath = repo.vfs.join(b'tmp-bundles', fname)
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   871
    bundle = requested_bundle.generating(fpath)
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   872
    update_bundle_list(repo, new_bundles=[bundle])
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   873
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   874
    requested_bundle.generate_bundle(repo, fpath)
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   875
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   876
    repo.invalidate()
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   877
    finalize_one_bundle(repo, bundle)
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   878
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   879
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   880
def make_auto_bundler(source_repo):
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   881
    reporef = weakref.ref(source_repo)
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   882
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   883
    def autobundle(tr):
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   884
        repo = reporef()
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   885
        assert repo is not None
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   886
        bundles = read_auto_gen(repo)
50431
971dc2369b04 clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50430
diff changeset
   887
        new, __ = auto_bundle_needed_actions(repo, bundles, b"%d_txn" % id(tr))
50430
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   888
        for data in new:
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   889
            start_one_bundle(repo, data)
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   890
        return None
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   891
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   892
    return autobundle
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   893
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   894
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   895
def reposetup(ui, repo):
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   896
    """install the two pieces needed for automatic clonebundle generation
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   897
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   898
    - add a "post-close" hook that fires bundling when needed
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   899
    - introduce a clone-bundle lock to let multiple processes meddle with the
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   900
      state files.
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   901
    """
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   902
    if not repo.local():
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   903
        return
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   904
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   905
    class autobundlesrepo(repo.__class__):
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   906
        def transaction(self, *args, **kwargs):
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   907
            tr = super(autobundlesrepo, self).transaction(*args, **kwargs)
50434
65fb4cedd5ea clone-bundles: add a configuration to control auto-generation on changes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50433
diff changeset
   908
            enabled = repo.ui.configbool(
65fb4cedd5ea clone-bundles: add a configuration to control auto-generation on changes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50433
diff changeset
   909
                b'clone-bundles',
65fb4cedd5ea clone-bundles: add a configuration to control auto-generation on changes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50433
diff changeset
   910
                b'auto-generate.on-change',
65fb4cedd5ea clone-bundles: add a configuration to control auto-generation on changes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50433
diff changeset
   911
            )
50430
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   912
            targets = repo.ui.configlist(
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   913
                b'clone-bundles', b'auto-generate.formats'
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   914
            )
50434
65fb4cedd5ea clone-bundles: add a configuration to control auto-generation on changes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50433
diff changeset
   915
            if enabled and targets:
50430
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   916
                tr.addpostclose(CAT_POSTCLOSE, make_auto_bundler(self))
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   917
            return tr
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   918
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   919
        @localrepo.unfilteredmethod
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   920
        def clonebundles_lock(self, wait=True):
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   921
            '''Lock the repository file related to clone bundles'''
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   922
            if not util.safehasattr(self, '_cb_lock_ref'):
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   923
                self._cb_lock_ref = None
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   924
            l = self._currentlock(self._cb_lock_ref)
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   925
            if l is not None:
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   926
                l.lock()
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   927
                return l
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   928
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   929
            l = self._lock(
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   930
                vfs=self.vfs,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   931
                lockname=b"clonebundleslock",
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   932
                wait=wait,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   933
                releasefn=None,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   934
                acquirefn=None,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   935
                desc=_(b'repository %s') % self.origroot,
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   936
            )
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   937
            self._cb_lock_ref = weakref.ref(l)
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   938
            return l
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   939
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   940
    repo._wlockfreeprefix.add(AUTO_GEN_FILE)
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   941
    repo._wlockfreeprefix.add(bundlecaches.CB_MANIFEST_FILE)
5ae30ff79c76 clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48875
diff changeset
   942
    repo.__class__ = autobundlesrepo
50435
23db5f15cc0c clone-bundles: introduce a command to refresh bundle
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50434
diff changeset
   943
23db5f15cc0c clone-bundles: introduce a command to refresh bundle
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50434
diff changeset
   944
50438
3973b1dc3ee3 clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50437
diff changeset
   945
@command(
3973b1dc3ee3 clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50437
diff changeset
   946
    b'admin::clone-bundles-refresh',
3973b1dc3ee3 clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50437
diff changeset
   947
    [
3973b1dc3ee3 clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50437
diff changeset
   948
        (
3973b1dc3ee3 clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50437
diff changeset
   949
            b'',
3973b1dc3ee3 clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50437
diff changeset
   950
            b'background',
3973b1dc3ee3 clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50437
diff changeset
   951
            False,
3973b1dc3ee3 clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50437
diff changeset
   952
            _(b'start bundle generation in the background'),
3973b1dc3ee3 clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50437
diff changeset
   953
        ),
3973b1dc3ee3 clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50437
diff changeset
   954
    ],
3973b1dc3ee3 clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50437
diff changeset
   955
    b'',
3973b1dc3ee3 clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50437
diff changeset
   956
)
3973b1dc3ee3 clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50437
diff changeset
   957
def cmd_admin_clone_bundles_refresh(
3973b1dc3ee3 clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50437
diff changeset
   958
    ui,
3973b1dc3ee3 clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50437
diff changeset
   959
    repo: localrepo.localrepository,
3973b1dc3ee3 clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50437
diff changeset
   960
    background=False,
3973b1dc3ee3 clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50437
diff changeset
   961
):
50435
23db5f15cc0c clone-bundles: introduce a command to refresh bundle
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50434
diff changeset
   962
    """generate clone bundles according to the configuration
23db5f15cc0c clone-bundles: introduce a command to refresh bundle
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50434
diff changeset
   963
23db5f15cc0c clone-bundles: introduce a command to refresh bundle
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50434
diff changeset
   964
    This runs the logic for automatic generation, removing outdated bundles and
23db5f15cc0c clone-bundles: introduce a command to refresh bundle
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50434
diff changeset
   965
    generating new ones if necessary. See :hg:`help -e clone-bundles` for
23db5f15cc0c clone-bundles: introduce a command to refresh bundle
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50434
diff changeset
   966
    details about how to configure this feature.
23db5f15cc0c clone-bundles: introduce a command to refresh bundle
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50434
diff changeset
   967
    """
23db5f15cc0c clone-bundles: introduce a command to refresh bundle
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50434
diff changeset
   968
    debug = repo.ui.configbool(b'devel', b'debug.clonebundles')
23db5f15cc0c clone-bundles: introduce a command to refresh bundle
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50434
diff changeset
   969
    bundles = read_auto_gen(repo)
23db5f15cc0c clone-bundles: introduce a command to refresh bundle
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50434
diff changeset
   970
    op_id = b"%d_acbr" % os.getpid()
23db5f15cc0c clone-bundles: introduce a command to refresh bundle
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50434
diff changeset
   971
    create, delete = auto_bundle_needed_actions(repo, bundles, op_id)
23db5f15cc0c clone-bundles: introduce a command to refresh bundle
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50434
diff changeset
   972
50438
3973b1dc3ee3 clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50437
diff changeset
   973
    # if some bundles are scheduled for creation in the background, they will
3973b1dc3ee3 clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50437
diff changeset
   974
    # deal with garbage collection too, so no need to synchroniously do it.
3973b1dc3ee3 clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50437
diff changeset
   975
    #
3973b1dc3ee3 clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50437
diff changeset
   976
    # However if no bundles are scheduled for creation, we need to explicitly do
3973b1dc3ee3 clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50437
diff changeset
   977
    # it here.
3973b1dc3ee3 clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50437
diff changeset
   978
    if not (background and create):
3973b1dc3ee3 clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50437
diff changeset
   979
        # we clean up outdated bundles before generating new ones to keep the
3973b1dc3ee3 clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50437
diff changeset
   980
        # last two versions of the bundle around for a while and avoid having to
3973b1dc3ee3 clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50437
diff changeset
   981
        # deal with clients that just got served a manifest.
3973b1dc3ee3 clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50437
diff changeset
   982
        for o in delete:
3973b1dc3ee3 clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50437
diff changeset
   983
            delete_bundle(repo, o)
3973b1dc3ee3 clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50437
diff changeset
   984
        update_bundle_list(repo, del_bundles=delete)
50435
23db5f15cc0c clone-bundles: introduce a command to refresh bundle
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50434
diff changeset
   985
23db5f15cc0c clone-bundles: introduce a command to refresh bundle
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50434
diff changeset
   986
    if create:
23db5f15cc0c clone-bundles: introduce a command to refresh bundle
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50434
diff changeset
   987
        fpath = repo.vfs.makedirs(b'tmp-bundles')
50438
3973b1dc3ee3 clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50437
diff changeset
   988
3973b1dc3ee3 clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50437
diff changeset
   989
    if background:
3973b1dc3ee3 clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50437
diff changeset
   990
        for requested_bundle in create:
3973b1dc3ee3 clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50437
diff changeset
   991
            start_one_bundle(repo, requested_bundle)
3973b1dc3ee3 clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50437
diff changeset
   992
    else:
3973b1dc3ee3 clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50437
diff changeset
   993
        for requested_bundle in create:
3973b1dc3ee3 clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50437
diff changeset
   994
            if debug:
3973b1dc3ee3 clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50437
diff changeset
   995
                msg = b'clone-bundles: starting bundle generation: %s\n'
3973b1dc3ee3 clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50437
diff changeset
   996
                repo.ui.write(msg % requested_bundle.bundle_type)
3973b1dc3ee3 clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50437
diff changeset
   997
            fname = requested_bundle.suggested_filename
3973b1dc3ee3 clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50437
diff changeset
   998
            fpath = repo.vfs.join(b'tmp-bundles', fname)
3973b1dc3ee3 clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50437
diff changeset
   999
            generating_bundle = requested_bundle.generating(fpath)
3973b1dc3ee3 clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50437
diff changeset
  1000
            update_bundle_list(repo, new_bundles=[generating_bundle])
3973b1dc3ee3 clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50437
diff changeset
  1001
            requested_bundle.generate_bundle(repo, fpath)
3973b1dc3ee3 clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50437
diff changeset
  1002
            result = upload_bundle(repo, generating_bundle)
3973b1dc3ee3 clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50437
diff changeset
  1003
            update_bundle_list(repo, new_bundles=[result])
3973b1dc3ee3 clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50437
diff changeset
  1004
            update_ondisk_manifest(repo)
3973b1dc3ee3 clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50437
diff changeset
  1005
            cleanup_tmp_bundle(repo, generating_bundle)
50437
10318b88b1d6 clone-bundles: add a command to clear all bundles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50435
diff changeset
  1006
10318b88b1d6 clone-bundles: add a command to clear all bundles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50435
diff changeset
  1007
10318b88b1d6 clone-bundles: add a command to clear all bundles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50435
diff changeset
  1008
@command(b'admin::clone-bundles-clear', [], b'')
10318b88b1d6 clone-bundles: add a command to clear all bundles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50435
diff changeset
  1009
def cmd_admin_clone_bundles_clear(ui, repo: localrepo.localrepository):
10318b88b1d6 clone-bundles: add a command to clear all bundles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50435
diff changeset
  1010
    """remove existing clone bundle caches
10318b88b1d6 clone-bundles: add a command to clear all bundles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50435
diff changeset
  1011
10318b88b1d6 clone-bundles: add a command to clear all bundles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50435
diff changeset
  1012
    See `hg help admin::clone-bundles-refresh` for details on how to regenerate
10318b88b1d6 clone-bundles: add a command to clear all bundles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50435
diff changeset
  1013
    them.
10318b88b1d6 clone-bundles: add a command to clear all bundles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50435
diff changeset
  1014
10318b88b1d6 clone-bundles: add a command to clear all bundles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50435
diff changeset
  1015
    This command will only affect bundles currently available, it will not
10318b88b1d6 clone-bundles: add a command to clear all bundles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50435
diff changeset
  1016
    affect bundles being asynchronously generated.
10318b88b1d6 clone-bundles: add a command to clear all bundles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50435
diff changeset
  1017
    """
10318b88b1d6 clone-bundles: add a command to clear all bundles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50435
diff changeset
  1018
    bundles = read_auto_gen(repo)
10318b88b1d6 clone-bundles: add a command to clear all bundles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50435
diff changeset
  1019
    delete = [b for b in bundles if b.ready]
10318b88b1d6 clone-bundles: add a command to clear all bundles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50435
diff changeset
  1020
    for o in delete:
10318b88b1d6 clone-bundles: add a command to clear all bundles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50435
diff changeset
  1021
        delete_bundle(repo, o)
10318b88b1d6 clone-bundles: add a command to clear all bundles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50435
diff changeset
  1022
    update_bundle_list(repo, del_bundles=delete)