annotate hgext/clonebundles.py @ 51711:832a1aeb576f

pytype: only try the hacky way of finding PYTHON if not provided This allows us to work in more environments, like when using pyenv. This syntax is compatible with all POSIX shells.
author Raphaël Gomès <rgomes@octobus.net>
date Tue, 23 Jul 2024 12:10:31 +0200
parents d718eddf01d9
children f4733654f144
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
26623
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
1 # This software may be used and distributed according to the terms of the
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
2 # GNU General Public License version 2 or any later version.
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
3
27738
a0e783d26e81 exchange: make clone bundles non-experimental and enabled by default
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27737
diff changeset
4 """advertise pre-generated bundles to seed clones
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
5
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
6 "clonebundles" is a server-side extension used to advertise the existence
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
7 of pre-generated, externally hosted bundle files to clients that are
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
8 cloning so that cloning can be faster, more reliable, and require less
37498
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
9 resources on the server. "pullbundles" is a related feature for sending
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
10 pre-generated bundle files to clients as part of pull operations.
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
11
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
12 Cloning can be a CPU and I/O intensive operation on servers. Traditionally,
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
13 the server, in response to a client's request to clone, dynamically generates
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
14 a bundle containing the entire repository content and sends it to the client.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
15 There is no caching on the server and the server will have to redundantly
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
16 generate the same outgoing bundle in response to each clone request. For
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
17 servers with large repositories or with high clone volume, the load from
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
18 clones can make scaling the server challenging and costly.
26623
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
19
37498
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
20 This extension provides server operators the ability to offload
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
21 potentially expensive clone load to an external service. Pre-generated
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
22 bundles also allow using more CPU intensive compression, reducing the
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
23 effective bandwidth requirements.
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
24
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
25 Here's how clone bundles work:
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
26
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
27 1. A server operator establishes a mechanism for making bundle files available
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
28 on a hosting service where Mercurial clients can fetch them.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
29 2. A manifest file listing available bundle URLs and some optional metadata
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
30 is added to the Mercurial repository on the server.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
31 3. A client initiates a clone against a clone bundles aware server.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
32 4. The client sees the server is advertising clone bundles and fetches the
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
33 manifest listing available bundles.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
34 5. The client filters and sorts the available bundles based on what it
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
35 supports and prefers.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
36 6. The client downloads and applies an available bundle from the
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
37 server-specified URL.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
38 7. The client reconnects to the original server and performs the equivalent
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
39 of :hg:`pull` to retrieve all repository data not in the bundle. (The
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
40 repository could have been updated between when the bundle was created
37498
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
41 and when the client started the clone.) This may use "pullbundles".
26623
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
42
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
43 Instead of the server generating full repository bundles for every clone
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
44 request, it generates full bundles once and they are subsequently reused to
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
45 bootstrap new clones. The server may still transfer data at clone time.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
46 However, this is only data that has been added/changed since the bundle was
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
47 created. For large, established repositories, this can reduce server load for
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
48 clones to less than 1% of original.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
49
37498
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
50 Here's how pullbundles work:
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
51
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
52 1. A manifest file listing available bundles and describing the revisions
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
53 is added to the Mercurial repository on the server.
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
54 2. A new-enough client informs the server that it supports partial pulls
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
55 and initiates a pull.
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
56 3. If the server has pull bundles enabled and sees the client advertising
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
57 partial pulls, it checks for a matching pull bundle in the manifest.
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
58 A bundle matches if the format is supported by the client, the client
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
59 has the required revisions already and needs something from the bundle.
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
60 4. If there is at least one matching bundle, the server sends it to the client.
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
61 5. The client applies the bundle and notices that the server reply was
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
62 incomplete. It initiates another pull.
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
63
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
64 To work, this extension requires the following of server operators:
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
65
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
66 * Generating bundle files of repository content (typically periodically,
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
67 such as once per day).
37498
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
68 * Clone bundles: A file server that clients have network access to and that
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
69 Python knows how to talk to through its normal URL handling facility
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
70 (typically an HTTP/HTTPS server).
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
71 * A process for keeping the bundles manifest in sync with available bundle
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
72 files.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
73
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
74 Strictly speaking, using a static file hosting server isn't required: a server
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
75 operator could use a dynamic service for retrieving bundle data. However,
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
76 static file hosting services are simple and scalable and should be sufficient
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
77 for most needs.
26623
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
78
26884
762bf510b42c clonebundles: fix typo s/comand/command/
Javi Merino <merino.jav@gmail.com>
parents: 26857
diff changeset
79 Bundle files can be generated with the :hg:`bundle` command. Typically
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
80 :hg:`bundle --all` is used to produce a bundle of the entire repository.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
81
50645
bf7404f2e22d bundles: clarify streaming v2 bundle usage
Mathias De Mare <mathias.de_mare@nokia.com>
parents: 48875
diff changeset
82 The bundlespec option `stream` (see :hg:`help bundlespec`)
bf7404f2e22d bundles: clarify streaming v2 bundle usage
Mathias De Mare <mathias.de_mare@nokia.com>
parents: 48875
diff changeset
83 can be used to produce a special *streaming clonebundle*, typically using
bf7404f2e22d bundles: clarify streaming v2 bundle usage
Mathias De Mare <mathias.de_mare@nokia.com>
parents: 48875
diff changeset
84 :hg:`bundle --all --type="none-streamv2"`.
bf7404f2e22d bundles: clarify streaming v2 bundle usage
Mathias De Mare <mathias.de_mare@nokia.com>
parents: 48875
diff changeset
85 These are bundle files that are extremely efficient
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
86 to produce and consume (read: fast). However, they are larger than
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
87 traditional bundle formats and require that clients support the exact set
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
88 of repository data store formats in use by the repository that created them.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
89 Typically, a newer server can serve data that is compatible with older clients.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
90 However, *streaming clone bundles* don't have this guarantee. **Server
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
91 operators need to be aware that newer versions of Mercurial may produce
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
92 streaming clone bundles incompatible with older Mercurial versions.**
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
93
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
94 A server operator is responsible for creating a ``.hg/clonebundles.manifest``
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
95 file containing the list of available bundle files suitable for seeding
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
96 clones. If this file does not exist, the repository will not advertise the
37498
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
97 existence of clone bundles when clients connect. For pull bundles,
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
98 ``.hg/pullbundles.manifest`` is used.
26623
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
99
32454
702af1ad3b18 clonebundles: fix missing newline character
Matt Harbison <matt_harbison@yahoo.com>
parents: 31146
diff changeset
100 The manifest file contains a newline (\\n) delimited list of entries.
26623
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
101
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
102 Each line in this file defines an available bundle. Lines have the format:
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
103
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
104 <URL> [<key>=<value>[ <key>=<value>]]
26623
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
105
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
106 That is, a URL followed by an optional, space-delimited list of key=value
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
107 pairs describing additional properties of this bundle. Both keys and values
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
108 are URI encoded.
26623
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
109
37498
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
110 For pull bundles, the URL is a path under the ``.hg`` directory of the
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
111 repository.
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
112
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
113 Keys in UPPERCASE are reserved for use by Mercurial and are defined below.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
114 All non-uppercase keys can be used by site installations. An example use
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
115 for custom properties is to use the *datacenter* attribute to define which
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
116 data center a file is hosted in. Clients could then prefer a server in the
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
117 data center closest to them.
26623
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
118
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
119 The following reserved keys are currently defined:
26623
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
120
26644
74de1c59f71c clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26623
diff changeset
121 BUNDLESPEC
74de1c59f71c clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26623
diff changeset
122 A "bundle specification" string that describes the type of the bundle.
74de1c59f71c clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26623
diff changeset
123
74de1c59f71c clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26623
diff changeset
124 These are string values that are accepted by the "--type" argument of
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
125 :hg:`bundle`.
26644
74de1c59f71c clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26623
diff changeset
126
74de1c59f71c clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26623
diff changeset
127 The values are parsed in strict mode, which means they must be of the
74de1c59f71c clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26623
diff changeset
128 "<compression>-<type>" form. See
74de1c59f71c clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26623
diff changeset
129 mercurial.exchange.parsebundlespec() for more details.
74de1c59f71c clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26623
diff changeset
130
27886
0288e63ea3be clonebundles: improve BUNDLESPEC documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27738
diff changeset
131 :hg:`debugbundle --spec` can be used to print the bundle specification
0288e63ea3be clonebundles: improve BUNDLESPEC documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27738
diff changeset
132 string for a bundle file. The output of this command can be used verbatim
0288e63ea3be clonebundles: improve BUNDLESPEC documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27738
diff changeset
133 for the value of ``BUNDLESPEC`` (it is already escaped).
0288e63ea3be clonebundles: improve BUNDLESPEC documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27738
diff changeset
134
26644
74de1c59f71c clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26623
diff changeset
135 Clients will automatically filter out specifications that are unknown or
74de1c59f71c clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26623
diff changeset
136 unsupported so they won't attempt to download something that likely won't
74de1c59f71c clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26623
diff changeset
137 apply.
74de1c59f71c clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26623
diff changeset
138
74de1c59f71c clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26623
diff changeset
139 The actual value doesn't impact client behavior beyond filtering:
74de1c59f71c clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26623
diff changeset
140 clients will still sniff the bundle type from the header of downloaded
74de1c59f71c clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26623
diff changeset
141 files.
26645
2faa7671a4b3 clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26644
diff changeset
142
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
143 **Use of this key is highly recommended**, as it allows clients to
27886
0288e63ea3be clonebundles: improve BUNDLESPEC documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27738
diff changeset
144 easily skip unsupported bundles. If this key is not defined, an old
0288e63ea3be clonebundles: improve BUNDLESPEC documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27738
diff changeset
145 client may attempt to apply a bundle that it is incapable of reading.
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
146
26645
2faa7671a4b3 clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26644
diff changeset
147 REQUIRESNI
2faa7671a4b3 clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26644
diff changeset
148 Whether Server Name Indication (SNI) is required to connect to the URL.
2faa7671a4b3 clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26644
diff changeset
149 SNI allows servers to use multiple certificates on the same IP. It is
2faa7671a4b3 clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26644
diff changeset
150 somewhat common in CDNs and other hosting providers. Older Python
2faa7671a4b3 clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26644
diff changeset
151 versions do not support SNI. Defining this attribute enables clients
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
152 with older Python versions to filter this entry without experiencing
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
153 an opaque SSL failure at connection time.
26645
2faa7671a4b3 clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26644
diff changeset
154
2faa7671a4b3 clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26644
diff changeset
155 If this is defined, it is important to advertise a non-SNI fallback
2faa7671a4b3 clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26644
diff changeset
156 URL or clients running old Python releases may not be able to clone
2faa7671a4b3 clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26644
diff changeset
157 with the clonebundles facility.
2faa7671a4b3 clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26644
diff changeset
158
2faa7671a4b3 clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26644
diff changeset
159 Value should be "true".
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
160
45252
20d110e6eea6 clonebundles: document REQUIREDRAM key
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43077
diff changeset
161 REQUIREDRAM
20d110e6eea6 clonebundles: document REQUIREDRAM key
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43077
diff changeset
162 Value specifies expected memory requirements to decode the payload.
20d110e6eea6 clonebundles: document REQUIREDRAM key
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43077
diff changeset
163 Values can have suffixes for common bytes sizes. e.g. "64MB".
20d110e6eea6 clonebundles: document REQUIREDRAM key
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43077
diff changeset
164
20d110e6eea6 clonebundles: document REQUIREDRAM key
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43077
diff changeset
165 This key is often used with zstd-compressed bundles using a high
20d110e6eea6 clonebundles: document REQUIREDRAM key
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43077
diff changeset
166 compression level / window size, which can require 100+ MB of memory
20d110e6eea6 clonebundles: document REQUIREDRAM key
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43077
diff changeset
167 to decode.
20d110e6eea6 clonebundles: document REQUIREDRAM key
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43077
diff changeset
168
37498
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
169 heads
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
170 Used for pull bundles. This contains the ``;`` separated changeset
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
171 hashes of the heads of the bundle content.
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
172
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
173 bases
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
174 Used for pull bundles. This contains the ``;`` separated changeset
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
175 hashes of the roots of the bundle content. This can be skipped if
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
176 the bundle was created without ``--base``.
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
177
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
178 Manifests can contain multiple entries. Assuming metadata is defined, clients
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
179 will filter entries from the manifest that they don't support. The remaining
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
180 entries are optionally sorted by client preferences
32773
d25802b0eef5 clonebundles: reference correct config option
Gregory Szorc <gregory.szorc@gmail.com>
parents: 32454
diff changeset
181 (``ui.clonebundleprefers`` config option). The client then attempts
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
182 to fetch the bundle at the first URL in the remaining list.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
183
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
184 **Errors when downloading a bundle will fail the entire clone operation:
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
185 clients do not automatically fall back to a traditional clone.** The reason
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
186 for this is that if a server is using clone bundles, it is probably doing so
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
187 because the feature is necessary to help it scale. In other words, there
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
188 is an assumption that clone load will be offloaded to another service and
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
189 that the Mercurial server isn't responsible for serving this clone load.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
190 If that other service experiences issues and clients start mass falling back to
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
191 the original Mercurial server, the added clone load could overwhelm the server
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
192 due to unexpected load and effectively take it offline. Not having clients
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
193 automatically fall back to cloning from the original server mitigates this
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
194 scenario.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
195
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
196 Because there is no automatic Mercurial server fallback on failure of the
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
197 bundle hosting service, it is important for server operators to view the bundle
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
198 hosting service as an extension of the Mercurial server in terms of
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
199 availability and service level agreements: if the bundle hosting service goes
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
200 down, so does the ability for clients to clone. Note: clients will see a
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
201 message informing them how to bypass the clone bundles facility when a failure
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
202 occurs. So server operators should prepare for some people to follow these
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
203 instructions when a failure occurs, thus driving more load to the original
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
204 Mercurial server when the bundle hosting service fails.
50686
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
205
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
206
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
207 inline clonebundles
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
208 -------------------
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
209
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
210 It is possible to transmit clonebundles inline in case repositories are
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
211 accessed over SSH. This avoids having to setup an external HTTPS server
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
212 and results in the same access control as already present for the SSH setup.
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
213
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
214 Inline clonebundles should be placed into the `.hg/bundle-cache` directory.
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
215 A clonebundle at `.hg/bundle-cache/mybundle.bundle` is referred to
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
216 in the `clonebundles.manifest` file as `peer-bundle-cache://mybundle.bundle`.
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
217
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
218
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
219 auto-generation of clone bundles
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
220 --------------------------------
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
221
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
222 It is possible to set Mercurial to automatically re-generate clone bundles when
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
223 enough new content is available.
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
224
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
225 Mercurial will take care of the process asynchronously. The defined list of
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
226 bundle-type will be generated, uploaded, and advertised. Older bundles will get
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
227 decommissioned as newer ones replace them.
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
228
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
229 Bundles Generation:
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
230 ...................
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
231
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
232 The extension can generate multiple variants of the clone bundle. Each
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
233 different variant will be defined by the "bundle-spec" they use::
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
234
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
235 [clone-bundles]
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
236 auto-generate.formats= zstd-v2, gzip-v2
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
237
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
238 See `hg help bundlespec` for details about available options.
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
239
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
240 By default, new bundles are generated when 5% of the repository contents or at
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
241 least 1000 revisions are not contained in the cached bundles. This option can
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
242 be controlled by the `clone-bundles.trigger.below-bundled-ratio` option
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
243 (default 0.95) and the `clone-bundles.trigger.revs` option (default 1000)::
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
244
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
245 [clone-bundles]
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
246 trigger.below-bundled-ratio=0.95
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
247 trigger.revs=1000
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
248
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
249 This logic can be manually triggered using the `admin::clone-bundles-refresh`
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
250 command, or automatically on each repository change if
50735
2b0598121a71 clonebundles: fix display of auto-generate.on-change lines
Mathias De Mare <mathias.de_mare@nokia.com>
parents: 50713
diff changeset
251 `clone-bundles.auto-generate.on-change` is set to `yes`::
50686
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
252
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
253 [clone-bundles]
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
254 auto-generate.on-change=yes
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
255 auto-generate.formats= zstd-v2, gzip-v2
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
256
50711
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
257 Automatic Inline serving
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
258 ........................
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
259
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
260 The simplest way to serve the generated bundle is through the Mercurial
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
261 protocol. However it is not the most efficient as request will still be served
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
262 by that main server. It is useful in case where authentication is complexe or
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
263 when an efficient mirror system is already in use anyway. See the `inline
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
264 clonebundles` section above for details about inline clonebundles
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
265
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
266 To automatically serve generated bundle through inline clonebundle, simply set
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
267 the following option::
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
268
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
269 auto-generate.serve-inline=yes
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
270
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
271 Enabling this option disable the managed upload and serving explained below.
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
272
50686
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
273 Bundles Upload and Serving:
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
274 ...........................
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
275
50711
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
276 This is the most efficient way to serve automatically generated clone bundles,
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
277 but requires some setup.
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
278
50686
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
279 The generated bundles need to be made available to users through a "public" URL.
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
280 This should be donne through `clone-bundles.upload-command` configuration. The
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
281 value of this command should be a shell command. It will have access to the
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
282 bundle file path through the `$HGCB_BUNDLE_PATH` variable. And the expected
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
283 basename in the "public" URL is accessible at::
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
284
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
285 [clone-bundles]
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
286 upload-command=sftp put $HGCB_BUNDLE_PATH \
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
287 sftp://bundles.host/clone-bundles/$HGCB_BUNDLE_BASENAME
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
288
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
289 If the file was already uploaded, the command must still succeed.
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
290
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
291 After upload, the file should be available at an url defined by
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
292 `clone-bundles.url-template`.
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
293
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
294 [clone-bundles]
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
295 url-template=https://bundles.host/cache/clone-bundles/{basename}
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
296
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
297 Old bundles cleanup:
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
298 ....................
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
299
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
300 When new bundles are generated, the older ones are no longer necessary and can
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
301 be removed from storage. This is done through the `clone-bundles.delete-command`
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
302 configuration. The command is given the url of the artifact to delete through
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
303 the `$HGCB_BUNDLE_URL` environment variable.
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
304
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
305 [clone-bundles]
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
306 delete-command=sftp rm sftp://bundles.host/clone-bundles/$HGCB_BUNDLE_BASENAME
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
307
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
308 If the file was already deleted, the command must still succeed.
26623
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
309 """
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
310
28095
7fa139eaebb4 clonebundles: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27886
diff changeset
311
50686
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
312 import os
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
313 import weakref
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
314
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
315 from mercurial.i18n import _
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
316
26623
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
317 from mercurial import (
45785
80f32ec8653a clonebundle: move the manifest filename to a constant
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 45252
diff changeset
318 bundlecaches,
50686
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
319 commands,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
320 error,
26623
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
321 extensions,
50686
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
322 localrepo,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
323 lock,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
324 node,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
325 registrar,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
326 util,
37785
b4d85bc122bd wireproto: rename wireproto to wireprotov1server (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37498
diff changeset
327 wireprotov1server,
26623
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
328 )
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
329
50686
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
330
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
331 from mercurial.utils import (
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
332 procutil,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
333 )
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
334
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
335 testedwith = b'ships-with-hg-core'
26623
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
336
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 37785
diff changeset
337
26623
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
338 def capabilities(orig, repo, proto):
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
339 caps = orig(repo, proto)
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
340
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
341 # Only advertise if a manifest exists. This does add some I/O to requests.
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
342 # But this should be cheaper than a wasted network round trip due to
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
343 # missing file.
45785
80f32ec8653a clonebundle: move the manifest filename to a constant
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 45252
diff changeset
344 if repo.vfs.exists(bundlecaches.CB_MANIFEST_FILE):
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
345 caps.append(b'clonebundles')
50701
4238e6b22fc4 clonebundles: introduce a new write protocol command
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50686
diff changeset
346 caps.append(b'clonebundles_manifest')
26623
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
347
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
348 return caps
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
349
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 37785
diff changeset
350
26623
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
351 def extsetup(ui):
50776
335aeb3d670b wrapfunction: use sysstr instead of bytes as argument in "clonebundles"
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50736
diff changeset
352 extensions.wrapfunction(wireprotov1server, '_capabilities', capabilities)
50686
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
353
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
354
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
355 # logic for bundle auto-generation
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
356
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
357
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
358 configtable = {}
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
359 configitem = registrar.configitem(configtable)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
360
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
361 cmdtable = {}
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
362 command = registrar.command(cmdtable)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
363
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
364 configitem(b'clone-bundles', b'auto-generate.on-change', default=False)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
365 configitem(b'clone-bundles', b'auto-generate.formats', default=list)
50711
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
366 configitem(b'clone-bundles', b'auto-generate.serve-inline', default=False)
50686
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
367 configitem(b'clone-bundles', b'trigger.below-bundled-ratio', default=0.95)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
368 configitem(b'clone-bundles', b'trigger.revs', default=1000)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
369
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
370 configitem(b'clone-bundles', b'upload-command', default=None)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
371
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
372 configitem(b'clone-bundles', b'delete-command', default=None)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
373
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
374 configitem(b'clone-bundles', b'url-template', default=None)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
375
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
376 configitem(b'devel', b'debug.clonebundles', default=False)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
377
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
378
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
379 # category for the post-close transaction hooks
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
380 CAT_POSTCLOSE = b"clonebundles-autobundles"
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
381
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
382 # template for bundle file names
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
383 BUNDLE_MASK = (
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
384 b"full-%(bundle_type)s-%(revs)d_revs-%(tip_short)s_tip-%(op_id)s.hg"
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
385 )
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
386
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
387
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
388 # file in .hg/ use to track clonebundles being auto-generated
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
389 AUTO_GEN_FILE = b'clonebundles.auto-gen'
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
390
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
391
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
392 class BundleBase(object):
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
393 """represents the core of properties that matters for us in a bundle
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
394
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
395 :bundle_type: the bundlespec (see hg help bundlespec)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
396 :revs: the number of revisions in the repo at bundle creation time
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
397 :tip_rev: the rev-num of the tip revision
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
398 :tip_node: the node id of the tip-most revision in the bundle
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
399
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
400 :ready: True if the bundle is ready to be served
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
401 """
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
402
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
403 ready = False
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
404
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
405 def __init__(self, bundle_type, revs, tip_rev, tip_node):
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
406 self.bundle_type = bundle_type
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
407 self.revs = revs
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
408 self.tip_rev = tip_rev
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
409 self.tip_node = tip_node
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
410
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
411 def valid_for(self, repo):
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
412 """is this bundle applicable to the current repository
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
413
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
414 This is useful for detecting bundles made irrelevant by stripping.
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
415 """
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
416 tip_node = node.bin(self.tip_node)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
417 return repo.changelog.index.get_rev(tip_node) == self.tip_rev
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
418
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
419 def __eq__(self, other):
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
420 left = (self.ready, self.bundle_type, self.tip_rev, self.tip_node)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
421 right = (other.ready, other.bundle_type, other.tip_rev, other.tip_node)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
422 return left == right
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
423
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
424 def __neq__(self, other):
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
425 return not self == other
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
426
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
427 def __cmp__(self, other):
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
428 if self == other:
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
429 return 0
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
430 return -1
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
431
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
432
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
433 class RequestedBundle(BundleBase):
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
434 """A bundle that should be generated.
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
435
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
436 Additional attributes compared to BundleBase
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
437 :heads: list of head revisions (as rev-num)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
438 :op_id: a "unique" identifier for the operation triggering the change
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
439 """
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
440
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
441 def __init__(self, bundle_type, revs, tip_rev, tip_node, head_revs, op_id):
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
442 self.head_revs = head_revs
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
443 self.op_id = op_id
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
444 super(RequestedBundle, self).__init__(
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
445 bundle_type,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
446 revs,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
447 tip_rev,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
448 tip_node,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
449 )
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
450
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
451 @property
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
452 def suggested_filename(self):
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
453 """A filename that can be used for the generated bundle"""
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
454 data = {
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
455 b'bundle_type': self.bundle_type,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
456 b'revs': self.revs,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
457 b'heads': self.head_revs,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
458 b'tip_rev': self.tip_rev,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
459 b'tip_node': self.tip_node,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
460 b'tip_short': self.tip_node[:12],
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
461 b'op_id': self.op_id,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
462 }
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
463 return BUNDLE_MASK % data
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
464
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
465 def generate_bundle(self, repo, file_path):
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
466 """generate the bundle at `filepath`"""
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
467 commands.bundle(
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
468 repo.ui,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
469 repo,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
470 file_path,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
471 base=[b"null"],
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
472 rev=self.head_revs,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
473 type=self.bundle_type,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
474 quiet=True,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
475 )
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
476
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
477 def generating(self, file_path, hostname=None, pid=None):
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
478 """return a GeneratingBundle object from this object"""
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
479 if pid is None:
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
480 pid = os.getpid()
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
481 if hostname is None:
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
482 hostname = lock._getlockprefix()
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
483 return GeneratingBundle(
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
484 self.bundle_type,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
485 self.revs,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
486 self.tip_rev,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
487 self.tip_node,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
488 hostname,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
489 pid,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
490 file_path,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
491 )
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
492
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
493
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
494 class GeneratingBundle(BundleBase):
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
495 """A bundle being generated
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
496
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
497 extra attributes compared to BundleBase:
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
498
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
499 :hostname: the hostname of the machine generating the bundle
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
500 :pid: the pid of the process generating the bundle
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
501 :filepath: the target filename of the bundle
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
502
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
503 These attributes exist to help detect stalled generation processes.
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
504 """
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
505
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
506 ready = False
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
507
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
508 def __init__(
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
509 self, bundle_type, revs, tip_rev, tip_node, hostname, pid, filepath
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
510 ):
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
511 self.hostname = hostname
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
512 self.pid = pid
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
513 self.filepath = filepath
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
514 super(GeneratingBundle, self).__init__(
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
515 bundle_type, revs, tip_rev, tip_node
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
516 )
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
517
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
518 @classmethod
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
519 def from_line(cls, line):
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
520 """create an object by deserializing a line from AUTO_GEN_FILE"""
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
521 assert line.startswith(b'PENDING-v1 ')
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
522 (
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
523 __,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
524 bundle_type,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
525 revs,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
526 tip_rev,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
527 tip_node,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
528 hostname,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
529 pid,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
530 filepath,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
531 ) = line.split()
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
532 hostname = util.urlreq.unquote(hostname)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
533 filepath = util.urlreq.unquote(filepath)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
534 revs = int(revs)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
535 tip_rev = int(tip_rev)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
536 pid = int(pid)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
537 return cls(
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
538 bundle_type, revs, tip_rev, tip_node, hostname, pid, filepath
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
539 )
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
540
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
541 def to_line(self):
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
542 """serialize the object to include as a line in AUTO_GEN_FILE"""
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
543 templ = b"PENDING-v1 %s %d %d %s %s %d %s"
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
544 data = (
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
545 self.bundle_type,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
546 self.revs,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
547 self.tip_rev,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
548 self.tip_node,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
549 util.urlreq.quote(self.hostname),
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
550 self.pid,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
551 util.urlreq.quote(self.filepath),
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
552 )
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
553 return templ % data
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
554
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
555 def __eq__(self, other):
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
556 if not super(GeneratingBundle, self).__eq__(other):
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
557 return False
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
558 left = (self.hostname, self.pid, self.filepath)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
559 right = (other.hostname, other.pid, other.filepath)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
560 return left == right
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
561
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
562 def uploaded(self, url, basename):
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
563 """return a GeneratedBundle from this object"""
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
564 return GeneratedBundle(
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
565 self.bundle_type,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
566 self.revs,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
567 self.tip_rev,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
568 self.tip_node,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
569 url,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
570 basename,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
571 )
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
572
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
573
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
574 class GeneratedBundle(BundleBase):
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
575 """A bundle that is done being generated and can be served
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
576
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
577 extra attributes compared to BundleBase:
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
578
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
579 :file_url: the url where the bundle is available.
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
580 :basename: the "basename" used to upload (useful for deletion)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
581
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
582 These attributes exist to generate a bundle manifest
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
583 (.hg/pullbundles.manifest)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
584 """
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
585
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
586 ready = True
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
587
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
588 def __init__(
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
589 self, bundle_type, revs, tip_rev, tip_node, file_url, basename
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
590 ):
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
591 self.file_url = file_url
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
592 self.basename = basename
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
593 super(GeneratedBundle, self).__init__(
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
594 bundle_type, revs, tip_rev, tip_node
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
595 )
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
596
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
597 @classmethod
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
598 def from_line(cls, line):
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
599 """create an object by deserializing a line from AUTO_GEN_FILE"""
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
600 assert line.startswith(b'DONE-v1 ')
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
601 (
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
602 __,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
603 bundle_type,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
604 revs,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
605 tip_rev,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
606 tip_node,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
607 file_url,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
608 basename,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
609 ) = line.split()
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
610 revs = int(revs)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
611 tip_rev = int(tip_rev)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
612 file_url = util.urlreq.unquote(file_url)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
613 return cls(bundle_type, revs, tip_rev, tip_node, file_url, basename)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
614
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
615 def to_line(self):
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
616 """serialize the object to include as a line in AUTO_GEN_FILE"""
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
617 templ = b"DONE-v1 %s %d %d %s %s %s"
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
618 data = (
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
619 self.bundle_type,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
620 self.revs,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
621 self.tip_rev,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
622 self.tip_node,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
623 util.urlreq.quote(self.file_url),
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
624 self.basename,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
625 )
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
626 return templ % data
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
627
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
628 def manifest_line(self):
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
629 """serialize the object to include as a line in pullbundles.manifest"""
50710
1299525832d0 clone-bundle: only add the REQUIRESNI bit for http(s)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50701
diff changeset
630 templ = b"%s BUNDLESPEC=%s"
1299525832d0 clone-bundle: only add the REQUIRESNI bit for http(s)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50701
diff changeset
631 if self.file_url.startswith(b'http'):
1299525832d0 clone-bundle: only add the REQUIRESNI bit for http(s)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50701
diff changeset
632 templ += b" REQUIRESNI=true"
50686
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
633 return templ % (self.file_url, self.bundle_type)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
634
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
635 def __eq__(self, other):
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
636 if not super(GeneratedBundle, self).__eq__(other):
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
637 return False
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
638 return self.file_url == other.file_url
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
639
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
640
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
641 def parse_auto_gen(content):
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
642 """parse the AUTO_GEN_FILE to return a list of Bundle object"""
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
643 bundles = []
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
644 for line in content.splitlines():
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
645 if line.startswith(b'PENDING-v1 '):
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
646 bundles.append(GeneratingBundle.from_line(line))
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
647 elif line.startswith(b'DONE-v1 '):
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
648 bundles.append(GeneratedBundle.from_line(line))
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
649 return bundles
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
650
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
651
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
652 def dumps_auto_gen(bundles):
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
653 """serialize a list of Bundle as a AUTO_GEN_FILE content"""
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
654 lines = []
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
655 for b in bundles:
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
656 lines.append(b"%s\n" % b.to_line())
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
657 lines.sort()
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
658 return b"".join(lines)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
659
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
660
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
661 def read_auto_gen(repo):
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
662 """read the AUTO_GEN_FILE for the <repo> a list of Bundle object"""
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
663 data = repo.vfs.tryread(AUTO_GEN_FILE)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
664 if not data:
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
665 return []
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
666 return parse_auto_gen(data)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
667
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
668
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
669 def write_auto_gen(repo, bundles):
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
670 """write a list of Bundle objects into the repo's AUTO_GEN_FILE"""
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
671 assert repo._cb_lock_ref is not None
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
672 data = dumps_auto_gen(bundles)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
673 with repo.vfs(AUTO_GEN_FILE, mode=b'wb', atomictemp=True) as f:
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
674 f.write(data)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
675
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
676
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
677 def generate_manifest(bundles):
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
678 """write a list of Bundle objects into the repo's AUTO_GEN_FILE"""
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
679 bundles = list(bundles)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
680 bundles.sort(key=lambda b: b.bundle_type)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
681 lines = []
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
682 for b in bundles:
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
683 lines.append(b"%s\n" % b.manifest_line())
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
684 return b"".join(lines)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
685
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
686
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
687 def update_ondisk_manifest(repo):
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
688 """update the clonebundle manifest with latest url"""
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
689 with repo.clonebundles_lock():
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
690 bundles = read_auto_gen(repo)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
691
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
692 per_types = {}
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
693 for b in bundles:
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
694 if not (b.ready and b.valid_for(repo)):
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
695 continue
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
696 current = per_types.get(b.bundle_type)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
697 if current is not None and current.revs >= b.revs:
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
698 continue
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
699 per_types[b.bundle_type] = b
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
700 manifest = generate_manifest(per_types.values())
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
701 with repo.vfs(
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
702 bundlecaches.CB_MANIFEST_FILE, mode=b"wb", atomictemp=True
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
703 ) as f:
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
704 f.write(manifest)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
705
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
706
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
707 def update_bundle_list(repo, new_bundles=(), del_bundles=()):
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
708 """modify the repo's AUTO_GEN_FILE
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
709
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
710 This method also regenerates the clone bundle manifest when needed"""
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
711 with repo.clonebundles_lock():
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
712 bundles = read_auto_gen(repo)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
713 if del_bundles:
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
714 bundles = [b for b in bundles if b not in del_bundles]
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
715 new_bundles = [b for b in new_bundles if b not in bundles]
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
716 bundles.extend(new_bundles)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
717 write_auto_gen(repo, bundles)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
718 all_changed = []
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
719 all_changed.extend(new_bundles)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
720 all_changed.extend(del_bundles)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
721 if any(b.ready for b in all_changed):
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
722 update_ondisk_manifest(repo)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
723
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
724
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
725 def cleanup_tmp_bundle(repo, target):
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
726 """remove a GeneratingBundle file and entry"""
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
727 assert not target.ready
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
728 with repo.clonebundles_lock():
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
729 repo.vfs.tryunlink(target.filepath)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
730 update_bundle_list(repo, del_bundles=[target])
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
731
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
732
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
733 def finalize_one_bundle(repo, target):
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
734 """upload a generated bundle and advertise it in the clonebundles.manifest"""
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
735 with repo.clonebundles_lock():
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
736 bundles = read_auto_gen(repo)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
737 if target in bundles and target.valid_for(repo):
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
738 result = upload_bundle(repo, target)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
739 update_bundle_list(repo, new_bundles=[result])
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
740 cleanup_tmp_bundle(repo, target)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
741
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
742
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
743 def find_outdated_bundles(repo, bundles):
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
744 """finds outdated bundles"""
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
745 olds = []
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
746 per_types = {}
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
747 for b in bundles:
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
748 if not b.valid_for(repo):
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
749 olds.append(b)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
750 continue
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
751 l = per_types.setdefault(b.bundle_type, [])
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
752 l.append(b)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
753 for key in sorted(per_types):
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
754 all = per_types[key]
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
755 if len(all) > 1:
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
756 all.sort(key=lambda b: b.revs, reverse=True)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
757 olds.extend(all[1:])
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
758 return olds
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
759
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
760
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
761 def collect_garbage(repo):
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
762 """finds outdated bundles and get them deleted"""
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
763 with repo.clonebundles_lock():
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
764 bundles = read_auto_gen(repo)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
765 olds = find_outdated_bundles(repo, bundles)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
766 for o in olds:
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
767 delete_bundle(repo, o)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
768 update_bundle_list(repo, del_bundles=olds)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
769
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
770
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
771 def upload_bundle(repo, bundle):
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
772 """upload the result of a GeneratingBundle and return a GeneratedBundle
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
773
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
774 The upload is done using the `clone-bundles.upload-command`
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
775 """
50711
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
776 inline = repo.ui.config(b'clone-bundles', b'auto-generate.serve-inline')
50686
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
777 basename = repo.vfs.basename(bundle.filepath)
50711
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
778 if inline:
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
779 dest_dir = repo.vfs.join(bundlecaches.BUNDLE_CACHE_DIR)
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
780 repo.vfs.makedirs(dest_dir)
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
781 dest = repo.vfs.join(dest_dir, basename)
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
782 util.copyfiles(bundle.filepath, dest, hardlink=True)
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
783 url = bundlecaches.CLONEBUNDLESCHEME + basename
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
784 return bundle.uploaded(url, basename)
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
785 else:
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
786 cmd = repo.ui.config(b'clone-bundles', b'upload-command')
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
787 url = repo.ui.config(b'clone-bundles', b'url-template')
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
788 filepath = procutil.shellquote(bundle.filepath)
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
789 variables = {
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
790 b'HGCB_BUNDLE_PATH': filepath,
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
791 b'HGCB_BUNDLE_BASENAME': basename,
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
792 }
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
793 env = procutil.shellenviron(environ=variables)
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
794 ret = repo.ui.system(cmd, environ=env)
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
795 if ret:
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
796 raise error.Abort(b"command returned status %d: %s" % (ret, cmd))
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
797 url = (
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
798 url.decode('utf8')
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
799 .format(basename=basename.decode('utf8'))
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
800 .encode('utf8')
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
801 )
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
802 return bundle.uploaded(url, basename)
50686
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
803
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
804
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
805 def delete_bundle(repo, bundle):
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
806 """delete a bundle from storage"""
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
807 assert bundle.ready
50711
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
808
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
809 inline = bundle.file_url.startswith(bundlecaches.CLONEBUNDLESCHEME)
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
810
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
811 if inline:
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
812 msg = b'clone-bundles: deleting inline bundle %s\n'
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
813 else:
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
814 msg = b'clone-bundles: deleting bundle %s\n'
50686
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
815 msg %= bundle.basename
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
816 if repo.ui.configbool(b'devel', b'debug.clonebundles'):
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
817 repo.ui.write(msg)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
818 else:
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
819 repo.ui.debug(msg)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
820
50711
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
821 if inline:
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
822 inline_path = repo.vfs.join(
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
823 bundlecaches.BUNDLE_CACHE_DIR,
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
824 bundle.basename,
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
825 )
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
826 util.tryunlink(inline_path)
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
827 else:
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
828 cmd = repo.ui.config(b'clone-bundles', b'delete-command')
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
829 variables = {
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
830 b'HGCB_BUNDLE_URL': bundle.file_url,
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
831 b'HGCB_BASENAME': bundle.basename,
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
832 }
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
833 env = procutil.shellenviron(environ=variables)
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
834 ret = repo.ui.system(cmd, environ=env)
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
835 if ret:
40638610c6ee clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50710
diff changeset
836 raise error.Abort(b"command returned status %d: %s" % (ret, cmd))
50686
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
837
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
838
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
839 def auto_bundle_needed_actions(repo, bundles, op_id):
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
840 """find the list of bundles that need action
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
841
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
842 returns a list of RequestedBundle objects that need to be generated and
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
843 uploaded."""
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
844 create_bundles = []
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
845 delete_bundles = []
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
846 repo = repo.filtered(b"immutable")
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
847 targets = repo.ui.configlist(b'clone-bundles', b'auto-generate.formats')
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
848 ratio = float(
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
849 repo.ui.config(b'clone-bundles', b'trigger.below-bundled-ratio')
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
850 )
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
851 abs_revs = repo.ui.configint(b'clone-bundles', b'trigger.revs')
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
852 revs = len(repo.changelog)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
853 generic_data = {
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
854 'revs': revs,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
855 'head_revs': repo.changelog.headrevs(),
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
856 'tip_rev': repo.changelog.tiprev(),
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
857 'tip_node': node.hex(repo.changelog.tip()),
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
858 'op_id': op_id,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
859 }
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
860 for t in targets:
50713
ddc55fb220ba clone-bundles: check we generate a valid specification
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50711
diff changeset
861 t = bundlecaches.parsebundlespec(repo, t, strict=False).as_spec()
50686
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
862 if new_bundle_needed(repo, bundles, ratio, abs_revs, t, revs):
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
863 data = generic_data.copy()
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
864 data['bundle_type'] = t
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
865 b = RequestedBundle(**data)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
866 create_bundles.append(b)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
867 delete_bundles.extend(find_outdated_bundles(repo, bundles))
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
868 return create_bundles, delete_bundles
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
869
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
870
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
871 def new_bundle_needed(repo, bundles, ratio, abs_revs, bundle_type, revs):
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
872 """consider the current cached content and trigger new bundles if needed"""
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
873 threshold = max((revs * ratio), (revs - abs_revs))
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
874 for b in bundles:
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
875 if not b.valid_for(repo) or b.bundle_type != bundle_type:
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
876 continue
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
877 if b.revs > threshold:
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
878 return False
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
879 return True
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
880
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
881
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
882 def start_one_bundle(repo, bundle):
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
883 """start the generation of a single bundle file
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
884
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
885 the `bundle` argument should be a RequestedBundle object.
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
886
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
887 This data is passed to the `debugmakeclonebundles` "as is".
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
888 """
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
889 data = util.pickle.dumps(bundle)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
890 cmd = [procutil.hgexecutable(), b'--cwd', repo.path, INTERNAL_CMD]
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
891 env = procutil.shellenviron()
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
892 msg = b'clone-bundles: starting bundle generation: %s\n'
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
893 stdout = None
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
894 stderr = None
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
895 waits = []
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
896 record_wait = None
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
897 if repo.ui.configbool(b'devel', b'debug.clonebundles'):
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
898 stdout = procutil.stdout
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
899 stderr = procutil.stderr
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
900 repo.ui.write(msg % bundle.bundle_type)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
901 record_wait = waits.append
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
902 else:
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
903 repo.ui.debug(msg % bundle.bundle_type)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
904 bg = procutil.runbgcommand
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
905 bg(
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
906 cmd,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
907 env,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
908 stdin_bytes=data,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
909 stdout=stdout,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
910 stderr=stderr,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
911 record_wait=record_wait,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
912 )
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
913 for f in waits:
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
914 f()
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
915
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
916
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
917 INTERNAL_CMD = b'debug::internal-make-clone-bundles'
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
918
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
919
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
920 @command(INTERNAL_CMD, [], b'')
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
921 def debugmakeclonebundles(ui, repo):
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
922 """Internal command to auto-generate debug bundles"""
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
923 requested_bundle = util.pickle.load(procutil.stdin)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
924 procutil.stdin.close()
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
925
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
926 collect_garbage(repo)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
927
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
928 fname = requested_bundle.suggested_filename
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
929 fpath = repo.vfs.makedirs(b'tmp-bundles')
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
930 fpath = repo.vfs.join(b'tmp-bundles', fname)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
931 bundle = requested_bundle.generating(fpath)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
932 update_bundle_list(repo, new_bundles=[bundle])
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
933
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
934 requested_bundle.generate_bundle(repo, fpath)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
935
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
936 repo.invalidate()
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
937 finalize_one_bundle(repo, bundle)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
938
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
939
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
940 def make_auto_bundler(source_repo):
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
941 reporef = weakref.ref(source_repo)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
942
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
943 def autobundle(tr):
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
944 repo = reporef()
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
945 assert repo is not None
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
946 bundles = read_auto_gen(repo)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
947 new, __ = auto_bundle_needed_actions(repo, bundles, b"%d_txn" % id(tr))
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
948 for data in new:
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
949 start_one_bundle(repo, data)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
950 return None
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
951
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
952 return autobundle
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
953
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
954
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
955 def reposetup(ui, repo):
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
956 """install the two pieces needed for automatic clonebundle generation
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
957
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
958 - add a "post-close" hook that fires bundling when needed
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
959 - introduce a clone-bundle lock to let multiple processes meddle with the
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
960 state files.
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
961 """
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
962 if not repo.local():
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
963 return
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
964
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
965 class autobundlesrepo(repo.__class__):
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
966 def transaction(self, *args, **kwargs):
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
967 tr = super(autobundlesrepo, self).transaction(*args, **kwargs)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
968 enabled = repo.ui.configbool(
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
969 b'clone-bundles',
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
970 b'auto-generate.on-change',
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
971 )
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
972 targets = repo.ui.configlist(
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
973 b'clone-bundles', b'auto-generate.formats'
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
974 )
50736
b852c34dc2b8 clonebundles: add warning if auto-generate is enabled without formats
Mathias De Mare <mathias.de_mare@nokia.com>
parents: 50735
diff changeset
975 if enabled:
b852c34dc2b8 clonebundles: add warning if auto-generate is enabled without formats
Mathias De Mare <mathias.de_mare@nokia.com>
parents: 50735
diff changeset
976 if not targets:
b852c34dc2b8 clonebundles: add warning if auto-generate is enabled without formats
Mathias De Mare <mathias.de_mare@nokia.com>
parents: 50735
diff changeset
977 repo.ui.warn(
b852c34dc2b8 clonebundles: add warning if auto-generate is enabled without formats
Mathias De Mare <mathias.de_mare@nokia.com>
parents: 50735
diff changeset
978 _(
b852c34dc2b8 clonebundles: add warning if auto-generate is enabled without formats
Mathias De Mare <mathias.de_mare@nokia.com>
parents: 50735
diff changeset
979 b'clone-bundle auto-generate enabled, '
b852c34dc2b8 clonebundles: add warning if auto-generate is enabled without formats
Mathias De Mare <mathias.de_mare@nokia.com>
parents: 50735
diff changeset
980 b'but no formats specified: disabling generation\n'
b852c34dc2b8 clonebundles: add warning if auto-generate is enabled without formats
Mathias De Mare <mathias.de_mare@nokia.com>
parents: 50735
diff changeset
981 )
b852c34dc2b8 clonebundles: add warning if auto-generate is enabled without formats
Mathias De Mare <mathias.de_mare@nokia.com>
parents: 50735
diff changeset
982 )
b852c34dc2b8 clonebundles: add warning if auto-generate is enabled without formats
Mathias De Mare <mathias.de_mare@nokia.com>
parents: 50735
diff changeset
983 else:
b852c34dc2b8 clonebundles: add warning if auto-generate is enabled without formats
Mathias De Mare <mathias.de_mare@nokia.com>
parents: 50735
diff changeset
984 tr.addpostclose(CAT_POSTCLOSE, make_auto_bundler(self))
50686
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
985 return tr
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
986
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
987 @localrepo.unfilteredmethod
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
988 def clonebundles_lock(self, wait=True):
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
989 '''Lock the repository file related to clone bundles'''
50928
d718eddf01d9 safehasattr: drop usage in favor of hasattr
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50776
diff changeset
990 if not hasattr(self, '_cb_lock_ref'):
50686
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
991 self._cb_lock_ref = None
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
992 l = self._currentlock(self._cb_lock_ref)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
993 if l is not None:
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
994 l.lock()
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
995 return l
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
996
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
997 l = self._lock(
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
998 vfs=self.vfs,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
999 lockname=b"clonebundleslock",
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1000 wait=wait,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1001 releasefn=None,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1002 acquirefn=None,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1003 desc=_(b'repository %s') % self.origroot,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1004 )
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1005 self._cb_lock_ref = weakref.ref(l)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1006 return l
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1007
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1008 repo._wlockfreeprefix.add(AUTO_GEN_FILE)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1009 repo._wlockfreeprefix.add(bundlecaches.CB_MANIFEST_FILE)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1010 repo.__class__ = autobundlesrepo
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1011
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1012
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1013 @command(
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1014 b'admin::clone-bundles-refresh',
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1015 [
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1016 (
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1017 b'',
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1018 b'background',
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1019 False,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1020 _(b'start bundle generation in the background'),
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1021 ),
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1022 ],
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1023 b'',
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1024 )
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1025 def cmd_admin_clone_bundles_refresh(
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1026 ui,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1027 repo: localrepo.localrepository,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1028 background=False,
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1029 ):
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1030 """generate clone bundles according to the configuration
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1031
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1032 This runs the logic for automatic generation, removing outdated bundles and
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1033 generating new ones if necessary. See :hg:`help -e clone-bundles` for
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1034 details about how to configure this feature.
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1035 """
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1036 debug = repo.ui.configbool(b'devel', b'debug.clonebundles')
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1037 bundles = read_auto_gen(repo)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1038 op_id = b"%d_acbr" % os.getpid()
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1039 create, delete = auto_bundle_needed_actions(repo, bundles, op_id)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1040
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1041 # if some bundles are scheduled for creation in the background, they will
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1042 # deal with garbage collection too, so no need to synchroniously do it.
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1043 #
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1044 # However if no bundles are scheduled for creation, we need to explicitly do
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1045 # it here.
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1046 if not (background and create):
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1047 # we clean up outdated bundles before generating new ones to keep the
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1048 # last two versions of the bundle around for a while and avoid having to
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1049 # deal with clients that just got served a manifest.
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1050 for o in delete:
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1051 delete_bundle(repo, o)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1052 update_bundle_list(repo, del_bundles=delete)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1053
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1054 if create:
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1055 fpath = repo.vfs.makedirs(b'tmp-bundles')
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1056
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1057 if background:
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1058 for requested_bundle in create:
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1059 start_one_bundle(repo, requested_bundle)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1060 else:
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1061 for requested_bundle in create:
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1062 if debug:
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1063 msg = b'clone-bundles: starting bundle generation: %s\n'
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1064 repo.ui.write(msg % requested_bundle.bundle_type)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1065 fname = requested_bundle.suggested_filename
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1066 fpath = repo.vfs.join(b'tmp-bundles', fname)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1067 generating_bundle = requested_bundle.generating(fpath)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1068 update_bundle_list(repo, new_bundles=[generating_bundle])
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1069 requested_bundle.generate_bundle(repo, fpath)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1070 result = upload_bundle(repo, generating_bundle)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1071 update_bundle_list(repo, new_bundles=[result])
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1072 update_ondisk_manifest(repo)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1073 cleanup_tmp_bundle(repo, generating_bundle)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1074
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1075
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1076 @command(b'admin::clone-bundles-clear', [], b'')
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1077 def cmd_admin_clone_bundles_clear(ui, repo: localrepo.localrepository):
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1078 """remove existing clone bundle caches
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1079
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1080 See `hg help admin::clone-bundles-refresh` for details on how to regenerate
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1081 them.
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1082
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1083 This command will only affect bundles currently available, it will not
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1084 affect bundles being asynchronously generated.
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1085 """
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1086 bundles = read_auto_gen(repo)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1087 delete = [b for b in bundles if b.ready]
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1088 for o in delete:
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1089 delete_bundle(repo, o)
a41eeb877d07 branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50645
diff changeset
1090 update_bundle_list(repo, del_bundles=delete)