author | Matt Harbison <matt_harbison@yahoo.com> |
Thu, 10 Oct 2024 17:52:26 -0400 | |
changeset 52007 | 521df44118a8 |
parent 51863 | f4733654f144 |
child 52096 | 93484d43be22 |
permissions | -rw-r--r-- |
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
1 |
# This software may be used and distributed according to the terms of the |
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
2 |
# GNU General Public License version 2 or any later version. |
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
3 |
|
27738
a0e783d26e81
exchange: make clone bundles non-experimental and enabled by default
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27737
diff
changeset
|
4 |
"""advertise pre-generated bundles to seed clones |
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
5 |
|
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
6 |
"clonebundles" is a server-side extension used to advertise the existence |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
7 |
of pre-generated, externally hosted bundle files to clients that are |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
8 |
cloning so that cloning can be faster, more reliable, and require less |
37498
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
9 |
resources on the server. "pullbundles" is a related feature for sending |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
10 |
pre-generated bundle files to clients as part of pull operations. |
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
11 |
|
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
12 |
Cloning can be a CPU and I/O intensive operation on servers. Traditionally, |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
13 |
the server, in response to a client's request to clone, dynamically generates |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
14 |
a bundle containing the entire repository content and sends it to the client. |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
15 |
There is no caching on the server and the server will have to redundantly |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
16 |
generate the same outgoing bundle in response to each clone request. For |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
17 |
servers with large repositories or with high clone volume, the load from |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
18 |
clones can make scaling the server challenging and costly. |
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
19 |
|
37498
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
20 |
This extension provides server operators the ability to offload |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
21 |
potentially expensive clone load to an external service. Pre-generated |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
22 |
bundles also allow using more CPU intensive compression, reducing the |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
23 |
effective bandwidth requirements. |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
24 |
|
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
25 |
Here's how clone bundles work: |
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
26 |
|
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
27 |
1. A server operator establishes a mechanism for making bundle files available |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
28 |
on a hosting service where Mercurial clients can fetch them. |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
29 |
2. A manifest file listing available bundle URLs and some optional metadata |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
30 |
is added to the Mercurial repository on the server. |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
31 |
3. A client initiates a clone against a clone bundles aware server. |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
32 |
4. The client sees the server is advertising clone bundles and fetches the |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
33 |
manifest listing available bundles. |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
34 |
5. The client filters and sorts the available bundles based on what it |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
35 |
supports and prefers. |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
36 |
6. The client downloads and applies an available bundle from the |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
37 |
server-specified URL. |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
38 |
7. The client reconnects to the original server and performs the equivalent |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
39 |
of :hg:`pull` to retrieve all repository data not in the bundle. (The |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
40 |
repository could have been updated between when the bundle was created |
37498
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
41 |
and when the client started the clone.) This may use "pullbundles". |
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
42 |
|
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
43 |
Instead of the server generating full repository bundles for every clone |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
44 |
request, it generates full bundles once and they are subsequently reused to |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
45 |
bootstrap new clones. The server may still transfer data at clone time. |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
46 |
However, this is only data that has been added/changed since the bundle was |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
47 |
created. For large, established repositories, this can reduce server load for |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
48 |
clones to less than 1% of original. |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
49 |
|
37498
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
50 |
Here's how pullbundles work: |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
51 |
|
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
52 |
1. A manifest file listing available bundles and describing the revisions |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
53 |
is added to the Mercurial repository on the server. |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
54 |
2. A new-enough client informs the server that it supports partial pulls |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
55 |
and initiates a pull. |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
56 |
3. If the server has pull bundles enabled and sees the client advertising |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
57 |
partial pulls, it checks for a matching pull bundle in the manifest. |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
58 |
A bundle matches if the format is supported by the client, the client |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
59 |
has the required revisions already and needs something from the bundle. |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
60 |
4. If there is at least one matching bundle, the server sends it to the client. |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
61 |
5. The client applies the bundle and notices that the server reply was |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
62 |
incomplete. It initiates another pull. |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
63 |
|
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
64 |
To work, this extension requires the following of server operators: |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
65 |
|
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
66 |
* Generating bundle files of repository content (typically periodically, |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
67 |
such as once per day). |
37498
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
68 |
* Clone bundles: A file server that clients have network access to and that |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
69 |
Python knows how to talk to through its normal URL handling facility |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
70 |
(typically an HTTP/HTTPS server). |
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
71 |
* A process for keeping the bundles manifest in sync with available bundle |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
72 |
files. |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
73 |
|
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
74 |
Strictly speaking, using a static file hosting server isn't required: a server |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
75 |
operator could use a dynamic service for retrieving bundle data. However, |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
76 |
static file hosting services are simple and scalable and should be sufficient |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
77 |
for most needs. |
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
78 |
|
26884
762bf510b42c
clonebundles: fix typo s/comand/command/
Javi Merino <merino.jav@gmail.com>
parents:
26857
diff
changeset
|
79 |
Bundle files can be generated with the :hg:`bundle` command. Typically |
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
80 |
:hg:`bundle --all` is used to produce a bundle of the entire repository. |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
81 |
|
50645
bf7404f2e22d
bundles: clarify streaming v2 bundle usage
Mathias De Mare <mathias.de_mare@nokia.com>
parents:
48875
diff
changeset
|
82 |
The bundlespec option `stream` (see :hg:`help bundlespec`) |
bf7404f2e22d
bundles: clarify streaming v2 bundle usage
Mathias De Mare <mathias.de_mare@nokia.com>
parents:
48875
diff
changeset
|
83 |
can be used to produce a special *streaming clonebundle*, typically using |
bf7404f2e22d
bundles: clarify streaming v2 bundle usage
Mathias De Mare <mathias.de_mare@nokia.com>
parents:
48875
diff
changeset
|
84 |
:hg:`bundle --all --type="none-streamv2"`. |
bf7404f2e22d
bundles: clarify streaming v2 bundle usage
Mathias De Mare <mathias.de_mare@nokia.com>
parents:
48875
diff
changeset
|
85 |
These are bundle files that are extremely efficient |
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
86 |
to produce and consume (read: fast). However, they are larger than |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
87 |
traditional bundle formats and require that clients support the exact set |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
88 |
of repository data store formats in use by the repository that created them. |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
89 |
Typically, a newer server can serve data that is compatible with older clients. |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
90 |
However, *streaming clone bundles* don't have this guarantee. **Server |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
91 |
operators need to be aware that newer versions of Mercurial may produce |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
92 |
streaming clone bundles incompatible with older Mercurial versions.** |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
93 |
|
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
94 |
A server operator is responsible for creating a ``.hg/clonebundles.manifest`` |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
95 |
file containing the list of available bundle files suitable for seeding |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
96 |
clones. If this file does not exist, the repository will not advertise the |
37498
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
97 |
existence of clone bundles when clients connect. For pull bundles, |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
98 |
``.hg/pullbundles.manifest`` is used. |
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
99 |
|
32454
702af1ad3b18
clonebundles: fix missing newline character
Matt Harbison <matt_harbison@yahoo.com>
parents:
31146
diff
changeset
|
100 |
The manifest file contains a newline (\\n) delimited list of entries. |
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
101 |
|
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
102 |
Each line in this file defines an available bundle. Lines have the format: |
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
103 |
|
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
104 |
<URL> [<key>=<value>[ <key>=<value>]] |
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
105 |
|
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
106 |
That is, a URL followed by an optional, space-delimited list of key=value |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
107 |
pairs describing additional properties of this bundle. Both keys and values |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
108 |
are URI encoded. |
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
109 |
|
37498
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
110 |
For pull bundles, the URL is a path under the ``.hg`` directory of the |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
111 |
repository. |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
112 |
|
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
113 |
Keys in UPPERCASE are reserved for use by Mercurial and are defined below. |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
114 |
All non-uppercase keys can be used by site installations. An example use |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
115 |
for custom properties is to use the *datacenter* attribute to define which |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
116 |
data center a file is hosted in. Clients could then prefer a server in the |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
117 |
data center closest to them. |
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
118 |
|
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
119 |
The following reserved keys are currently defined: |
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
120 |
|
26644
74de1c59f71c
clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26623
diff
changeset
|
121 |
BUNDLESPEC |
74de1c59f71c
clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26623
diff
changeset
|
122 |
A "bundle specification" string that describes the type of the bundle. |
74de1c59f71c
clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26623
diff
changeset
|
123 |
|
74de1c59f71c
clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26623
diff
changeset
|
124 |
These are string values that are accepted by the "--type" argument of |
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
125 |
:hg:`bundle`. |
26644
74de1c59f71c
clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26623
diff
changeset
|
126 |
|
74de1c59f71c
clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26623
diff
changeset
|
127 |
The values are parsed in strict mode, which means they must be of the |
74de1c59f71c
clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26623
diff
changeset
|
128 |
"<compression>-<type>" form. See |
74de1c59f71c
clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26623
diff
changeset
|
129 |
mercurial.exchange.parsebundlespec() for more details. |
74de1c59f71c
clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26623
diff
changeset
|
130 |
|
27886
0288e63ea3be
clonebundles: improve BUNDLESPEC documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27738
diff
changeset
|
131 |
:hg:`debugbundle --spec` can be used to print the bundle specification |
0288e63ea3be
clonebundles: improve BUNDLESPEC documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27738
diff
changeset
|
132 |
string for a bundle file. The output of this command can be used verbatim |
0288e63ea3be
clonebundles: improve BUNDLESPEC documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27738
diff
changeset
|
133 |
for the value of ``BUNDLESPEC`` (it is already escaped). |
0288e63ea3be
clonebundles: improve BUNDLESPEC documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27738
diff
changeset
|
134 |
|
26644
74de1c59f71c
clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26623
diff
changeset
|
135 |
Clients will automatically filter out specifications that are unknown or |
74de1c59f71c
clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26623
diff
changeset
|
136 |
unsupported so they won't attempt to download something that likely won't |
74de1c59f71c
clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26623
diff
changeset
|
137 |
apply. |
74de1c59f71c
clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26623
diff
changeset
|
138 |
|
74de1c59f71c
clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26623
diff
changeset
|
139 |
The actual value doesn't impact client behavior beyond filtering: |
74de1c59f71c
clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26623
diff
changeset
|
140 |
clients will still sniff the bundle type from the header of downloaded |
74de1c59f71c
clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26623
diff
changeset
|
141 |
files. |
26645
2faa7671a4b3
clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26644
diff
changeset
|
142 |
|
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
143 |
**Use of this key is highly recommended**, as it allows clients to |
27886
0288e63ea3be
clonebundles: improve BUNDLESPEC documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27738
diff
changeset
|
144 |
easily skip unsupported bundles. If this key is not defined, an old |
0288e63ea3be
clonebundles: improve BUNDLESPEC documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27738
diff
changeset
|
145 |
client may attempt to apply a bundle that it is incapable of reading. |
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
146 |
|
26645
2faa7671a4b3
clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26644
diff
changeset
|
147 |
REQUIRESNI |
2faa7671a4b3
clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26644
diff
changeset
|
148 |
Whether Server Name Indication (SNI) is required to connect to the URL. |
2faa7671a4b3
clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26644
diff
changeset
|
149 |
SNI allows servers to use multiple certificates on the same IP. It is |
2faa7671a4b3
clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26644
diff
changeset
|
150 |
somewhat common in CDNs and other hosting providers. Older Python |
2faa7671a4b3
clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26644
diff
changeset
|
151 |
versions do not support SNI. Defining this attribute enables clients |
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
152 |
with older Python versions to filter this entry without experiencing |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
153 |
an opaque SSL failure at connection time. |
26645
2faa7671a4b3
clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26644
diff
changeset
|
154 |
|
2faa7671a4b3
clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26644
diff
changeset
|
155 |
If this is defined, it is important to advertise a non-SNI fallback |
2faa7671a4b3
clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26644
diff
changeset
|
156 |
URL or clients running old Python releases may not be able to clone |
2faa7671a4b3
clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26644
diff
changeset
|
157 |
with the clonebundles facility. |
2faa7671a4b3
clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26644
diff
changeset
|
158 |
|
2faa7671a4b3
clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26644
diff
changeset
|
159 |
Value should be "true". |
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
160 |
|
45252
20d110e6eea6
clonebundles: document REQUIREDRAM key
Gregory Szorc <gregory.szorc@gmail.com>
parents:
43077
diff
changeset
|
161 |
REQUIREDRAM |
20d110e6eea6
clonebundles: document REQUIREDRAM key
Gregory Szorc <gregory.szorc@gmail.com>
parents:
43077
diff
changeset
|
162 |
Value specifies expected memory requirements to decode the payload. |
20d110e6eea6
clonebundles: document REQUIREDRAM key
Gregory Szorc <gregory.szorc@gmail.com>
parents:
43077
diff
changeset
|
163 |
Values can have suffixes for common bytes sizes. e.g. "64MB". |
20d110e6eea6
clonebundles: document REQUIREDRAM key
Gregory Szorc <gregory.szorc@gmail.com>
parents:
43077
diff
changeset
|
164 |
|
20d110e6eea6
clonebundles: document REQUIREDRAM key
Gregory Szorc <gregory.szorc@gmail.com>
parents:
43077
diff
changeset
|
165 |
This key is often used with zstd-compressed bundles using a high |
20d110e6eea6
clonebundles: document REQUIREDRAM key
Gregory Szorc <gregory.szorc@gmail.com>
parents:
43077
diff
changeset
|
166 |
compression level / window size, which can require 100+ MB of memory |
20d110e6eea6
clonebundles: document REQUIREDRAM key
Gregory Szorc <gregory.szorc@gmail.com>
parents:
43077
diff
changeset
|
167 |
to decode. |
20d110e6eea6
clonebundles: document REQUIREDRAM key
Gregory Szorc <gregory.szorc@gmail.com>
parents:
43077
diff
changeset
|
168 |
|
37498
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
169 |
heads |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
170 |
Used for pull bundles. This contains the ``;`` separated changeset |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
171 |
hashes of the heads of the bundle content. |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
172 |
|
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
173 |
bases |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
174 |
Used for pull bundles. This contains the ``;`` separated changeset |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
175 |
hashes of the roots of the bundle content. This can be skipped if |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
176 |
the bundle was created without ``--base``. |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
177 |
|
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
178 |
Manifests can contain multiple entries. Assuming metadata is defined, clients |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
179 |
will filter entries from the manifest that they don't support. The remaining |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
180 |
entries are optionally sorted by client preferences |
32773
d25802b0eef5
clonebundles: reference correct config option
Gregory Szorc <gregory.szorc@gmail.com>
parents:
32454
diff
changeset
|
181 |
(``ui.clonebundleprefers`` config option). The client then attempts |
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
182 |
to fetch the bundle at the first URL in the remaining list. |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
183 |
|
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
184 |
**Errors when downloading a bundle will fail the entire clone operation: |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
185 |
clients do not automatically fall back to a traditional clone.** The reason |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
186 |
for this is that if a server is using clone bundles, it is probably doing so |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
187 |
because the feature is necessary to help it scale. In other words, there |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
188 |
is an assumption that clone load will be offloaded to another service and |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
189 |
that the Mercurial server isn't responsible for serving this clone load. |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
190 |
If that other service experiences issues and clients start mass falling back to |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
191 |
the original Mercurial server, the added clone load could overwhelm the server |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
192 |
due to unexpected load and effectively take it offline. Not having clients |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
193 |
automatically fall back to cloning from the original server mitigates this |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
194 |
scenario. |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
195 |
|
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
196 |
Because there is no automatic Mercurial server fallback on failure of the |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
197 |
bundle hosting service, it is important for server operators to view the bundle |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
198 |
hosting service as an extension of the Mercurial server in terms of |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
199 |
availability and service level agreements: if the bundle hosting service goes |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
200 |
down, so does the ability for clients to clone. Note: clients will see a |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
201 |
message informing them how to bypass the clone bundles facility when a failure |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
202 |
occurs. So server operators should prepare for some people to follow these |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
203 |
instructions when a failure occurs, thus driving more load to the original |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
204 |
Mercurial server when the bundle hosting service fails. |
50686
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
205 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
206 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
207 |
inline clonebundles |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
208 |
------------------- |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
209 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
210 |
It is possible to transmit clonebundles inline in case repositories are |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
211 |
accessed over SSH. This avoids having to setup an external HTTPS server |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
212 |
and results in the same access control as already present for the SSH setup. |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
213 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
214 |
Inline clonebundles should be placed into the `.hg/bundle-cache` directory. |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
215 |
A clonebundle at `.hg/bundle-cache/mybundle.bundle` is referred to |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
216 |
in the `clonebundles.manifest` file as `peer-bundle-cache://mybundle.bundle`. |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
217 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
218 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
219 |
auto-generation of clone bundles |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
220 |
-------------------------------- |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
221 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
222 |
It is possible to set Mercurial to automatically re-generate clone bundles when |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
223 |
enough new content is available. |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
224 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
225 |
Mercurial will take care of the process asynchronously. The defined list of |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
226 |
bundle-type will be generated, uploaded, and advertised. Older bundles will get |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
227 |
decommissioned as newer ones replace them. |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
228 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
229 |
Bundles Generation: |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
230 |
................... |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
231 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
232 |
The extension can generate multiple variants of the clone bundle. Each |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
233 |
different variant will be defined by the "bundle-spec" they use:: |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
234 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
235 |
[clone-bundles] |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
236 |
auto-generate.formats= zstd-v2, gzip-v2 |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
237 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
238 |
See `hg help bundlespec` for details about available options. |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
239 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
240 |
By default, new bundles are generated when 5% of the repository contents or at |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
241 |
least 1000 revisions are not contained in the cached bundles. This option can |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
242 |
be controlled by the `clone-bundles.trigger.below-bundled-ratio` option |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
243 |
(default 0.95) and the `clone-bundles.trigger.revs` option (default 1000):: |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
244 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
245 |
[clone-bundles] |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
246 |
trigger.below-bundled-ratio=0.95 |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
247 |
trigger.revs=1000 |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
248 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
249 |
This logic can be manually triggered using the `admin::clone-bundles-refresh` |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
250 |
command, or automatically on each repository change if |
50735
2b0598121a71
clonebundles: fix display of auto-generate.on-change lines
Mathias De Mare <mathias.de_mare@nokia.com>
parents:
50713
diff
changeset
|
251 |
`clone-bundles.auto-generate.on-change` is set to `yes`:: |
50686
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
252 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
253 |
[clone-bundles] |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
254 |
auto-generate.on-change=yes |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
255 |
auto-generate.formats= zstd-v2, gzip-v2 |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
256 |
|
50711
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
257 |
Automatic Inline serving |
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
258 |
........................ |
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
259 |
|
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
260 |
The simplest way to serve the generated bundle is through the Mercurial |
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
261 |
protocol. However it is not the most efficient as request will still be served |
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
262 |
by that main server. It is useful in case where authentication is complexe or |
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
263 |
when an efficient mirror system is already in use anyway. See the `inline |
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
264 |
clonebundles` section above for details about inline clonebundles |
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
265 |
|
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
266 |
To automatically serve generated bundle through inline clonebundle, simply set |
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
267 |
the following option:: |
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
268 |
|
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
269 |
auto-generate.serve-inline=yes |
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
270 |
|
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
271 |
Enabling this option disable the managed upload and serving explained below. |
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
272 |
|
50686
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
273 |
Bundles Upload and Serving: |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
274 |
........................... |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
275 |
|
50711
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
276 |
This is the most efficient way to serve automatically generated clone bundles, |
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
277 |
but requires some setup. |
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
278 |
|
50686
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
279 |
The generated bundles need to be made available to users through a "public" URL. |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
280 |
This should be donne through `clone-bundles.upload-command` configuration. The |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
281 |
value of this command should be a shell command. It will have access to the |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
282 |
bundle file path through the `$HGCB_BUNDLE_PATH` variable. And the expected |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
283 |
basename in the "public" URL is accessible at:: |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
284 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
285 |
[clone-bundles] |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
286 |
upload-command=sftp put $HGCB_BUNDLE_PATH \ |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
287 |
sftp://bundles.host/clone-bundles/$HGCB_BUNDLE_BASENAME |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
288 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
289 |
If the file was already uploaded, the command must still succeed. |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
290 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
291 |
After upload, the file should be available at an url defined by |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
292 |
`clone-bundles.url-template`. |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
293 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
294 |
[clone-bundles] |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
295 |
url-template=https://bundles.host/cache/clone-bundles/{basename} |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
296 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
297 |
Old bundles cleanup: |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
298 |
.................... |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
299 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
300 |
When new bundles are generated, the older ones are no longer necessary and can |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
301 |
be removed from storage. This is done through the `clone-bundles.delete-command` |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
302 |
configuration. The command is given the url of the artifact to delete through |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
303 |
the `$HGCB_BUNDLE_URL` environment variable. |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
304 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
305 |
[clone-bundles] |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
306 |
delete-command=sftp rm sftp://bundles.host/clone-bundles/$HGCB_BUNDLE_BASENAME |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
307 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
308 |
If the file was already deleted, the command must still succeed. |
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
309 |
""" |
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
310 |
|
51863
f4733654f144
typing: add `from __future__ import annotations` to most files
Matt Harbison <matt_harbison@yahoo.com>
parents:
50928
diff
changeset
|
311 |
from __future__ import annotations |
28095
7fa139eaebb4
clonebundles: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27886
diff
changeset
|
312 |
|
50686
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
313 |
import os |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
314 |
import weakref |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
315 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
316 |
from mercurial.i18n import _ |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
317 |
|
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
318 |
from mercurial import ( |
45785
80f32ec8653a
clonebundle: move the manifest filename to a constant
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
45252
diff
changeset
|
319 |
bundlecaches, |
50686
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
320 |
commands, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
321 |
error, |
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
322 |
extensions, |
50686
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
323 |
localrepo, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
324 |
lock, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
325 |
node, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
326 |
registrar, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
327 |
util, |
37785
b4d85bc122bd
wireproto: rename wireproto to wireprotov1server (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37498
diff
changeset
|
328 |
wireprotov1server, |
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
329 |
) |
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
330 |
|
50686
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
331 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
332 |
from mercurial.utils import ( |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
333 |
procutil, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
334 |
) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
335 |
|
43077
687b865b95ad
formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents:
43076
diff
changeset
|
336 |
testedwith = b'ships-with-hg-core' |
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
337 |
|
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
37785
diff
changeset
|
338 |
|
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
339 |
def capabilities(orig, repo, proto): |
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
340 |
caps = orig(repo, proto) |
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
341 |
|
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
342 |
# Only advertise if a manifest exists. This does add some I/O to requests. |
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
343 |
# But this should be cheaper than a wasted network round trip due to |
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
344 |
# missing file. |
45785
80f32ec8653a
clonebundle: move the manifest filename to a constant
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
45252
diff
changeset
|
345 |
if repo.vfs.exists(bundlecaches.CB_MANIFEST_FILE): |
43077
687b865b95ad
formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents:
43076
diff
changeset
|
346 |
caps.append(b'clonebundles') |
50701
4238e6b22fc4
clonebundles: introduce a new write protocol command
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50686
diff
changeset
|
347 |
caps.append(b'clonebundles_manifest') |
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
348 |
|
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
349 |
return caps |
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
350 |
|
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
37785
diff
changeset
|
351 |
|
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
352 |
def extsetup(ui): |
50776
335aeb3d670b
wrapfunction: use sysstr instead of bytes as argument in "clonebundles"
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50736
diff
changeset
|
353 |
extensions.wrapfunction(wireprotov1server, '_capabilities', capabilities) |
50686
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
354 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
355 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
356 |
# logic for bundle auto-generation |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
357 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
358 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
359 |
configtable = {} |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
360 |
configitem = registrar.configitem(configtable) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
361 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
362 |
cmdtable = {} |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
363 |
command = registrar.command(cmdtable) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
364 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
365 |
configitem(b'clone-bundles', b'auto-generate.on-change', default=False) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
366 |
configitem(b'clone-bundles', b'auto-generate.formats', default=list) |
50711
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
367 |
configitem(b'clone-bundles', b'auto-generate.serve-inline', default=False) |
50686
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
368 |
configitem(b'clone-bundles', b'trigger.below-bundled-ratio', default=0.95) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
369 |
configitem(b'clone-bundles', b'trigger.revs', default=1000) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
370 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
371 |
configitem(b'clone-bundles', b'upload-command', default=None) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
372 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
373 |
configitem(b'clone-bundles', b'delete-command', default=None) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
374 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
375 |
configitem(b'clone-bundles', b'url-template', default=None) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
376 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
377 |
configitem(b'devel', b'debug.clonebundles', default=False) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
378 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
379 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
380 |
# category for the post-close transaction hooks |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
381 |
CAT_POSTCLOSE = b"clonebundles-autobundles" |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
382 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
383 |
# template for bundle file names |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
384 |
BUNDLE_MASK = ( |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
385 |
b"full-%(bundle_type)s-%(revs)d_revs-%(tip_short)s_tip-%(op_id)s.hg" |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
386 |
) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
387 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
388 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
389 |
# file in .hg/ use to track clonebundles being auto-generated |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
390 |
AUTO_GEN_FILE = b'clonebundles.auto-gen' |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
391 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
392 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
393 |
class BundleBase(object): |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
394 |
"""represents the core of properties that matters for us in a bundle |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
395 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
396 |
:bundle_type: the bundlespec (see hg help bundlespec) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
397 |
:revs: the number of revisions in the repo at bundle creation time |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
398 |
:tip_rev: the rev-num of the tip revision |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
399 |
:tip_node: the node id of the tip-most revision in the bundle |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
400 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
401 |
:ready: True if the bundle is ready to be served |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
402 |
""" |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
403 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
404 |
ready = False |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
405 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
406 |
def __init__(self, bundle_type, revs, tip_rev, tip_node): |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
407 |
self.bundle_type = bundle_type |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
408 |
self.revs = revs |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
409 |
self.tip_rev = tip_rev |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
410 |
self.tip_node = tip_node |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
411 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
412 |
def valid_for(self, repo): |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
413 |
"""is this bundle applicable to the current repository |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
414 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
415 |
This is useful for detecting bundles made irrelevant by stripping. |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
416 |
""" |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
417 |
tip_node = node.bin(self.tip_node) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
418 |
return repo.changelog.index.get_rev(tip_node) == self.tip_rev |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
419 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
420 |
def __eq__(self, other): |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
421 |
left = (self.ready, self.bundle_type, self.tip_rev, self.tip_node) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
422 |
right = (other.ready, other.bundle_type, other.tip_rev, other.tip_node) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
423 |
return left == right |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
424 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
425 |
def __neq__(self, other): |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
426 |
return not self == other |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
427 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
428 |
def __cmp__(self, other): |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
429 |
if self == other: |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
430 |
return 0 |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
431 |
return -1 |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
432 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
433 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
434 |
class RequestedBundle(BundleBase): |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
435 |
"""A bundle that should be generated. |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
436 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
437 |
Additional attributes compared to BundleBase |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
438 |
:heads: list of head revisions (as rev-num) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
439 |
:op_id: a "unique" identifier for the operation triggering the change |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
440 |
""" |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
441 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
442 |
def __init__(self, bundle_type, revs, tip_rev, tip_node, head_revs, op_id): |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
443 |
self.head_revs = head_revs |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
444 |
self.op_id = op_id |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
445 |
super(RequestedBundle, self).__init__( |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
446 |
bundle_type, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
447 |
revs, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
448 |
tip_rev, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
449 |
tip_node, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
450 |
) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
451 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
452 |
@property |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
453 |
def suggested_filename(self): |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
454 |
"""A filename that can be used for the generated bundle""" |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
455 |
data = { |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
456 |
b'bundle_type': self.bundle_type, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
457 |
b'revs': self.revs, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
458 |
b'heads': self.head_revs, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
459 |
b'tip_rev': self.tip_rev, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
460 |
b'tip_node': self.tip_node, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
461 |
b'tip_short': self.tip_node[:12], |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
462 |
b'op_id': self.op_id, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
463 |
} |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
464 |
return BUNDLE_MASK % data |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
465 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
466 |
def generate_bundle(self, repo, file_path): |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
467 |
"""generate the bundle at `filepath`""" |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
468 |
commands.bundle( |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
469 |
repo.ui, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
470 |
repo, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
471 |
file_path, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
472 |
base=[b"null"], |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
473 |
rev=self.head_revs, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
474 |
type=self.bundle_type, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
475 |
quiet=True, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
476 |
) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
477 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
478 |
def generating(self, file_path, hostname=None, pid=None): |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
479 |
"""return a GeneratingBundle object from this object""" |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
480 |
if pid is None: |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
481 |
pid = os.getpid() |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
482 |
if hostname is None: |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
483 |
hostname = lock._getlockprefix() |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
484 |
return GeneratingBundle( |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
485 |
self.bundle_type, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
486 |
self.revs, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
487 |
self.tip_rev, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
488 |
self.tip_node, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
489 |
hostname, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
490 |
pid, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
491 |
file_path, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
492 |
) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
493 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
494 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
495 |
class GeneratingBundle(BundleBase): |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
496 |
"""A bundle being generated |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
497 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
498 |
extra attributes compared to BundleBase: |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
499 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
500 |
:hostname: the hostname of the machine generating the bundle |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
501 |
:pid: the pid of the process generating the bundle |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
502 |
:filepath: the target filename of the bundle |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
503 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
504 |
These attributes exist to help detect stalled generation processes. |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
505 |
""" |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
506 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
507 |
ready = False |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
508 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
509 |
def __init__( |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
510 |
self, bundle_type, revs, tip_rev, tip_node, hostname, pid, filepath |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
511 |
): |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
512 |
self.hostname = hostname |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
513 |
self.pid = pid |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
514 |
self.filepath = filepath |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
515 |
super(GeneratingBundle, self).__init__( |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
516 |
bundle_type, revs, tip_rev, tip_node |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
517 |
) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
518 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
519 |
@classmethod |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
520 |
def from_line(cls, line): |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
521 |
"""create an object by deserializing a line from AUTO_GEN_FILE""" |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
522 |
assert line.startswith(b'PENDING-v1 ') |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
523 |
( |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
524 |
__, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
525 |
bundle_type, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
526 |
revs, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
527 |
tip_rev, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
528 |
tip_node, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
529 |
hostname, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
530 |
pid, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
531 |
filepath, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
532 |
) = line.split() |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
533 |
hostname = util.urlreq.unquote(hostname) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
534 |
filepath = util.urlreq.unquote(filepath) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
535 |
revs = int(revs) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
536 |
tip_rev = int(tip_rev) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
537 |
pid = int(pid) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
538 |
return cls( |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
539 |
bundle_type, revs, tip_rev, tip_node, hostname, pid, filepath |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
540 |
) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
541 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
542 |
def to_line(self): |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
543 |
"""serialize the object to include as a line in AUTO_GEN_FILE""" |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
544 |
templ = b"PENDING-v1 %s %d %d %s %s %d %s" |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
545 |
data = ( |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
546 |
self.bundle_type, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
547 |
self.revs, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
548 |
self.tip_rev, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
549 |
self.tip_node, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
550 |
util.urlreq.quote(self.hostname), |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
551 |
self.pid, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
552 |
util.urlreq.quote(self.filepath), |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
553 |
) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
554 |
return templ % data |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
555 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
556 |
def __eq__(self, other): |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
557 |
if not super(GeneratingBundle, self).__eq__(other): |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
558 |
return False |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
559 |
left = (self.hostname, self.pid, self.filepath) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
560 |
right = (other.hostname, other.pid, other.filepath) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
561 |
return left == right |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
562 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
563 |
def uploaded(self, url, basename): |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
564 |
"""return a GeneratedBundle from this object""" |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
565 |
return GeneratedBundle( |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
566 |
self.bundle_type, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
567 |
self.revs, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
568 |
self.tip_rev, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
569 |
self.tip_node, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
570 |
url, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
571 |
basename, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
572 |
) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
573 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
574 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
575 |
class GeneratedBundle(BundleBase): |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
576 |
"""A bundle that is done being generated and can be served |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
577 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
578 |
extra attributes compared to BundleBase: |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
579 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
580 |
:file_url: the url where the bundle is available. |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
581 |
:basename: the "basename" used to upload (useful for deletion) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
582 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
583 |
These attributes exist to generate a bundle manifest |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
584 |
(.hg/pullbundles.manifest) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
585 |
""" |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
586 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
587 |
ready = True |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
588 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
589 |
def __init__( |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
590 |
self, bundle_type, revs, tip_rev, tip_node, file_url, basename |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
591 |
): |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
592 |
self.file_url = file_url |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
593 |
self.basename = basename |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
594 |
super(GeneratedBundle, self).__init__( |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
595 |
bundle_type, revs, tip_rev, tip_node |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
596 |
) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
597 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
598 |
@classmethod |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
599 |
def from_line(cls, line): |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
600 |
"""create an object by deserializing a line from AUTO_GEN_FILE""" |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
601 |
assert line.startswith(b'DONE-v1 ') |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
602 |
( |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
603 |
__, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
604 |
bundle_type, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
605 |
revs, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
606 |
tip_rev, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
607 |
tip_node, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
608 |
file_url, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
609 |
basename, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
610 |
) = line.split() |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
611 |
revs = int(revs) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
612 |
tip_rev = int(tip_rev) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
613 |
file_url = util.urlreq.unquote(file_url) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
614 |
return cls(bundle_type, revs, tip_rev, tip_node, file_url, basename) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
615 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
616 |
def to_line(self): |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
617 |
"""serialize the object to include as a line in AUTO_GEN_FILE""" |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
618 |
templ = b"DONE-v1 %s %d %d %s %s %s" |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
619 |
data = ( |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
620 |
self.bundle_type, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
621 |
self.revs, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
622 |
self.tip_rev, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
623 |
self.tip_node, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
624 |
util.urlreq.quote(self.file_url), |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
625 |
self.basename, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
626 |
) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
627 |
return templ % data |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
628 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
629 |
def manifest_line(self): |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
630 |
"""serialize the object to include as a line in pullbundles.manifest""" |
50710
1299525832d0
clone-bundle: only add the REQUIRESNI bit for http(s)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50701
diff
changeset
|
631 |
templ = b"%s BUNDLESPEC=%s" |
1299525832d0
clone-bundle: only add the REQUIRESNI bit for http(s)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50701
diff
changeset
|
632 |
if self.file_url.startswith(b'http'): |
1299525832d0
clone-bundle: only add the REQUIRESNI bit for http(s)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50701
diff
changeset
|
633 |
templ += b" REQUIRESNI=true" |
50686
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
634 |
return templ % (self.file_url, self.bundle_type) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
635 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
636 |
def __eq__(self, other): |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
637 |
if not super(GeneratedBundle, self).__eq__(other): |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
638 |
return False |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
639 |
return self.file_url == other.file_url |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
640 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
641 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
642 |
def parse_auto_gen(content): |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
643 |
"""parse the AUTO_GEN_FILE to return a list of Bundle object""" |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
644 |
bundles = [] |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
645 |
for line in content.splitlines(): |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
646 |
if line.startswith(b'PENDING-v1 '): |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
647 |
bundles.append(GeneratingBundle.from_line(line)) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
648 |
elif line.startswith(b'DONE-v1 '): |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
649 |
bundles.append(GeneratedBundle.from_line(line)) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
650 |
return bundles |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
651 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
652 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
653 |
def dumps_auto_gen(bundles): |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
654 |
"""serialize a list of Bundle as a AUTO_GEN_FILE content""" |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
655 |
lines = [] |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
656 |
for b in bundles: |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
657 |
lines.append(b"%s\n" % b.to_line()) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
658 |
lines.sort() |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
659 |
return b"".join(lines) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
660 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
661 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
662 |
def read_auto_gen(repo): |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
663 |
"""read the AUTO_GEN_FILE for the <repo> a list of Bundle object""" |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
664 |
data = repo.vfs.tryread(AUTO_GEN_FILE) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
665 |
if not data: |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
666 |
return [] |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
667 |
return parse_auto_gen(data) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
668 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
669 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
670 |
def write_auto_gen(repo, bundles): |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
671 |
"""write a list of Bundle objects into the repo's AUTO_GEN_FILE""" |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
672 |
assert repo._cb_lock_ref is not None |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
673 |
data = dumps_auto_gen(bundles) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
674 |
with repo.vfs(AUTO_GEN_FILE, mode=b'wb', atomictemp=True) as f: |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
675 |
f.write(data) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
676 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
677 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
678 |
def generate_manifest(bundles): |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
679 |
"""write a list of Bundle objects into the repo's AUTO_GEN_FILE""" |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
680 |
bundles = list(bundles) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
681 |
bundles.sort(key=lambda b: b.bundle_type) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
682 |
lines = [] |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
683 |
for b in bundles: |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
684 |
lines.append(b"%s\n" % b.manifest_line()) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
685 |
return b"".join(lines) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
686 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
687 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
688 |
def update_ondisk_manifest(repo): |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
689 |
"""update the clonebundle manifest with latest url""" |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
690 |
with repo.clonebundles_lock(): |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
691 |
bundles = read_auto_gen(repo) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
692 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
693 |
per_types = {} |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
694 |
for b in bundles: |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
695 |
if not (b.ready and b.valid_for(repo)): |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
696 |
continue |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
697 |
current = per_types.get(b.bundle_type) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
698 |
if current is not None and current.revs >= b.revs: |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
699 |
continue |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
700 |
per_types[b.bundle_type] = b |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
701 |
manifest = generate_manifest(per_types.values()) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
702 |
with repo.vfs( |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
703 |
bundlecaches.CB_MANIFEST_FILE, mode=b"wb", atomictemp=True |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
704 |
) as f: |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
705 |
f.write(manifest) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
706 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
707 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
708 |
def update_bundle_list(repo, new_bundles=(), del_bundles=()): |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
709 |
"""modify the repo's AUTO_GEN_FILE |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
710 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
711 |
This method also regenerates the clone bundle manifest when needed""" |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
712 |
with repo.clonebundles_lock(): |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
713 |
bundles = read_auto_gen(repo) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
714 |
if del_bundles: |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
715 |
bundles = [b for b in bundles if b not in del_bundles] |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
716 |
new_bundles = [b for b in new_bundles if b not in bundles] |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
717 |
bundles.extend(new_bundles) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
718 |
write_auto_gen(repo, bundles) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
719 |
all_changed = [] |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
720 |
all_changed.extend(new_bundles) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
721 |
all_changed.extend(del_bundles) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
722 |
if any(b.ready for b in all_changed): |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
723 |
update_ondisk_manifest(repo) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
724 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
725 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
726 |
def cleanup_tmp_bundle(repo, target): |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
727 |
"""remove a GeneratingBundle file and entry""" |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
728 |
assert not target.ready |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
729 |
with repo.clonebundles_lock(): |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
730 |
repo.vfs.tryunlink(target.filepath) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
731 |
update_bundle_list(repo, del_bundles=[target]) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
732 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
733 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
734 |
def finalize_one_bundle(repo, target): |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
735 |
"""upload a generated bundle and advertise it in the clonebundles.manifest""" |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
736 |
with repo.clonebundles_lock(): |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
737 |
bundles = read_auto_gen(repo) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
738 |
if target in bundles and target.valid_for(repo): |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
739 |
result = upload_bundle(repo, target) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
740 |
update_bundle_list(repo, new_bundles=[result]) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
741 |
cleanup_tmp_bundle(repo, target) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
742 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
743 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
744 |
def find_outdated_bundles(repo, bundles): |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
745 |
"""finds outdated bundles""" |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
746 |
olds = [] |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
747 |
per_types = {} |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
748 |
for b in bundles: |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
749 |
if not b.valid_for(repo): |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
750 |
olds.append(b) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
751 |
continue |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
752 |
l = per_types.setdefault(b.bundle_type, []) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
753 |
l.append(b) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
754 |
for key in sorted(per_types): |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
755 |
all = per_types[key] |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
756 |
if len(all) > 1: |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
757 |
all.sort(key=lambda b: b.revs, reverse=True) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
758 |
olds.extend(all[1:]) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
759 |
return olds |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
760 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
761 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
762 |
def collect_garbage(repo): |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
763 |
"""finds outdated bundles and get them deleted""" |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
764 |
with repo.clonebundles_lock(): |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
765 |
bundles = read_auto_gen(repo) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
766 |
olds = find_outdated_bundles(repo, bundles) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
767 |
for o in olds: |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
768 |
delete_bundle(repo, o) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
769 |
update_bundle_list(repo, del_bundles=olds) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
770 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
771 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
772 |
def upload_bundle(repo, bundle): |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
773 |
"""upload the result of a GeneratingBundle and return a GeneratedBundle |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
774 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
775 |
The upload is done using the `clone-bundles.upload-command` |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
776 |
""" |
50711
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
777 |
inline = repo.ui.config(b'clone-bundles', b'auto-generate.serve-inline') |
50686
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
778 |
basename = repo.vfs.basename(bundle.filepath) |
50711
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
779 |
if inline: |
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
780 |
dest_dir = repo.vfs.join(bundlecaches.BUNDLE_CACHE_DIR) |
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
781 |
repo.vfs.makedirs(dest_dir) |
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
782 |
dest = repo.vfs.join(dest_dir, basename) |
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
783 |
util.copyfiles(bundle.filepath, dest, hardlink=True) |
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
784 |
url = bundlecaches.CLONEBUNDLESCHEME + basename |
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
785 |
return bundle.uploaded(url, basename) |
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
786 |
else: |
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
787 |
cmd = repo.ui.config(b'clone-bundles', b'upload-command') |
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
788 |
url = repo.ui.config(b'clone-bundles', b'url-template') |
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
789 |
filepath = procutil.shellquote(bundle.filepath) |
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
790 |
variables = { |
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
791 |
b'HGCB_BUNDLE_PATH': filepath, |
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
792 |
b'HGCB_BUNDLE_BASENAME': basename, |
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
793 |
} |
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
794 |
env = procutil.shellenviron(environ=variables) |
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
795 |
ret = repo.ui.system(cmd, environ=env) |
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
796 |
if ret: |
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
797 |
raise error.Abort(b"command returned status %d: %s" % (ret, cmd)) |
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
798 |
url = ( |
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
799 |
url.decode('utf8') |
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
800 |
.format(basename=basename.decode('utf8')) |
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
801 |
.encode('utf8') |
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
802 |
) |
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
803 |
return bundle.uploaded(url, basename) |
50686
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
804 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
805 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
806 |
def delete_bundle(repo, bundle): |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
807 |
"""delete a bundle from storage""" |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
808 |
assert bundle.ready |
50711
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
809 |
|
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
810 |
inline = bundle.file_url.startswith(bundlecaches.CLONEBUNDLESCHEME) |
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
811 |
|
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
812 |
if inline: |
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
813 |
msg = b'clone-bundles: deleting inline bundle %s\n' |
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
814 |
else: |
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
815 |
msg = b'clone-bundles: deleting bundle %s\n' |
50686
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
816 |
msg %= bundle.basename |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
817 |
if repo.ui.configbool(b'devel', b'debug.clonebundles'): |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
818 |
repo.ui.write(msg) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
819 |
else: |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
820 |
repo.ui.debug(msg) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
821 |
|
50711
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
822 |
if inline: |
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
823 |
inline_path = repo.vfs.join( |
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
824 |
bundlecaches.BUNDLE_CACHE_DIR, |
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
825 |
bundle.basename, |
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
826 |
) |
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
827 |
util.tryunlink(inline_path) |
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
828 |
else: |
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
829 |
cmd = repo.ui.config(b'clone-bundles', b'delete-command') |
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
830 |
variables = { |
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
831 |
b'HGCB_BUNDLE_URL': bundle.file_url, |
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
832 |
b'HGCB_BASENAME': bundle.basename, |
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
833 |
} |
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
834 |
env = procutil.shellenviron(environ=variables) |
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
835 |
ret = repo.ui.system(cmd, environ=env) |
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
836 |
if ret: |
40638610c6ee
clonebundles: adds a auto-generate.serve-inline option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50710
diff
changeset
|
837 |
raise error.Abort(b"command returned status %d: %s" % (ret, cmd)) |
50686
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
838 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
839 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
840 |
def auto_bundle_needed_actions(repo, bundles, op_id): |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
841 |
"""find the list of bundles that need action |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
842 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
843 |
returns a list of RequestedBundle objects that need to be generated and |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
844 |
uploaded.""" |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
845 |
create_bundles = [] |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
846 |
delete_bundles = [] |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
847 |
repo = repo.filtered(b"immutable") |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
848 |
targets = repo.ui.configlist(b'clone-bundles', b'auto-generate.formats') |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
849 |
ratio = float( |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
850 |
repo.ui.config(b'clone-bundles', b'trigger.below-bundled-ratio') |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
851 |
) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
852 |
abs_revs = repo.ui.configint(b'clone-bundles', b'trigger.revs') |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
853 |
revs = len(repo.changelog) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
854 |
generic_data = { |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
855 |
'revs': revs, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
856 |
'head_revs': repo.changelog.headrevs(), |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
857 |
'tip_rev': repo.changelog.tiprev(), |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
858 |
'tip_node': node.hex(repo.changelog.tip()), |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
859 |
'op_id': op_id, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
860 |
} |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
861 |
for t in targets: |
50713
ddc55fb220ba
clone-bundles: check we generate a valid specification
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50711
diff
changeset
|
862 |
t = bundlecaches.parsebundlespec(repo, t, strict=False).as_spec() |
50686
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
863 |
if new_bundle_needed(repo, bundles, ratio, abs_revs, t, revs): |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
864 |
data = generic_data.copy() |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
865 |
data['bundle_type'] = t |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
866 |
b = RequestedBundle(**data) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
867 |
create_bundles.append(b) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
868 |
delete_bundles.extend(find_outdated_bundles(repo, bundles)) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
869 |
return create_bundles, delete_bundles |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
870 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
871 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
872 |
def new_bundle_needed(repo, bundles, ratio, abs_revs, bundle_type, revs): |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
873 |
"""consider the current cached content and trigger new bundles if needed""" |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
874 |
threshold = max((revs * ratio), (revs - abs_revs)) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
875 |
for b in bundles: |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
876 |
if not b.valid_for(repo) or b.bundle_type != bundle_type: |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
877 |
continue |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
878 |
if b.revs > threshold: |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
879 |
return False |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
880 |
return True |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
881 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
882 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
883 |
def start_one_bundle(repo, bundle): |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
884 |
"""start the generation of a single bundle file |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
885 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
886 |
the `bundle` argument should be a RequestedBundle object. |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
887 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
888 |
This data is passed to the `debugmakeclonebundles` "as is". |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
889 |
""" |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
890 |
data = util.pickle.dumps(bundle) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
891 |
cmd = [procutil.hgexecutable(), b'--cwd', repo.path, INTERNAL_CMD] |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
892 |
env = procutil.shellenviron() |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
893 |
msg = b'clone-bundles: starting bundle generation: %s\n' |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
894 |
stdout = None |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
895 |
stderr = None |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
896 |
waits = [] |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
897 |
record_wait = None |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
898 |
if repo.ui.configbool(b'devel', b'debug.clonebundles'): |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
899 |
stdout = procutil.stdout |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
900 |
stderr = procutil.stderr |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
901 |
repo.ui.write(msg % bundle.bundle_type) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
902 |
record_wait = waits.append |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
903 |
else: |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
904 |
repo.ui.debug(msg % bundle.bundle_type) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
905 |
bg = procutil.runbgcommand |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
906 |
bg( |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
907 |
cmd, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
908 |
env, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
909 |
stdin_bytes=data, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
910 |
stdout=stdout, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
911 |
stderr=stderr, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
912 |
record_wait=record_wait, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
913 |
) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
914 |
for f in waits: |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
915 |
f() |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
916 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
917 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
918 |
INTERNAL_CMD = b'debug::internal-make-clone-bundles' |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
919 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
920 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
921 |
@command(INTERNAL_CMD, [], b'') |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
922 |
def debugmakeclonebundles(ui, repo): |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
923 |
"""Internal command to auto-generate debug bundles""" |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
924 |
requested_bundle = util.pickle.load(procutil.stdin) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
925 |
procutil.stdin.close() |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
926 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
927 |
collect_garbage(repo) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
928 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
929 |
fname = requested_bundle.suggested_filename |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
930 |
fpath = repo.vfs.makedirs(b'tmp-bundles') |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
931 |
fpath = repo.vfs.join(b'tmp-bundles', fname) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
932 |
bundle = requested_bundle.generating(fpath) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
933 |
update_bundle_list(repo, new_bundles=[bundle]) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
934 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
935 |
requested_bundle.generate_bundle(repo, fpath) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
936 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
937 |
repo.invalidate() |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
938 |
finalize_one_bundle(repo, bundle) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
939 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
940 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
941 |
def make_auto_bundler(source_repo): |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
942 |
reporef = weakref.ref(source_repo) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
943 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
944 |
def autobundle(tr): |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
945 |
repo = reporef() |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
946 |
assert repo is not None |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
947 |
bundles = read_auto_gen(repo) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
948 |
new, __ = auto_bundle_needed_actions(repo, bundles, b"%d_txn" % id(tr)) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
949 |
for data in new: |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
950 |
start_one_bundle(repo, data) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
951 |
return None |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
952 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
953 |
return autobundle |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
954 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
955 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
956 |
def reposetup(ui, repo): |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
957 |
"""install the two pieces needed for automatic clonebundle generation |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
958 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
959 |
- add a "post-close" hook that fires bundling when needed |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
960 |
- introduce a clone-bundle lock to let multiple processes meddle with the |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
961 |
state files. |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
962 |
""" |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
963 |
if not repo.local(): |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
964 |
return |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
965 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
966 |
class autobundlesrepo(repo.__class__): |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
967 |
def transaction(self, *args, **kwargs): |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
968 |
tr = super(autobundlesrepo, self).transaction(*args, **kwargs) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
969 |
enabled = repo.ui.configbool( |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
970 |
b'clone-bundles', |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
971 |
b'auto-generate.on-change', |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
972 |
) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
973 |
targets = repo.ui.configlist( |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
974 |
b'clone-bundles', b'auto-generate.formats' |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
975 |
) |
50736
b852c34dc2b8
clonebundles: add warning if auto-generate is enabled without formats
Mathias De Mare <mathias.de_mare@nokia.com>
parents:
50735
diff
changeset
|
976 |
if enabled: |
b852c34dc2b8
clonebundles: add warning if auto-generate is enabled without formats
Mathias De Mare <mathias.de_mare@nokia.com>
parents:
50735
diff
changeset
|
977 |
if not targets: |
b852c34dc2b8
clonebundles: add warning if auto-generate is enabled without formats
Mathias De Mare <mathias.de_mare@nokia.com>
parents:
50735
diff
changeset
|
978 |
repo.ui.warn( |
b852c34dc2b8
clonebundles: add warning if auto-generate is enabled without formats
Mathias De Mare <mathias.de_mare@nokia.com>
parents:
50735
diff
changeset
|
979 |
_( |
b852c34dc2b8
clonebundles: add warning if auto-generate is enabled without formats
Mathias De Mare <mathias.de_mare@nokia.com>
parents:
50735
diff
changeset
|
980 |
b'clone-bundle auto-generate enabled, ' |
b852c34dc2b8
clonebundles: add warning if auto-generate is enabled without formats
Mathias De Mare <mathias.de_mare@nokia.com>
parents:
50735
diff
changeset
|
981 |
b'but no formats specified: disabling generation\n' |
b852c34dc2b8
clonebundles: add warning if auto-generate is enabled without formats
Mathias De Mare <mathias.de_mare@nokia.com>
parents:
50735
diff
changeset
|
982 |
) |
b852c34dc2b8
clonebundles: add warning if auto-generate is enabled without formats
Mathias De Mare <mathias.de_mare@nokia.com>
parents:
50735
diff
changeset
|
983 |
) |
b852c34dc2b8
clonebundles: add warning if auto-generate is enabled without formats
Mathias De Mare <mathias.de_mare@nokia.com>
parents:
50735
diff
changeset
|
984 |
else: |
b852c34dc2b8
clonebundles: add warning if auto-generate is enabled without formats
Mathias De Mare <mathias.de_mare@nokia.com>
parents:
50735
diff
changeset
|
985 |
tr.addpostclose(CAT_POSTCLOSE, make_auto_bundler(self)) |
50686
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
986 |
return tr |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
987 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
988 |
@localrepo.unfilteredmethod |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
989 |
def clonebundles_lock(self, wait=True): |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
990 |
'''Lock the repository file related to clone bundles''' |
50928
d718eddf01d9
safehasattr: drop usage in favor of hasattr
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50776
diff
changeset
|
991 |
if not hasattr(self, '_cb_lock_ref'): |
50686
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
992 |
self._cb_lock_ref = None |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
993 |
l = self._currentlock(self._cb_lock_ref) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
994 |
if l is not None: |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
995 |
l.lock() |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
996 |
return l |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
997 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
998 |
l = self._lock( |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
999 |
vfs=self.vfs, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1000 |
lockname=b"clonebundleslock", |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1001 |
wait=wait, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1002 |
releasefn=None, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1003 |
acquirefn=None, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1004 |
desc=_(b'repository %s') % self.origroot, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1005 |
) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1006 |
self._cb_lock_ref = weakref.ref(l) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1007 |
return l |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1008 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1009 |
repo._wlockfreeprefix.add(AUTO_GEN_FILE) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1010 |
repo._wlockfreeprefix.add(bundlecaches.CB_MANIFEST_FILE) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1011 |
repo.__class__ = autobundlesrepo |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1012 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1013 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1014 |
@command( |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1015 |
b'admin::clone-bundles-refresh', |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1016 |
[ |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1017 |
( |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1018 |
b'', |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1019 |
b'background', |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1020 |
False, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1021 |
_(b'start bundle generation in the background'), |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1022 |
), |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1023 |
], |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1024 |
b'', |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1025 |
) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1026 |
def cmd_admin_clone_bundles_refresh( |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1027 |
ui, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1028 |
repo: localrepo.localrepository, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1029 |
background=False, |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1030 |
): |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1031 |
"""generate clone bundles according to the configuration |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1032 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1033 |
This runs the logic for automatic generation, removing outdated bundles and |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1034 |
generating new ones if necessary. See :hg:`help -e clone-bundles` for |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1035 |
details about how to configure this feature. |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1036 |
""" |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1037 |
debug = repo.ui.configbool(b'devel', b'debug.clonebundles') |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1038 |
bundles = read_auto_gen(repo) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1039 |
op_id = b"%d_acbr" % os.getpid() |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1040 |
create, delete = auto_bundle_needed_actions(repo, bundles, op_id) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1041 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1042 |
# if some bundles are scheduled for creation in the background, they will |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1043 |
# deal with garbage collection too, so no need to synchroniously do it. |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1044 |
# |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1045 |
# However if no bundles are scheduled for creation, we need to explicitly do |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1046 |
# it here. |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1047 |
if not (background and create): |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1048 |
# we clean up outdated bundles before generating new ones to keep the |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1049 |
# last two versions of the bundle around for a while and avoid having to |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1050 |
# deal with clients that just got served a manifest. |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1051 |
for o in delete: |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1052 |
delete_bundle(repo, o) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1053 |
update_bundle_list(repo, del_bundles=delete) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1054 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1055 |
if create: |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1056 |
fpath = repo.vfs.makedirs(b'tmp-bundles') |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1057 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1058 |
if background: |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1059 |
for requested_bundle in create: |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1060 |
start_one_bundle(repo, requested_bundle) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1061 |
else: |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1062 |
for requested_bundle in create: |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1063 |
if debug: |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1064 |
msg = b'clone-bundles: starting bundle generation: %s\n' |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1065 |
repo.ui.write(msg % requested_bundle.bundle_type) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1066 |
fname = requested_bundle.suggested_filename |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1067 |
fpath = repo.vfs.join(b'tmp-bundles', fname) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1068 |
generating_bundle = requested_bundle.generating(fpath) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1069 |
update_bundle_list(repo, new_bundles=[generating_bundle]) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1070 |
requested_bundle.generate_bundle(repo, fpath) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1071 |
result = upload_bundle(repo, generating_bundle) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1072 |
update_bundle_list(repo, new_bundles=[result]) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1073 |
update_ondisk_manifest(repo) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1074 |
cleanup_tmp_bundle(repo, generating_bundle) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1075 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1076 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1077 |
@command(b'admin::clone-bundles-clear', [], b'') |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1078 |
def cmd_admin_clone_bundles_clear(ui, repo: localrepo.localrepository): |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1079 |
"""remove existing clone bundle caches |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1080 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1081 |
See `hg help admin::clone-bundles-refresh` for details on how to regenerate |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1082 |
them. |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1083 |
|
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1084 |
This command will only affect bundles currently available, it will not |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1085 |
affect bundles being asynchronously generated. |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1086 |
""" |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1087 |
bundles = read_auto_gen(repo) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1088 |
delete = [b for b in bundles if b.ready] |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1089 |
for o in delete: |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1090 |
delete_bundle(repo, o) |
a41eeb877d07
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50645
diff
changeset
|
1091 |
update_bundle_list(repo, del_bundles=delete) |