Mercurial > hg
annotate hgext/clonebundles.py @ 50431:971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
See inline documentation for details.
author | Pierre-Yves David <pierre-yves.david@octobus.net> |
---|---|
date | Tue, 24 Mar 2020 03:25:33 +0100 |
parents | 5ae30ff79c76 |
children | 5b70b9f5a2f9 |
rev | line source |
---|---|
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
1 # This software may be used and distributed according to the terms of the |
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
2 # GNU General Public License version 2 or any later version. |
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
3 |
27738
a0e783d26e81
exchange: make clone bundles non-experimental and enabled by default
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27737
diff
changeset
|
4 """advertise pre-generated bundles to seed clones |
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
5 |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
6 "clonebundles" is a server-side extension used to advertise the existence |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
7 of pre-generated, externally hosted bundle files to clients that are |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
8 cloning so that cloning can be faster, more reliable, and require less |
37498
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
9 resources on the server. "pullbundles" is a related feature for sending |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
10 pre-generated bundle files to clients as part of pull operations. |
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
11 |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
12 Cloning can be a CPU and I/O intensive operation on servers. Traditionally, |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
13 the server, in response to a client's request to clone, dynamically generates |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
14 a bundle containing the entire repository content and sends it to the client. |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
15 There is no caching on the server and the server will have to redundantly |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
16 generate the same outgoing bundle in response to each clone request. For |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
17 servers with large repositories or with high clone volume, the load from |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
18 clones can make scaling the server challenging and costly. |
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
19 |
37498
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
20 This extension provides server operators the ability to offload |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
21 potentially expensive clone load to an external service. Pre-generated |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
22 bundles also allow using more CPU intensive compression, reducing the |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
23 effective bandwidth requirements. |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
24 |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
25 Here's how clone bundles work: |
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
26 |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
27 1. A server operator establishes a mechanism for making bundle files available |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
28 on a hosting service where Mercurial clients can fetch them. |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
29 2. A manifest file listing available bundle URLs and some optional metadata |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
30 is added to the Mercurial repository on the server. |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
31 3. A client initiates a clone against a clone bundles aware server. |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
32 4. The client sees the server is advertising clone bundles and fetches the |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
33 manifest listing available bundles. |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
34 5. The client filters and sorts the available bundles based on what it |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
35 supports and prefers. |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
36 6. The client downloads and applies an available bundle from the |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
37 server-specified URL. |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
38 7. The client reconnects to the original server and performs the equivalent |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
39 of :hg:`pull` to retrieve all repository data not in the bundle. (The |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
40 repository could have been updated between when the bundle was created |
37498
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
41 and when the client started the clone.) This may use "pullbundles". |
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
42 |
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
43 Instead of the server generating full repository bundles for every clone |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
44 request, it generates full bundles once and they are subsequently reused to |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
45 bootstrap new clones. The server may still transfer data at clone time. |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
46 However, this is only data that has been added/changed since the bundle was |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
47 created. For large, established repositories, this can reduce server load for |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
48 clones to less than 1% of original. |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
49 |
37498
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
50 Here's how pullbundles work: |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
51 |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
52 1. A manifest file listing available bundles and describing the revisions |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
53 is added to the Mercurial repository on the server. |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
54 2. A new-enough client informs the server that it supports partial pulls |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
55 and initiates a pull. |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
56 3. If the server has pull bundles enabled and sees the client advertising |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
57 partial pulls, it checks for a matching pull bundle in the manifest. |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
58 A bundle matches if the format is supported by the client, the client |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
59 has the required revisions already and needs something from the bundle. |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
60 4. If there is at least one matching bundle, the server sends it to the client. |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
61 5. The client applies the bundle and notices that the server reply was |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
62 incomplete. It initiates another pull. |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
63 |
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
64 To work, this extension requires the following of server operators: |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
65 |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
66 * Generating bundle files of repository content (typically periodically, |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
67 such as once per day). |
37498
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
68 * Clone bundles: A file server that clients have network access to and that |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
69 Python knows how to talk to through its normal URL handling facility |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
70 (typically an HTTP/HTTPS server). |
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
71 * A process for keeping the bundles manifest in sync with available bundle |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
72 files. |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
73 |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
74 Strictly speaking, using a static file hosting server isn't required: a server |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
75 operator could use a dynamic service for retrieving bundle data. However, |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
76 static file hosting services are simple and scalable and should be sufficient |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
77 for most needs. |
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
78 |
26884
762bf510b42c
clonebundles: fix typo s/comand/command/
Javi Merino <merino.jav@gmail.com>
parents:
26857
diff
changeset
|
79 Bundle files can be generated with the :hg:`bundle` command. Typically |
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
80 :hg:`bundle --all` is used to produce a bundle of the entire repository. |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
81 |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
82 :hg:`debugcreatestreamclonebundle` can be used to produce a special |
37498
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
83 *streaming clonebundle*. These are bundle files that are extremely efficient |
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
84 to produce and consume (read: fast). However, they are larger than |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
85 traditional bundle formats and require that clients support the exact set |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
86 of repository data store formats in use by the repository that created them. |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
87 Typically, a newer server can serve data that is compatible with older clients. |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
88 However, *streaming clone bundles* don't have this guarantee. **Server |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
89 operators need to be aware that newer versions of Mercurial may produce |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
90 streaming clone bundles incompatible with older Mercurial versions.** |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
91 |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
92 A server operator is responsible for creating a ``.hg/clonebundles.manifest`` |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
93 file containing the list of available bundle files suitable for seeding |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
94 clones. If this file does not exist, the repository will not advertise the |
37498
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
95 existence of clone bundles when clients connect. For pull bundles, |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
96 ``.hg/pullbundles.manifest`` is used. |
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
97 |
32454
702af1ad3b18
clonebundles: fix missing newline character
Matt Harbison <matt_harbison@yahoo.com>
parents:
31146
diff
changeset
|
98 The manifest file contains a newline (\\n) delimited list of entries. |
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
99 |
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
100 Each line in this file defines an available bundle. Lines have the format: |
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
101 |
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
102 <URL> [<key>=<value>[ <key>=<value>]] |
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
103 |
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
104 That is, a URL followed by an optional, space-delimited list of key=value |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
105 pairs describing additional properties of this bundle. Both keys and values |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
106 are URI encoded. |
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
107 |
37498
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
108 For pull bundles, the URL is a path under the ``.hg`` directory of the |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
109 repository. |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
110 |
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
111 Keys in UPPERCASE are reserved for use by Mercurial and are defined below. |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
112 All non-uppercase keys can be used by site installations. An example use |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
113 for custom properties is to use the *datacenter* attribute to define which |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
114 data center a file is hosted in. Clients could then prefer a server in the |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
115 data center closest to them. |
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
116 |
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
117 The following reserved keys are currently defined: |
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
118 |
26644
74de1c59f71c
clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26623
diff
changeset
|
119 BUNDLESPEC |
74de1c59f71c
clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26623
diff
changeset
|
120 A "bundle specification" string that describes the type of the bundle. |
74de1c59f71c
clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26623
diff
changeset
|
121 |
74de1c59f71c
clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26623
diff
changeset
|
122 These are string values that are accepted by the "--type" argument of |
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
123 :hg:`bundle`. |
26644
74de1c59f71c
clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26623
diff
changeset
|
124 |
74de1c59f71c
clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26623
diff
changeset
|
125 The values are parsed in strict mode, which means they must be of the |
74de1c59f71c
clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26623
diff
changeset
|
126 "<compression>-<type>" form. See |
74de1c59f71c
clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26623
diff
changeset
|
127 mercurial.exchange.parsebundlespec() for more details. |
74de1c59f71c
clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26623
diff
changeset
|
128 |
27886
0288e63ea3be
clonebundles: improve BUNDLESPEC documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27738
diff
changeset
|
129 :hg:`debugbundle --spec` can be used to print the bundle specification |
0288e63ea3be
clonebundles: improve BUNDLESPEC documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27738
diff
changeset
|
130 string for a bundle file. The output of this command can be used verbatim |
0288e63ea3be
clonebundles: improve BUNDLESPEC documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27738
diff
changeset
|
131 for the value of ``BUNDLESPEC`` (it is already escaped). |
0288e63ea3be
clonebundles: improve BUNDLESPEC documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27738
diff
changeset
|
132 |
26644
74de1c59f71c
clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26623
diff
changeset
|
133 Clients will automatically filter out specifications that are unknown or |
74de1c59f71c
clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26623
diff
changeset
|
134 unsupported so they won't attempt to download something that likely won't |
74de1c59f71c
clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26623
diff
changeset
|
135 apply. |
74de1c59f71c
clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26623
diff
changeset
|
136 |
74de1c59f71c
clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26623
diff
changeset
|
137 The actual value doesn't impact client behavior beyond filtering: |
74de1c59f71c
clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26623
diff
changeset
|
138 clients will still sniff the bundle type from the header of downloaded |
74de1c59f71c
clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26623
diff
changeset
|
139 files. |
26645
2faa7671a4b3
clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26644
diff
changeset
|
140 |
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
141 **Use of this key is highly recommended**, as it allows clients to |
27886
0288e63ea3be
clonebundles: improve BUNDLESPEC documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27738
diff
changeset
|
142 easily skip unsupported bundles. If this key is not defined, an old |
0288e63ea3be
clonebundles: improve BUNDLESPEC documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27738
diff
changeset
|
143 client may attempt to apply a bundle that it is incapable of reading. |
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
144 |
26645
2faa7671a4b3
clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26644
diff
changeset
|
145 REQUIRESNI |
2faa7671a4b3
clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26644
diff
changeset
|
146 Whether Server Name Indication (SNI) is required to connect to the URL. |
2faa7671a4b3
clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26644
diff
changeset
|
147 SNI allows servers to use multiple certificates on the same IP. It is |
2faa7671a4b3
clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26644
diff
changeset
|
148 somewhat common in CDNs and other hosting providers. Older Python |
2faa7671a4b3
clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26644
diff
changeset
|
149 versions do not support SNI. Defining this attribute enables clients |
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
150 with older Python versions to filter this entry without experiencing |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
151 an opaque SSL failure at connection time. |
26645
2faa7671a4b3
clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26644
diff
changeset
|
152 |
2faa7671a4b3
clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26644
diff
changeset
|
153 If this is defined, it is important to advertise a non-SNI fallback |
2faa7671a4b3
clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26644
diff
changeset
|
154 URL or clients running old Python releases may not be able to clone |
2faa7671a4b3
clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26644
diff
changeset
|
155 with the clonebundles facility. |
2faa7671a4b3
clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26644
diff
changeset
|
156 |
2faa7671a4b3
clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26644
diff
changeset
|
157 Value should be "true". |
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
158 |
45252
20d110e6eea6
clonebundles: document REQUIREDRAM key
Gregory Szorc <gregory.szorc@gmail.com>
parents:
43077
diff
changeset
|
159 REQUIREDRAM |
20d110e6eea6
clonebundles: document REQUIREDRAM key
Gregory Szorc <gregory.szorc@gmail.com>
parents:
43077
diff
changeset
|
160 Value specifies expected memory requirements to decode the payload. |
20d110e6eea6
clonebundles: document REQUIREDRAM key
Gregory Szorc <gregory.szorc@gmail.com>
parents:
43077
diff
changeset
|
161 Values can have suffixes for common bytes sizes. e.g. "64MB". |
20d110e6eea6
clonebundles: document REQUIREDRAM key
Gregory Szorc <gregory.szorc@gmail.com>
parents:
43077
diff
changeset
|
162 |
20d110e6eea6
clonebundles: document REQUIREDRAM key
Gregory Szorc <gregory.szorc@gmail.com>
parents:
43077
diff
changeset
|
163 This key is often used with zstd-compressed bundles using a high |
20d110e6eea6
clonebundles: document REQUIREDRAM key
Gregory Szorc <gregory.szorc@gmail.com>
parents:
43077
diff
changeset
|
164 compression level / window size, which can require 100+ MB of memory |
20d110e6eea6
clonebundles: document REQUIREDRAM key
Gregory Szorc <gregory.szorc@gmail.com>
parents:
43077
diff
changeset
|
165 to decode. |
20d110e6eea6
clonebundles: document REQUIREDRAM key
Gregory Szorc <gregory.szorc@gmail.com>
parents:
43077
diff
changeset
|
166 |
37498
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
167 heads |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
168 Used for pull bundles. This contains the ``;`` separated changeset |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
169 hashes of the heads of the bundle content. |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
170 |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
171 bases |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
172 Used for pull bundles. This contains the ``;`` separated changeset |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
173 hashes of the roots of the bundle content. This can be skipped if |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
174 the bundle was created without ``--base``. |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
175 |
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
176 Manifests can contain multiple entries. Assuming metadata is defined, clients |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
177 will filter entries from the manifest that they don't support. The remaining |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
178 entries are optionally sorted by client preferences |
32773
d25802b0eef5
clonebundles: reference correct config option
Gregory Szorc <gregory.szorc@gmail.com>
parents:
32454
diff
changeset
|
179 (``ui.clonebundleprefers`` config option). The client then attempts |
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
180 to fetch the bundle at the first URL in the remaining list. |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
181 |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
182 **Errors when downloading a bundle will fail the entire clone operation: |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
183 clients do not automatically fall back to a traditional clone.** The reason |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
184 for this is that if a server is using clone bundles, it is probably doing so |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
185 because the feature is necessary to help it scale. In other words, there |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
186 is an assumption that clone load will be offloaded to another service and |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
187 that the Mercurial server isn't responsible for serving this clone load. |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
188 If that other service experiences issues and clients start mass falling back to |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
189 the original Mercurial server, the added clone load could overwhelm the server |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
190 due to unexpected load and effectively take it offline. Not having clients |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
191 automatically fall back to cloning from the original server mitigates this |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
192 scenario. |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
193 |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
194 Because there is no automatic Mercurial server fallback on failure of the |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
195 bundle hosting service, it is important for server operators to view the bundle |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
196 hosting service as an extension of the Mercurial server in terms of |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
197 availability and service level agreements: if the bundle hosting service goes |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
198 down, so does the ability for clients to clone. Note: clients will see a |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
199 message informing them how to bypass the clone bundles facility when a failure |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
200 occurs. So server operators should prepare for some people to follow these |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
201 instructions when a failure occurs, thus driving more load to the original |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
202 Mercurial server when the bundle hosting service fails. |
50430
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
203 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
204 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
205 auto-generation of clone bundles |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
206 -------------------------------- |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
207 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
208 It is possible to set Mercurial to automatically re-generate clone bundles when |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
209 new content is available. |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
210 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
211 Mercurial will take care of the process asynchronously. The defined list of |
50431
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
212 bundle-type will be generated, uploaded, and advertised. Older bundles will get |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
213 decommissioned as newer ones replace them. |
50430
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
214 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
215 Bundles Generation: |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
216 ................... |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
217 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
218 The extension can generate multiple variants of the clone bundle. Each |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
219 different variant will be defined by the "bundle-spec" they use:: |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
220 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
221 [clone-bundles] |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
222 auto-generate.formats= zstd-v2, gzip-v2 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
223 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
224 See `hg help bundlespec` for details about available options. |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
225 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
226 Bundles Upload and Serving: |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
227 ........................... |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
228 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
229 The generated bundles need to be made available to users through a "public" URL. |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
230 This should be donne through `clone-bundles.upload-command` configuration. The |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
231 value of this command should be a shell command. It will have access to the |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
232 bundle file path through the `$HGCB_BUNDLE_PATH` variable. And the expected |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
233 basename in the "public" URL is accessible at:: |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
234 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
235 [clone-bundles] |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
236 upload-command=sftp put $HGCB_BUNDLE_PATH \ |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
237 sftp://bundles.host/clone-bundles/$HGCB_BUNDLE_BASENAME |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
238 |
50431
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
239 If the file was already uploaded, the command must still succeed. |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
240 |
50430
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
241 After upload, the file should be available at an url defined by |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
242 `clone-bundles.url-template`. |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
243 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
244 [clone-bundles] |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
245 url-template=https://bundles.host/cache/clone-bundles/{basename} |
50431
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
246 |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
247 Old bundles cleanup: |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
248 .................... |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
249 |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
250 When new bundles are generated, the older ones are no longer necessary and can |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
251 be removed from storage. This is done through the `clone-bundles.delete-command` |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
252 configuration. The command is given the url of the artifact to delete through |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
253 the `$HGCB_BUNDLE_URL` environment variable. |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
254 |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
255 [clone-bundles] |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
256 delete-command=sftp rm sftp://bundles.host/clone-bundles/$HGCB_BUNDLE_BASENAME |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
257 |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
258 If the file was already deleted, the command must still succeed. |
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
259 """ |
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
260 |
28095
7fa139eaebb4
clonebundles: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27886
diff
changeset
|
261 |
50430
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
262 import os |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
263 import weakref |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
264 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
265 from mercurial.i18n import _ |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
266 |
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
267 from mercurial import ( |
45785
80f32ec8653a
clonebundle: move the manifest filename to a constant
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
45252
diff
changeset
|
268 bundlecaches, |
50430
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
269 commands, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
270 error, |
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
271 extensions, |
50430
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
272 localrepo, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
273 lock, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
274 node, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
275 registrar, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
276 util, |
37785
b4d85bc122bd
wireproto: rename wireproto to wireprotov1server (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37498
diff
changeset
|
277 wireprotov1server, |
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
278 ) |
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
279 |
50430
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
280 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
281 from mercurial.utils import ( |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
282 procutil, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
283 ) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
284 |
43077
687b865b95ad
formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents:
43076
diff
changeset
|
285 testedwith = b'ships-with-hg-core' |
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
286 |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
37785
diff
changeset
|
287 |
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
288 def capabilities(orig, repo, proto): |
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
289 caps = orig(repo, proto) |
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
290 |
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
291 # Only advertise if a manifest exists. This does add some I/O to requests. |
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
292 # But this should be cheaper than a wasted network round trip due to |
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
293 # missing file. |
45785
80f32ec8653a
clonebundle: move the manifest filename to a constant
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
45252
diff
changeset
|
294 if repo.vfs.exists(bundlecaches.CB_MANIFEST_FILE): |
43077
687b865b95ad
formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents:
43076
diff
changeset
|
295 caps.append(b'clonebundles') |
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
296 |
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
297 return caps |
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
298 |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
37785
diff
changeset
|
299 |
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
300 def extsetup(ui): |
43077
687b865b95ad
formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents:
43076
diff
changeset
|
301 extensions.wrapfunction(wireprotov1server, b'_capabilities', capabilities) |
50430
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
302 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
303 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
304 # logic for bundle auto-generation |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
305 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
306 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
307 configtable = {} |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
308 configitem = registrar.configitem(configtable) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
309 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
310 cmdtable = {} |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
311 command = registrar.command(cmdtable) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
312 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
313 configitem(b'clone-bundles', b'auto-generate.formats', default=list) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
314 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
315 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
316 configitem(b'clone-bundles', b'upload-command', default=None) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
317 |
50431
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
318 configitem(b'clone-bundles', b'delete-command', default=None) |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
319 |
50430
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
320 configitem(b'clone-bundles', b'url-template', default=None) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
321 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
322 configitem(b'devel', b'debug.clonebundles', default=False) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
323 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
324 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
325 # category for the post-close transaction hooks |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
326 CAT_POSTCLOSE = b"clonebundles-autobundles" |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
327 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
328 # template for bundle file names |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
329 BUNDLE_MASK = ( |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
330 b"full-%(bundle_type)s-%(revs)d_revs-%(tip_short)s_tip-%(op_id)s.hg" |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
331 ) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
332 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
333 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
334 # file in .hg/ use to track clonebundles being auto-generated |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
335 AUTO_GEN_FILE = b'clonebundles.auto-gen' |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
336 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
337 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
338 class BundleBase(object): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
339 """represents the core of properties that matters for us in a bundle |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
340 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
341 :bundle_type: the bundlespec (see hg help bundlespec) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
342 :revs: the number of revisions in the repo at bundle creation time |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
343 :tip_rev: the rev-num of the tip revision |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
344 :tip_node: the node id of the tip-most revision in the bundle |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
345 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
346 :ready: True if the bundle is ready to be served |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
347 """ |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
348 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
349 ready = False |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
350 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
351 def __init__(self, bundle_type, revs, tip_rev, tip_node): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
352 self.bundle_type = bundle_type |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
353 self.revs = revs |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
354 self.tip_rev = tip_rev |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
355 self.tip_node = tip_node |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
356 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
357 def valid_for(self, repo): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
358 """is this bundle applicable to the current repository |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
359 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
360 This is useful for detecting bundles made irrelevant by stripping. |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
361 """ |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
362 tip_node = node.bin(self.tip_node) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
363 return repo.changelog.index.get_rev(tip_node) == self.tip_rev |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
364 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
365 def __eq__(self, other): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
366 left = (self.ready, self.bundle_type, self.tip_rev, self.tip_node) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
367 right = (other.ready, other.bundle_type, other.tip_rev, other.tip_node) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
368 return left == right |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
369 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
370 def __neq__(self, other): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
371 return not self == other |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
372 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
373 def __cmp__(self, other): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
374 if self == other: |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
375 return 0 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
376 return -1 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
377 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
378 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
379 class RequestedBundle(BundleBase): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
380 """A bundle that should be generated. |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
381 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
382 Additional attributes compared to BundleBase |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
383 :heads: list of head revisions (as rev-num) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
384 :op_id: a "unique" identifier for the operation triggering the change |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
385 """ |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
386 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
387 def __init__(self, bundle_type, revs, tip_rev, tip_node, head_revs, op_id): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
388 self.head_revs = head_revs |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
389 self.op_id = op_id |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
390 super(RequestedBundle, self).__init__( |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
391 bundle_type, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
392 revs, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
393 tip_rev, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
394 tip_node, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
395 ) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
396 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
397 @property |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
398 def suggested_filename(self): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
399 """A filename that can be used for the generated bundle""" |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
400 data = { |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
401 b'bundle_type': self.bundle_type, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
402 b'revs': self.revs, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
403 b'heads': self.head_revs, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
404 b'tip_rev': self.tip_rev, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
405 b'tip_node': self.tip_node, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
406 b'tip_short': self.tip_node[:12], |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
407 b'op_id': self.op_id, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
408 } |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
409 return BUNDLE_MASK % data |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
410 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
411 def generate_bundle(self, repo, file_path): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
412 """generate the bundle at `filepath`""" |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
413 commands.bundle( |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
414 repo.ui, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
415 repo, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
416 file_path, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
417 base=[b"null"], |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
418 rev=self.head_revs, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
419 type=self.bundle_type, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
420 quiet=True, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
421 ) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
422 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
423 def generating(self, file_path, hostname=None, pid=None): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
424 """return a GeneratingBundle object from this object""" |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
425 if pid is None: |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
426 pid = os.getpid() |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
427 if hostname is None: |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
428 hostname = lock._getlockprefix() |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
429 return GeneratingBundle( |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
430 self.bundle_type, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
431 self.revs, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
432 self.tip_rev, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
433 self.tip_node, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
434 hostname, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
435 pid, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
436 file_path, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
437 ) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
438 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
439 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
440 class GeneratingBundle(BundleBase): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
441 """A bundle being generated |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
442 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
443 extra attributes compared to BundleBase: |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
444 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
445 :hostname: the hostname of the machine generating the bundle |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
446 :pid: the pid of the process generating the bundle |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
447 :filepath: the target filename of the bundle |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
448 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
449 These attributes exist to help detect stalled generation processes. |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
450 """ |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
451 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
452 ready = False |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
453 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
454 def __init__( |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
455 self, bundle_type, revs, tip_rev, tip_node, hostname, pid, filepath |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
456 ): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
457 self.hostname = hostname |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
458 self.pid = pid |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
459 self.filepath = filepath |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
460 super(GeneratingBundle, self).__init__( |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
461 bundle_type, revs, tip_rev, tip_node |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
462 ) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
463 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
464 @classmethod |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
465 def from_line(cls, line): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
466 """create an object by deserializing a line from AUTO_GEN_FILE""" |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
467 assert line.startswith(b'PENDING-v1 ') |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
468 ( |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
469 __, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
470 bundle_type, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
471 revs, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
472 tip_rev, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
473 tip_node, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
474 hostname, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
475 pid, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
476 filepath, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
477 ) = line.split() |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
478 hostname = util.urlreq.unquote(hostname) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
479 filepath = util.urlreq.unquote(filepath) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
480 revs = int(revs) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
481 tip_rev = int(tip_rev) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
482 pid = int(pid) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
483 return cls( |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
484 bundle_type, revs, tip_rev, tip_node, hostname, pid, filepath |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
485 ) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
486 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
487 def to_line(self): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
488 """serialize the object to include as a line in AUTO_GEN_FILE""" |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
489 templ = b"PENDING-v1 %s %d %d %s %s %d %s" |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
490 data = ( |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
491 self.bundle_type, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
492 self.revs, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
493 self.tip_rev, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
494 self.tip_node, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
495 util.urlreq.quote(self.hostname), |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
496 self.pid, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
497 util.urlreq.quote(self.filepath), |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
498 ) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
499 return templ % data |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
500 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
501 def __eq__(self, other): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
502 if not super(GeneratingBundle, self).__eq__(other): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
503 return False |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
504 left = (self.hostname, self.pid, self.filepath) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
505 right = (other.hostname, other.pid, other.filepath) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
506 return left == right |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
507 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
508 def uploaded(self, url, basename): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
509 """return a GeneratedBundle from this object""" |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
510 return GeneratedBundle( |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
511 self.bundle_type, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
512 self.revs, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
513 self.tip_rev, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
514 self.tip_node, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
515 url, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
516 basename, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
517 ) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
518 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
519 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
520 class GeneratedBundle(BundleBase): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
521 """A bundle that is done being generated and can be served |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
522 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
523 extra attributes compared to BundleBase: |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
524 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
525 :file_url: the url where the bundle is available. |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
526 :basename: the "basename" used to upload (useful for deletion) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
527 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
528 These attributes exist to generate a bundle manifest |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
529 (.hg/pullbundles.manifest) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
530 """ |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
531 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
532 ready = True |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
533 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
534 def __init__( |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
535 self, bundle_type, revs, tip_rev, tip_node, file_url, basename |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
536 ): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
537 self.file_url = file_url |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
538 self.basename = basename |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
539 super(GeneratedBundle, self).__init__( |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
540 bundle_type, revs, tip_rev, tip_node |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
541 ) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
542 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
543 @classmethod |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
544 def from_line(cls, line): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
545 """create an object by deserializing a line from AUTO_GEN_FILE""" |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
546 assert line.startswith(b'DONE-v1 ') |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
547 ( |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
548 __, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
549 bundle_type, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
550 revs, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
551 tip_rev, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
552 tip_node, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
553 file_url, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
554 basename, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
555 ) = line.split() |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
556 revs = int(revs) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
557 tip_rev = int(tip_rev) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
558 file_url = util.urlreq.unquote(file_url) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
559 return cls(bundle_type, revs, tip_rev, tip_node, file_url, basename) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
560 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
561 def to_line(self): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
562 """serialize the object to include as a line in AUTO_GEN_FILE""" |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
563 templ = b"DONE-v1 %s %d %d %s %s %s" |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
564 data = ( |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
565 self.bundle_type, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
566 self.revs, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
567 self.tip_rev, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
568 self.tip_node, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
569 util.urlreq.quote(self.file_url), |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
570 self.basename, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
571 ) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
572 return templ % data |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
573 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
574 def manifest_line(self): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
575 """serialize the object to include as a line in pullbundles.manifest""" |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
576 templ = b"%s BUNDLESPEC=%s REQUIRESNI=true" |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
577 return templ % (self.file_url, self.bundle_type) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
578 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
579 def __eq__(self, other): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
580 if not super(GeneratedBundle, self).__eq__(other): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
581 return False |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
582 return self.file_url == other.file_url |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
583 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
584 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
585 def parse_auto_gen(content): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
586 """parse the AUTO_GEN_FILE to return a list of Bundle object""" |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
587 bundles = [] |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
588 for line in content.splitlines(): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
589 if line.startswith(b'PENDING-v1 '): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
590 bundles.append(GeneratingBundle.from_line(line)) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
591 elif line.startswith(b'DONE-v1 '): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
592 bundles.append(GeneratedBundle.from_line(line)) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
593 return bundles |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
594 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
595 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
596 def dumps_auto_gen(bundles): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
597 """serialize a list of Bundle as a AUTO_GEN_FILE content""" |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
598 lines = [] |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
599 for b in bundles: |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
600 lines.append(b"%s\n" % b.to_line()) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
601 lines.sort() |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
602 return b"".join(lines) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
603 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
604 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
605 def read_auto_gen(repo): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
606 """read the AUTO_GEN_FILE for the <repo> a list of Bundle object""" |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
607 data = repo.vfs.tryread(AUTO_GEN_FILE) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
608 if not data: |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
609 return [] |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
610 return parse_auto_gen(data) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
611 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
612 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
613 def write_auto_gen(repo, bundles): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
614 """write a list of Bundle objects into the repo's AUTO_GEN_FILE""" |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
615 assert repo._cb_lock_ref is not None |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
616 data = dumps_auto_gen(bundles) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
617 with repo.vfs(AUTO_GEN_FILE, mode=b'wb', atomictemp=True) as f: |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
618 f.write(data) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
619 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
620 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
621 def generate_manifest(bundles): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
622 """write a list of Bundle objects into the repo's AUTO_GEN_FILE""" |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
623 bundles = list(bundles) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
624 bundles.sort(key=lambda b: b.bundle_type) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
625 lines = [] |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
626 for b in bundles: |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
627 lines.append(b"%s\n" % b.manifest_line()) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
628 return b"".join(lines) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
629 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
630 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
631 def update_ondisk_manifest(repo): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
632 """update the clonebundle manifest with latest url""" |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
633 with repo.clonebundles_lock(): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
634 bundles = read_auto_gen(repo) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
635 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
636 per_types = {} |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
637 for b in bundles: |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
638 if not (b.ready and b.valid_for(repo)): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
639 continue |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
640 current = per_types.get(b.bundle_type) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
641 if current is not None and current.revs >= b.revs: |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
642 continue |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
643 per_types[b.bundle_type] = b |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
644 manifest = generate_manifest(per_types.values()) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
645 with repo.vfs( |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
646 bundlecaches.CB_MANIFEST_FILE, mode=b"wb", atomictemp=True |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
647 ) as f: |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
648 f.write(manifest) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
649 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
650 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
651 def update_bundle_list(repo, new_bundles=(), del_bundles=()): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
652 """modify the repo's AUTO_GEN_FILE |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
653 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
654 This method also regenerates the clone bundle manifest when needed""" |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
655 with repo.clonebundles_lock(): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
656 bundles = read_auto_gen(repo) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
657 if del_bundles: |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
658 bundles = [b for b in bundles if b not in del_bundles] |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
659 new_bundles = [b for b in new_bundles if b not in bundles] |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
660 bundles.extend(new_bundles) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
661 write_auto_gen(repo, bundles) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
662 all_changed = [] |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
663 all_changed.extend(new_bundles) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
664 all_changed.extend(del_bundles) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
665 if any(b.ready for b in all_changed): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
666 update_ondisk_manifest(repo) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
667 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
668 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
669 def cleanup_tmp_bundle(repo, target): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
670 """remove a GeneratingBundle file and entry""" |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
671 assert not target.ready |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
672 with repo.clonebundles_lock(): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
673 repo.vfs.tryunlink(target.filepath) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
674 update_bundle_list(repo, del_bundles=[target]) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
675 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
676 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
677 def finalize_one_bundle(repo, target): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
678 """upload a generated bundle and advertise it in the clonebundles.manifest""" |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
679 with repo.clonebundles_lock(): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
680 bundles = read_auto_gen(repo) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
681 if target in bundles and target.valid_for(repo): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
682 result = upload_bundle(repo, target) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
683 update_bundle_list(repo, new_bundles=[result]) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
684 cleanup_tmp_bundle(repo, target) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
685 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
686 |
50431
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
687 def find_outdated_bundles(repo, bundles): |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
688 """finds outdated bundles""" |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
689 olds = [] |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
690 per_types = {} |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
691 for b in bundles: |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
692 if not b.valid_for(repo): |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
693 olds.append(b) |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
694 continue |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
695 l = per_types.setdefault(b.bundle_type, []) |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
696 l.append(b) |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
697 for key in sorted(per_types): |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
698 all = per_types[key] |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
699 if len(all) > 1: |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
700 all.sort(key=lambda b: b.revs, reverse=True) |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
701 olds.extend(all[1:]) |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
702 return olds |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
703 |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
704 |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
705 def collect_garbage(repo): |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
706 """finds outdated bundles and get them deleted""" |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
707 with repo.clonebundles_lock(): |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
708 bundles = read_auto_gen(repo) |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
709 olds = find_outdated_bundles(repo, bundles) |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
710 for o in olds: |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
711 delete_bundle(repo, o) |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
712 update_bundle_list(repo, del_bundles=olds) |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
713 |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
714 |
50430
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
715 def upload_bundle(repo, bundle): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
716 """upload the result of a GeneratingBundle and return a GeneratedBundle |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
717 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
718 The upload is done using the `clone-bundles.upload-command` |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
719 """ |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
720 cmd = repo.ui.config(b'clone-bundles', b'upload-command') |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
721 url = repo.ui.config(b'clone-bundles', b'url-template') |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
722 basename = repo.vfs.basename(bundle.filepath) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
723 filepath = procutil.shellquote(bundle.filepath) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
724 variables = { |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
725 b'HGCB_BUNDLE_PATH': filepath, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
726 b'HGCB_BUNDLE_BASENAME': basename, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
727 } |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
728 env = procutil.shellenviron(environ=variables) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
729 ret = repo.ui.system(cmd, environ=env) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
730 if ret: |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
731 raise error.Abort(b"command returned status %d: %s" % (ret, cmd)) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
732 url = ( |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
733 url.decode('utf8') |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
734 .format(basename=basename.decode('utf8')) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
735 .encode('utf8') |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
736 ) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
737 return bundle.uploaded(url, basename) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
738 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
739 |
50431
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
740 def delete_bundle(repo, bundle): |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
741 """delete a bundle from storage""" |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
742 assert bundle.ready |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
743 msg = b'clone-bundles: deleting bundle %s\n' |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
744 msg %= bundle.basename |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
745 if repo.ui.configbool(b'devel', b'debug.clonebundles'): |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
746 repo.ui.write(msg) |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
747 else: |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
748 repo.ui.debug(msg) |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
749 |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
750 cmd = repo.ui.config(b'clone-bundles', b'delete-command') |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
751 variables = { |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
752 b'HGCB_BUNDLE_URL': bundle.file_url, |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
753 b'HGCB_BASENAME': bundle.basename, |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
754 } |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
755 env = procutil.shellenviron(environ=variables) |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
756 ret = repo.ui.system(cmd, environ=env) |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
757 if ret: |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
758 raise error.Abort(b"command returned status %d: %s" % (ret, cmd)) |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
759 |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
760 |
50430
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
761 def auto_bundle_needed_actions(repo, bundles, op_id): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
762 """find the list of bundles that need action |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
763 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
764 returns a list of RequestedBundle objects that need to be generated and |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
765 uploaded.""" |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
766 create_bundles = [] |
50431
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
767 delete_bundles = [] |
50430
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
768 repo = repo.filtered(b"immutable") |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
769 targets = repo.ui.configlist(b'clone-bundles', b'auto-generate.formats') |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
770 revs = len(repo.changelog) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
771 generic_data = { |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
772 'revs': revs, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
773 'head_revs': repo.changelog.headrevs(), |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
774 'tip_rev': repo.changelog.tiprev(), |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
775 'tip_node': node.hex(repo.changelog.tip()), |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
776 'op_id': op_id, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
777 } |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
778 for t in targets: |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
779 data = generic_data.copy() |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
780 data['bundle_type'] = t |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
781 b = RequestedBundle(**data) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
782 create_bundles.append(b) |
50431
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
783 delete_bundles.extend(find_outdated_bundles(repo, bundles)) |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
784 return create_bundles, delete_bundles |
50430
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
785 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
786 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
787 def start_one_bundle(repo, bundle): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
788 """start the generation of a single bundle file |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
789 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
790 the `bundle` argument should be a RequestedBundle object. |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
791 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
792 This data is passed to the `debugmakeclonebundles` "as is". |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
793 """ |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
794 data = util.pickle.dumps(bundle) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
795 cmd = [procutil.hgexecutable(), b'--cwd', repo.path, INTERNAL_CMD] |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
796 env = procutil.shellenviron() |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
797 msg = b'clone-bundles: starting bundle generation: %s\n' |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
798 stdout = None |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
799 stderr = None |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
800 waits = [] |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
801 record_wait = None |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
802 if repo.ui.configbool(b'devel', b'debug.clonebundles'): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
803 stdout = procutil.stdout |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
804 stderr = procutil.stderr |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
805 repo.ui.write(msg % bundle.bundle_type) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
806 record_wait = waits.append |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
807 else: |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
808 repo.ui.debug(msg % bundle.bundle_type) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
809 bg = procutil.runbgcommand |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
810 bg( |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
811 cmd, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
812 env, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
813 stdin_bytes=data, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
814 stdout=stdout, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
815 stderr=stderr, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
816 record_wait=record_wait, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
817 ) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
818 for f in waits: |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
819 f() |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
820 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
821 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
822 INTERNAL_CMD = b'debug::internal-make-clone-bundles' |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
823 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
824 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
825 @command(INTERNAL_CMD, [], b'') |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
826 def debugmakeclonebundles(ui, repo): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
827 """Internal command to auto-generate debug bundles""" |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
828 requested_bundle = util.pickle.load(procutil.stdin) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
829 procutil.stdin.close() |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
830 |
50431
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
831 collect_garbage(repo) |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
832 |
50430
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
833 fname = requested_bundle.suggested_filename |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
834 fpath = repo.vfs.makedirs(b'tmp-bundles') |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
835 fpath = repo.vfs.join(b'tmp-bundles', fname) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
836 bundle = requested_bundle.generating(fpath) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
837 update_bundle_list(repo, new_bundles=[bundle]) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
838 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
839 requested_bundle.generate_bundle(repo, fpath) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
840 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
841 repo.invalidate() |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
842 finalize_one_bundle(repo, bundle) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
843 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
844 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
845 def make_auto_bundler(source_repo): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
846 reporef = weakref.ref(source_repo) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
847 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
848 def autobundle(tr): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
849 repo = reporef() |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
850 assert repo is not None |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
851 bundles = read_auto_gen(repo) |
50431
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
852 new, __ = auto_bundle_needed_actions(repo, bundles, b"%d_txn" % id(tr)) |
50430
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
853 for data in new: |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
854 start_one_bundle(repo, data) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
855 return None |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
856 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
857 return autobundle |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
858 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
859 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
860 def reposetup(ui, repo): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
861 """install the two pieces needed for automatic clonebundle generation |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
862 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
863 - add a "post-close" hook that fires bundling when needed |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
864 - introduce a clone-bundle lock to let multiple processes meddle with the |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
865 state files. |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
866 """ |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
867 if not repo.local(): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
868 return |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
869 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
870 class autobundlesrepo(repo.__class__): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
871 def transaction(self, *args, **kwargs): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
872 tr = super(autobundlesrepo, self).transaction(*args, **kwargs) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
873 targets = repo.ui.configlist( |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
874 b'clone-bundles', b'auto-generate.formats' |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
875 ) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
876 if targets: |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
877 tr.addpostclose(CAT_POSTCLOSE, make_auto_bundler(self)) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
878 return tr |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
879 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
880 @localrepo.unfilteredmethod |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
881 def clonebundles_lock(self, wait=True): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
882 '''Lock the repository file related to clone bundles''' |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
883 if not util.safehasattr(self, '_cb_lock_ref'): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
884 self._cb_lock_ref = None |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
885 l = self._currentlock(self._cb_lock_ref) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
886 if l is not None: |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
887 l.lock() |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
888 return l |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
889 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
890 l = self._lock( |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
891 vfs=self.vfs, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
892 lockname=b"clonebundleslock", |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
893 wait=wait, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
894 releasefn=None, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
895 acquirefn=None, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
896 desc=_(b'repository %s') % self.origroot, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
897 ) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
898 self._cb_lock_ref = weakref.ref(l) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
899 return l |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
900 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
901 repo._wlockfreeprefix.add(AUTO_GEN_FILE) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
902 repo._wlockfreeprefix.add(bundlecaches.CB_MANIFEST_FILE) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
903 repo.__class__ = autobundlesrepo |