Mercurial > hg
annotate hgext/clonebundles.py @ 50620:26e63204c31e
safehasattr: pass attribute name as string instead of bytes
This is a step toward replacing `util.safehasattr` usage with plain `hasattr`.
The builtin function behave poorly in Python2 but this was fixed in Python3.
These change are done one by one as they tend to have a small odd to trigger
puzzling breackage.
author | Pierre-Yves David <pierre-yves.david@octobus.net> |
---|---|
date | Thu, 02 Feb 2023 17:37:51 +0100 |
parents | 3973b1dc3ee3 |
children | 60f9602b413e |
rev | line source |
---|---|
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
1 # This software may be used and distributed according to the terms of the |
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
2 # GNU General Public License version 2 or any later version. |
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
3 |
27738
a0e783d26e81
exchange: make clone bundles non-experimental and enabled by default
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27737
diff
changeset
|
4 """advertise pre-generated bundles to seed clones |
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
5 |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
6 "clonebundles" is a server-side extension used to advertise the existence |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
7 of pre-generated, externally hosted bundle files to clients that are |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
8 cloning so that cloning can be faster, more reliable, and require less |
37498
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
9 resources on the server. "pullbundles" is a related feature for sending |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
10 pre-generated bundle files to clients as part of pull operations. |
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
11 |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
12 Cloning can be a CPU and I/O intensive operation on servers. Traditionally, |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
13 the server, in response to a client's request to clone, dynamically generates |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
14 a bundle containing the entire repository content and sends it to the client. |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
15 There is no caching on the server and the server will have to redundantly |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
16 generate the same outgoing bundle in response to each clone request. For |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
17 servers with large repositories or with high clone volume, the load from |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
18 clones can make scaling the server challenging and costly. |
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
19 |
37498
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
20 This extension provides server operators the ability to offload |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
21 potentially expensive clone load to an external service. Pre-generated |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
22 bundles also allow using more CPU intensive compression, reducing the |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
23 effective bandwidth requirements. |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
24 |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
25 Here's how clone bundles work: |
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
26 |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
27 1. A server operator establishes a mechanism for making bundle files available |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
28 on a hosting service where Mercurial clients can fetch them. |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
29 2. A manifest file listing available bundle URLs and some optional metadata |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
30 is added to the Mercurial repository on the server. |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
31 3. A client initiates a clone against a clone bundles aware server. |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
32 4. The client sees the server is advertising clone bundles and fetches the |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
33 manifest listing available bundles. |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
34 5. The client filters and sorts the available bundles based on what it |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
35 supports and prefers. |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
36 6. The client downloads and applies an available bundle from the |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
37 server-specified URL. |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
38 7. The client reconnects to the original server and performs the equivalent |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
39 of :hg:`pull` to retrieve all repository data not in the bundle. (The |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
40 repository could have been updated between when the bundle was created |
37498
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
41 and when the client started the clone.) This may use "pullbundles". |
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
42 |
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
43 Instead of the server generating full repository bundles for every clone |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
44 request, it generates full bundles once and they are subsequently reused to |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
45 bootstrap new clones. The server may still transfer data at clone time. |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
46 However, this is only data that has been added/changed since the bundle was |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
47 created. For large, established repositories, this can reduce server load for |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
48 clones to less than 1% of original. |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
49 |
37498
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
50 Here's how pullbundles work: |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
51 |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
52 1. A manifest file listing available bundles and describing the revisions |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
53 is added to the Mercurial repository on the server. |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
54 2. A new-enough client informs the server that it supports partial pulls |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
55 and initiates a pull. |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
56 3. If the server has pull bundles enabled and sees the client advertising |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
57 partial pulls, it checks for a matching pull bundle in the manifest. |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
58 A bundle matches if the format is supported by the client, the client |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
59 has the required revisions already and needs something from the bundle. |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
60 4. If there is at least one matching bundle, the server sends it to the client. |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
61 5. The client applies the bundle and notices that the server reply was |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
62 incomplete. It initiates another pull. |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
63 |
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
64 To work, this extension requires the following of server operators: |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
65 |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
66 * Generating bundle files of repository content (typically periodically, |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
67 such as once per day). |
37498
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
68 * Clone bundles: A file server that clients have network access to and that |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
69 Python knows how to talk to through its normal URL handling facility |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
70 (typically an HTTP/HTTPS server). |
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
71 * A process for keeping the bundles manifest in sync with available bundle |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
72 files. |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
73 |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
74 Strictly speaking, using a static file hosting server isn't required: a server |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
75 operator could use a dynamic service for retrieving bundle data. However, |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
76 static file hosting services are simple and scalable and should be sufficient |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
77 for most needs. |
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
78 |
26884
762bf510b42c
clonebundles: fix typo s/comand/command/
Javi Merino <merino.jav@gmail.com>
parents:
26857
diff
changeset
|
79 Bundle files can be generated with the :hg:`bundle` command. Typically |
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
80 :hg:`bundle --all` is used to produce a bundle of the entire repository. |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
81 |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
82 :hg:`debugcreatestreamclonebundle` can be used to produce a special |
37498
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
83 *streaming clonebundle*. These are bundle files that are extremely efficient |
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
84 to produce and consume (read: fast). However, they are larger than |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
85 traditional bundle formats and require that clients support the exact set |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
86 of repository data store formats in use by the repository that created them. |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
87 Typically, a newer server can serve data that is compatible with older clients. |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
88 However, *streaming clone bundles* don't have this guarantee. **Server |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
89 operators need to be aware that newer versions of Mercurial may produce |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
90 streaming clone bundles incompatible with older Mercurial versions.** |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
91 |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
92 A server operator is responsible for creating a ``.hg/clonebundles.manifest`` |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
93 file containing the list of available bundle files suitable for seeding |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
94 clones. If this file does not exist, the repository will not advertise the |
37498
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
95 existence of clone bundles when clients connect. For pull bundles, |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
96 ``.hg/pullbundles.manifest`` is used. |
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
97 |
32454
702af1ad3b18
clonebundles: fix missing newline character
Matt Harbison <matt_harbison@yahoo.com>
parents:
31146
diff
changeset
|
98 The manifest file contains a newline (\\n) delimited list of entries. |
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
99 |
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
100 Each line in this file defines an available bundle. Lines have the format: |
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
101 |
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
102 <URL> [<key>=<value>[ <key>=<value>]] |
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
103 |
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
104 That is, a URL followed by an optional, space-delimited list of key=value |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
105 pairs describing additional properties of this bundle. Both keys and values |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
106 are URI encoded. |
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
107 |
37498
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
108 For pull bundles, the URL is a path under the ``.hg`` directory of the |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
109 repository. |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
110 |
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
111 Keys in UPPERCASE are reserved for use by Mercurial and are defined below. |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
112 All non-uppercase keys can be used by site installations. An example use |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
113 for custom properties is to use the *datacenter* attribute to define which |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
114 data center a file is hosted in. Clients could then prefer a server in the |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
115 data center closest to them. |
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
116 |
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
117 The following reserved keys are currently defined: |
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
118 |
26644
74de1c59f71c
clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26623
diff
changeset
|
119 BUNDLESPEC |
74de1c59f71c
clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26623
diff
changeset
|
120 A "bundle specification" string that describes the type of the bundle. |
74de1c59f71c
clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26623
diff
changeset
|
121 |
74de1c59f71c
clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26623
diff
changeset
|
122 These are string values that are accepted by the "--type" argument of |
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
123 :hg:`bundle`. |
26644
74de1c59f71c
clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26623
diff
changeset
|
124 |
74de1c59f71c
clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26623
diff
changeset
|
125 The values are parsed in strict mode, which means they must be of the |
74de1c59f71c
clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26623
diff
changeset
|
126 "<compression>-<type>" form. See |
74de1c59f71c
clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26623
diff
changeset
|
127 mercurial.exchange.parsebundlespec() for more details. |
74de1c59f71c
clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26623
diff
changeset
|
128 |
27886
0288e63ea3be
clonebundles: improve BUNDLESPEC documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27738
diff
changeset
|
129 :hg:`debugbundle --spec` can be used to print the bundle specification |
0288e63ea3be
clonebundles: improve BUNDLESPEC documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27738
diff
changeset
|
130 string for a bundle file. The output of this command can be used verbatim |
0288e63ea3be
clonebundles: improve BUNDLESPEC documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27738
diff
changeset
|
131 for the value of ``BUNDLESPEC`` (it is already escaped). |
0288e63ea3be
clonebundles: improve BUNDLESPEC documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27738
diff
changeset
|
132 |
26644
74de1c59f71c
clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26623
diff
changeset
|
133 Clients will automatically filter out specifications that are unknown or |
74de1c59f71c
clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26623
diff
changeset
|
134 unsupported so they won't attempt to download something that likely won't |
74de1c59f71c
clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26623
diff
changeset
|
135 apply. |
74de1c59f71c
clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26623
diff
changeset
|
136 |
74de1c59f71c
clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26623
diff
changeset
|
137 The actual value doesn't impact client behavior beyond filtering: |
74de1c59f71c
clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26623
diff
changeset
|
138 clients will still sniff the bundle type from the header of downloaded |
74de1c59f71c
clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26623
diff
changeset
|
139 files. |
26645
2faa7671a4b3
clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26644
diff
changeset
|
140 |
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
141 **Use of this key is highly recommended**, as it allows clients to |
27886
0288e63ea3be
clonebundles: improve BUNDLESPEC documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27738
diff
changeset
|
142 easily skip unsupported bundles. If this key is not defined, an old |
0288e63ea3be
clonebundles: improve BUNDLESPEC documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27738
diff
changeset
|
143 client may attempt to apply a bundle that it is incapable of reading. |
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
144 |
26645
2faa7671a4b3
clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26644
diff
changeset
|
145 REQUIRESNI |
2faa7671a4b3
clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26644
diff
changeset
|
146 Whether Server Name Indication (SNI) is required to connect to the URL. |
2faa7671a4b3
clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26644
diff
changeset
|
147 SNI allows servers to use multiple certificates on the same IP. It is |
2faa7671a4b3
clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26644
diff
changeset
|
148 somewhat common in CDNs and other hosting providers. Older Python |
2faa7671a4b3
clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26644
diff
changeset
|
149 versions do not support SNI. Defining this attribute enables clients |
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
150 with older Python versions to filter this entry without experiencing |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
151 an opaque SSL failure at connection time. |
26645
2faa7671a4b3
clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26644
diff
changeset
|
152 |
2faa7671a4b3
clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26644
diff
changeset
|
153 If this is defined, it is important to advertise a non-SNI fallback |
2faa7671a4b3
clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26644
diff
changeset
|
154 URL or clients running old Python releases may not be able to clone |
2faa7671a4b3
clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26644
diff
changeset
|
155 with the clonebundles facility. |
2faa7671a4b3
clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26644
diff
changeset
|
156 |
2faa7671a4b3
clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26644
diff
changeset
|
157 Value should be "true". |
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
158 |
45252
20d110e6eea6
clonebundles: document REQUIREDRAM key
Gregory Szorc <gregory.szorc@gmail.com>
parents:
43077
diff
changeset
|
159 REQUIREDRAM |
20d110e6eea6
clonebundles: document REQUIREDRAM key
Gregory Szorc <gregory.szorc@gmail.com>
parents:
43077
diff
changeset
|
160 Value specifies expected memory requirements to decode the payload. |
20d110e6eea6
clonebundles: document REQUIREDRAM key
Gregory Szorc <gregory.szorc@gmail.com>
parents:
43077
diff
changeset
|
161 Values can have suffixes for common bytes sizes. e.g. "64MB". |
20d110e6eea6
clonebundles: document REQUIREDRAM key
Gregory Szorc <gregory.szorc@gmail.com>
parents:
43077
diff
changeset
|
162 |
20d110e6eea6
clonebundles: document REQUIREDRAM key
Gregory Szorc <gregory.szorc@gmail.com>
parents:
43077
diff
changeset
|
163 This key is often used with zstd-compressed bundles using a high |
20d110e6eea6
clonebundles: document REQUIREDRAM key
Gregory Szorc <gregory.szorc@gmail.com>
parents:
43077
diff
changeset
|
164 compression level / window size, which can require 100+ MB of memory |
20d110e6eea6
clonebundles: document REQUIREDRAM key
Gregory Szorc <gregory.szorc@gmail.com>
parents:
43077
diff
changeset
|
165 to decode. |
20d110e6eea6
clonebundles: document REQUIREDRAM key
Gregory Szorc <gregory.szorc@gmail.com>
parents:
43077
diff
changeset
|
166 |
37498
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
167 heads |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
168 Used for pull bundles. This contains the ``;`` separated changeset |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
169 hashes of the heads of the bundle content. |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
170 |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
171 bases |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
172 Used for pull bundles. This contains the ``;`` separated changeset |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
173 hashes of the roots of the bundle content. This can be skipped if |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
174 the bundle was created without ``--base``. |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
175 |
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
176 Manifests can contain multiple entries. Assuming metadata is defined, clients |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
177 will filter entries from the manifest that they don't support. The remaining |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
178 entries are optionally sorted by client preferences |
32773
d25802b0eef5
clonebundles: reference correct config option
Gregory Szorc <gregory.szorc@gmail.com>
parents:
32454
diff
changeset
|
179 (``ui.clonebundleprefers`` config option). The client then attempts |
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
180 to fetch the bundle at the first URL in the remaining list. |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
181 |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
182 **Errors when downloading a bundle will fail the entire clone operation: |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
183 clients do not automatically fall back to a traditional clone.** The reason |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
184 for this is that if a server is using clone bundles, it is probably doing so |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
185 because the feature is necessary to help it scale. In other words, there |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
186 is an assumption that clone load will be offloaded to another service and |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
187 that the Mercurial server isn't responsible for serving this clone load. |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
188 If that other service experiences issues and clients start mass falling back to |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
189 the original Mercurial server, the added clone load could overwhelm the server |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
190 due to unexpected load and effectively take it offline. Not having clients |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
191 automatically fall back to cloning from the original server mitigates this |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
192 scenario. |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
193 |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
194 Because there is no automatic Mercurial server fallback on failure of the |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
195 bundle hosting service, it is important for server operators to view the bundle |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
196 hosting service as an extension of the Mercurial server in terms of |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
197 availability and service level agreements: if the bundle hosting service goes |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
198 down, so does the ability for clients to clone. Note: clients will see a |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
199 message informing them how to bypass the clone bundles facility when a failure |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
200 occurs. So server operators should prepare for some people to follow these |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
201 instructions when a failure occurs, thus driving more load to the original |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
202 Mercurial server when the bundle hosting service fails. |
50430
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
203 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
204 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
205 auto-generation of clone bundles |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
206 -------------------------------- |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
207 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
208 It is possible to set Mercurial to automatically re-generate clone bundles when |
50432
5b70b9f5a2f9
clone-bundles: only regenerate the clone bundle when cached ration is low
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50431
diff
changeset
|
209 enough new content is available. |
50430
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
210 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
211 Mercurial will take care of the process asynchronously. The defined list of |
50431
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
212 bundle-type will be generated, uploaded, and advertised. Older bundles will get |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
213 decommissioned as newer ones replace them. |
50430
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
214 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
215 Bundles Generation: |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
216 ................... |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
217 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
218 The extension can generate multiple variants of the clone bundle. Each |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
219 different variant will be defined by the "bundle-spec" they use:: |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
220 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
221 [clone-bundles] |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
222 auto-generate.formats= zstd-v2, gzip-v2 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
223 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
224 See `hg help bundlespec` for details about available options. |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
225 |
50434
65fb4cedd5ea
clone-bundles: add a configuration to control auto-generation on changes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50433
diff
changeset
|
226 By default, new bundles are generated when 5% of the repository contents or at |
65fb4cedd5ea
clone-bundles: add a configuration to control auto-generation on changes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50433
diff
changeset
|
227 least 1000 revisions are not contained in the cached bundles. This option can |
65fb4cedd5ea
clone-bundles: add a configuration to control auto-generation on changes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50433
diff
changeset
|
228 be controlled by the `clone-bundles.trigger.below-bundled-ratio` option |
65fb4cedd5ea
clone-bundles: add a configuration to control auto-generation on changes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50433
diff
changeset
|
229 (default 0.95) and the `clone-bundles.trigger.revs` option (default 1000):: |
50433
d611805e7374
clone-bundles: also control automation based on absolute number of revisions
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50432
diff
changeset
|
230 |
d611805e7374
clone-bundles: also control automation based on absolute number of revisions
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50432
diff
changeset
|
231 [clone-bundles] |
d611805e7374
clone-bundles: also control automation based on absolute number of revisions
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50432
diff
changeset
|
232 trigger.below-bundled-ratio=0.95 |
d611805e7374
clone-bundles: also control automation based on absolute number of revisions
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50432
diff
changeset
|
233 trigger.revs=1000 |
50432
5b70b9f5a2f9
clone-bundles: only regenerate the clone bundle when cached ration is low
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50431
diff
changeset
|
234 |
50435
23db5f15cc0c
clone-bundles: introduce a command to refresh bundle
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50434
diff
changeset
|
235 This logic can be manually triggered using the `admin::clone-bundles-refresh` |
23db5f15cc0c
clone-bundles: introduce a command to refresh bundle
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50434
diff
changeset
|
236 command, or automatically on each repository change if |
50434
65fb4cedd5ea
clone-bundles: add a configuration to control auto-generation on changes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50433
diff
changeset
|
237 `clone-bundles.auto-generate.on-change` is set to `yes`. |
65fb4cedd5ea
clone-bundles: add a configuration to control auto-generation on changes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50433
diff
changeset
|
238 |
65fb4cedd5ea
clone-bundles: add a configuration to control auto-generation on changes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50433
diff
changeset
|
239 [clone-bundles] |
65fb4cedd5ea
clone-bundles: add a configuration to control auto-generation on changes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50433
diff
changeset
|
240 auto-generate.on-change=yes |
65fb4cedd5ea
clone-bundles: add a configuration to control auto-generation on changes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50433
diff
changeset
|
241 auto-generate.formats= zstd-v2, gzip-v2 |
65fb4cedd5ea
clone-bundles: add a configuration to control auto-generation on changes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50433
diff
changeset
|
242 |
50430
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
243 Bundles Upload and Serving: |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
244 ........................... |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
245 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
246 The generated bundles need to be made available to users through a "public" URL. |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
247 This should be donne through `clone-bundles.upload-command` configuration. The |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
248 value of this command should be a shell command. It will have access to the |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
249 bundle file path through the `$HGCB_BUNDLE_PATH` variable. And the expected |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
250 basename in the "public" URL is accessible at:: |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
251 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
252 [clone-bundles] |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
253 upload-command=sftp put $HGCB_BUNDLE_PATH \ |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
254 sftp://bundles.host/clone-bundles/$HGCB_BUNDLE_BASENAME |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
255 |
50431
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
256 If the file was already uploaded, the command must still succeed. |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
257 |
50430
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
258 After upload, the file should be available at an url defined by |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
259 `clone-bundles.url-template`. |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
260 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
261 [clone-bundles] |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
262 url-template=https://bundles.host/cache/clone-bundles/{basename} |
50431
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
263 |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
264 Old bundles cleanup: |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
265 .................... |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
266 |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
267 When new bundles are generated, the older ones are no longer necessary and can |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
268 be removed from storage. This is done through the `clone-bundles.delete-command` |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
269 configuration. The command is given the url of the artifact to delete through |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
270 the `$HGCB_BUNDLE_URL` environment variable. |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
271 |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
272 [clone-bundles] |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
273 delete-command=sftp rm sftp://bundles.host/clone-bundles/$HGCB_BUNDLE_BASENAME |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
274 |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
275 If the file was already deleted, the command must still succeed. |
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
276 """ |
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
277 |
28095
7fa139eaebb4
clonebundles: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27886
diff
changeset
|
278 |
50430
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
279 import os |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
280 import weakref |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
281 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
282 from mercurial.i18n import _ |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
283 |
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
284 from mercurial import ( |
45785
80f32ec8653a
clonebundle: move the manifest filename to a constant
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
45252
diff
changeset
|
285 bundlecaches, |
50430
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
286 commands, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
287 error, |
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
288 extensions, |
50430
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
289 localrepo, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
290 lock, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
291 node, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
292 registrar, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
293 util, |
37785
b4d85bc122bd
wireproto: rename wireproto to wireprotov1server (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37498
diff
changeset
|
294 wireprotov1server, |
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
295 ) |
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
296 |
50430
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
297 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
298 from mercurial.utils import ( |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
299 procutil, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
300 ) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
301 |
43077
687b865b95ad
formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents:
43076
diff
changeset
|
302 testedwith = b'ships-with-hg-core' |
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
303 |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
37785
diff
changeset
|
304 |
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
305 def capabilities(orig, repo, proto): |
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
306 caps = orig(repo, proto) |
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
307 |
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
308 # Only advertise if a manifest exists. This does add some I/O to requests. |
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
309 # But this should be cheaper than a wasted network round trip due to |
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
310 # missing file. |
45785
80f32ec8653a
clonebundle: move the manifest filename to a constant
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
45252
diff
changeset
|
311 if repo.vfs.exists(bundlecaches.CB_MANIFEST_FILE): |
43077
687b865b95ad
formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents:
43076
diff
changeset
|
312 caps.append(b'clonebundles') |
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
313 |
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
314 return caps |
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
315 |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
37785
diff
changeset
|
316 |
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
317 def extsetup(ui): |
43077
687b865b95ad
formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents:
43076
diff
changeset
|
318 extensions.wrapfunction(wireprotov1server, b'_capabilities', capabilities) |
50430
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
319 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
320 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
321 # logic for bundle auto-generation |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
322 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
323 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
324 configtable = {} |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
325 configitem = registrar.configitem(configtable) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
326 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
327 cmdtable = {} |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
328 command = registrar.command(cmdtable) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
329 |
50434
65fb4cedd5ea
clone-bundles: add a configuration to control auto-generation on changes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50433
diff
changeset
|
330 configitem(b'clone-bundles', b'auto-generate.on-change', default=False) |
50430
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
331 configitem(b'clone-bundles', b'auto-generate.formats', default=list) |
50432
5b70b9f5a2f9
clone-bundles: only regenerate the clone bundle when cached ration is low
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50431
diff
changeset
|
332 configitem(b'clone-bundles', b'trigger.below-bundled-ratio', default=0.95) |
50433
d611805e7374
clone-bundles: also control automation based on absolute number of revisions
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50432
diff
changeset
|
333 configitem(b'clone-bundles', b'trigger.revs', default=1000) |
50430
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
334 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
335 configitem(b'clone-bundles', b'upload-command', default=None) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
336 |
50431
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
337 configitem(b'clone-bundles', b'delete-command', default=None) |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
338 |
50430
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
339 configitem(b'clone-bundles', b'url-template', default=None) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
340 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
341 configitem(b'devel', b'debug.clonebundles', default=False) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
342 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
343 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
344 # category for the post-close transaction hooks |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
345 CAT_POSTCLOSE = b"clonebundles-autobundles" |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
346 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
347 # template for bundle file names |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
348 BUNDLE_MASK = ( |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
349 b"full-%(bundle_type)s-%(revs)d_revs-%(tip_short)s_tip-%(op_id)s.hg" |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
350 ) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
351 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
352 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
353 # file in .hg/ use to track clonebundles being auto-generated |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
354 AUTO_GEN_FILE = b'clonebundles.auto-gen' |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
355 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
356 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
357 class BundleBase(object): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
358 """represents the core of properties that matters for us in a bundle |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
359 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
360 :bundle_type: the bundlespec (see hg help bundlespec) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
361 :revs: the number of revisions in the repo at bundle creation time |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
362 :tip_rev: the rev-num of the tip revision |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
363 :tip_node: the node id of the tip-most revision in the bundle |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
364 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
365 :ready: True if the bundle is ready to be served |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
366 """ |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
367 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
368 ready = False |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
369 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
370 def __init__(self, bundle_type, revs, tip_rev, tip_node): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
371 self.bundle_type = bundle_type |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
372 self.revs = revs |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
373 self.tip_rev = tip_rev |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
374 self.tip_node = tip_node |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
375 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
376 def valid_for(self, repo): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
377 """is this bundle applicable to the current repository |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
378 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
379 This is useful for detecting bundles made irrelevant by stripping. |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
380 """ |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
381 tip_node = node.bin(self.tip_node) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
382 return repo.changelog.index.get_rev(tip_node) == self.tip_rev |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
383 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
384 def __eq__(self, other): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
385 left = (self.ready, self.bundle_type, self.tip_rev, self.tip_node) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
386 right = (other.ready, other.bundle_type, other.tip_rev, other.tip_node) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
387 return left == right |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
388 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
389 def __neq__(self, other): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
390 return not self == other |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
391 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
392 def __cmp__(self, other): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
393 if self == other: |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
394 return 0 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
395 return -1 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
396 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
397 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
398 class RequestedBundle(BundleBase): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
399 """A bundle that should be generated. |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
400 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
401 Additional attributes compared to BundleBase |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
402 :heads: list of head revisions (as rev-num) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
403 :op_id: a "unique" identifier for the operation triggering the change |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
404 """ |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
405 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
406 def __init__(self, bundle_type, revs, tip_rev, tip_node, head_revs, op_id): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
407 self.head_revs = head_revs |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
408 self.op_id = op_id |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
409 super(RequestedBundle, self).__init__( |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
410 bundle_type, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
411 revs, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
412 tip_rev, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
413 tip_node, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
414 ) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
415 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
416 @property |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
417 def suggested_filename(self): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
418 """A filename that can be used for the generated bundle""" |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
419 data = { |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
420 b'bundle_type': self.bundle_type, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
421 b'revs': self.revs, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
422 b'heads': self.head_revs, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
423 b'tip_rev': self.tip_rev, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
424 b'tip_node': self.tip_node, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
425 b'tip_short': self.tip_node[:12], |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
426 b'op_id': self.op_id, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
427 } |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
428 return BUNDLE_MASK % data |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
429 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
430 def generate_bundle(self, repo, file_path): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
431 """generate the bundle at `filepath`""" |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
432 commands.bundle( |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
433 repo.ui, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
434 repo, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
435 file_path, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
436 base=[b"null"], |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
437 rev=self.head_revs, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
438 type=self.bundle_type, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
439 quiet=True, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
440 ) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
441 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
442 def generating(self, file_path, hostname=None, pid=None): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
443 """return a GeneratingBundle object from this object""" |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
444 if pid is None: |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
445 pid = os.getpid() |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
446 if hostname is None: |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
447 hostname = lock._getlockprefix() |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
448 return GeneratingBundle( |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
449 self.bundle_type, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
450 self.revs, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
451 self.tip_rev, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
452 self.tip_node, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
453 hostname, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
454 pid, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
455 file_path, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
456 ) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
457 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
458 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
459 class GeneratingBundle(BundleBase): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
460 """A bundle being generated |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
461 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
462 extra attributes compared to BundleBase: |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
463 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
464 :hostname: the hostname of the machine generating the bundle |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
465 :pid: the pid of the process generating the bundle |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
466 :filepath: the target filename of the bundle |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
467 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
468 These attributes exist to help detect stalled generation processes. |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
469 """ |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
470 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
471 ready = False |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
472 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
473 def __init__( |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
474 self, bundle_type, revs, tip_rev, tip_node, hostname, pid, filepath |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
475 ): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
476 self.hostname = hostname |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
477 self.pid = pid |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
478 self.filepath = filepath |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
479 super(GeneratingBundle, self).__init__( |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
480 bundle_type, revs, tip_rev, tip_node |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
481 ) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
482 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
483 @classmethod |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
484 def from_line(cls, line): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
485 """create an object by deserializing a line from AUTO_GEN_FILE""" |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
486 assert line.startswith(b'PENDING-v1 ') |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
487 ( |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
488 __, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
489 bundle_type, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
490 revs, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
491 tip_rev, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
492 tip_node, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
493 hostname, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
494 pid, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
495 filepath, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
496 ) = line.split() |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
497 hostname = util.urlreq.unquote(hostname) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
498 filepath = util.urlreq.unquote(filepath) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
499 revs = int(revs) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
500 tip_rev = int(tip_rev) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
501 pid = int(pid) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
502 return cls( |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
503 bundle_type, revs, tip_rev, tip_node, hostname, pid, filepath |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
504 ) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
505 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
506 def to_line(self): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
507 """serialize the object to include as a line in AUTO_GEN_FILE""" |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
508 templ = b"PENDING-v1 %s %d %d %s %s %d %s" |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
509 data = ( |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
510 self.bundle_type, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
511 self.revs, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
512 self.tip_rev, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
513 self.tip_node, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
514 util.urlreq.quote(self.hostname), |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
515 self.pid, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
516 util.urlreq.quote(self.filepath), |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
517 ) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
518 return templ % data |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
519 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
520 def __eq__(self, other): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
521 if not super(GeneratingBundle, self).__eq__(other): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
522 return False |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
523 left = (self.hostname, self.pid, self.filepath) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
524 right = (other.hostname, other.pid, other.filepath) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
525 return left == right |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
526 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
527 def uploaded(self, url, basename): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
528 """return a GeneratedBundle from this object""" |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
529 return GeneratedBundle( |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
530 self.bundle_type, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
531 self.revs, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
532 self.tip_rev, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
533 self.tip_node, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
534 url, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
535 basename, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
536 ) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
537 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
538 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
539 class GeneratedBundle(BundleBase): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
540 """A bundle that is done being generated and can be served |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
541 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
542 extra attributes compared to BundleBase: |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
543 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
544 :file_url: the url where the bundle is available. |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
545 :basename: the "basename" used to upload (useful for deletion) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
546 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
547 These attributes exist to generate a bundle manifest |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
548 (.hg/pullbundles.manifest) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
549 """ |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
550 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
551 ready = True |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
552 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
553 def __init__( |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
554 self, bundle_type, revs, tip_rev, tip_node, file_url, basename |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
555 ): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
556 self.file_url = file_url |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
557 self.basename = basename |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
558 super(GeneratedBundle, self).__init__( |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
559 bundle_type, revs, tip_rev, tip_node |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
560 ) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
561 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
562 @classmethod |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
563 def from_line(cls, line): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
564 """create an object by deserializing a line from AUTO_GEN_FILE""" |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
565 assert line.startswith(b'DONE-v1 ') |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
566 ( |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
567 __, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
568 bundle_type, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
569 revs, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
570 tip_rev, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
571 tip_node, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
572 file_url, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
573 basename, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
574 ) = line.split() |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
575 revs = int(revs) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
576 tip_rev = int(tip_rev) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
577 file_url = util.urlreq.unquote(file_url) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
578 return cls(bundle_type, revs, tip_rev, tip_node, file_url, basename) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
579 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
580 def to_line(self): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
581 """serialize the object to include as a line in AUTO_GEN_FILE""" |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
582 templ = b"DONE-v1 %s %d %d %s %s %s" |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
583 data = ( |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
584 self.bundle_type, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
585 self.revs, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
586 self.tip_rev, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
587 self.tip_node, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
588 util.urlreq.quote(self.file_url), |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
589 self.basename, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
590 ) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
591 return templ % data |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
592 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
593 def manifest_line(self): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
594 """serialize the object to include as a line in pullbundles.manifest""" |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
595 templ = b"%s BUNDLESPEC=%s REQUIRESNI=true" |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
596 return templ % (self.file_url, self.bundle_type) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
597 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
598 def __eq__(self, other): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
599 if not super(GeneratedBundle, self).__eq__(other): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
600 return False |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
601 return self.file_url == other.file_url |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
602 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
603 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
604 def parse_auto_gen(content): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
605 """parse the AUTO_GEN_FILE to return a list of Bundle object""" |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
606 bundles = [] |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
607 for line in content.splitlines(): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
608 if line.startswith(b'PENDING-v1 '): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
609 bundles.append(GeneratingBundle.from_line(line)) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
610 elif line.startswith(b'DONE-v1 '): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
611 bundles.append(GeneratedBundle.from_line(line)) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
612 return bundles |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
613 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
614 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
615 def dumps_auto_gen(bundles): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
616 """serialize a list of Bundle as a AUTO_GEN_FILE content""" |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
617 lines = [] |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
618 for b in bundles: |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
619 lines.append(b"%s\n" % b.to_line()) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
620 lines.sort() |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
621 return b"".join(lines) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
622 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
623 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
624 def read_auto_gen(repo): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
625 """read the AUTO_GEN_FILE for the <repo> a list of Bundle object""" |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
626 data = repo.vfs.tryread(AUTO_GEN_FILE) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
627 if not data: |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
628 return [] |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
629 return parse_auto_gen(data) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
630 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
631 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
632 def write_auto_gen(repo, bundles): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
633 """write a list of Bundle objects into the repo's AUTO_GEN_FILE""" |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
634 assert repo._cb_lock_ref is not None |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
635 data = dumps_auto_gen(bundles) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
636 with repo.vfs(AUTO_GEN_FILE, mode=b'wb', atomictemp=True) as f: |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
637 f.write(data) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
638 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
639 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
640 def generate_manifest(bundles): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
641 """write a list of Bundle objects into the repo's AUTO_GEN_FILE""" |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
642 bundles = list(bundles) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
643 bundles.sort(key=lambda b: b.bundle_type) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
644 lines = [] |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
645 for b in bundles: |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
646 lines.append(b"%s\n" % b.manifest_line()) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
647 return b"".join(lines) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
648 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
649 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
650 def update_ondisk_manifest(repo): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
651 """update the clonebundle manifest with latest url""" |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
652 with repo.clonebundles_lock(): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
653 bundles = read_auto_gen(repo) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
654 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
655 per_types = {} |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
656 for b in bundles: |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
657 if not (b.ready and b.valid_for(repo)): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
658 continue |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
659 current = per_types.get(b.bundle_type) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
660 if current is not None and current.revs >= b.revs: |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
661 continue |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
662 per_types[b.bundle_type] = b |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
663 manifest = generate_manifest(per_types.values()) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
664 with repo.vfs( |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
665 bundlecaches.CB_MANIFEST_FILE, mode=b"wb", atomictemp=True |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
666 ) as f: |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
667 f.write(manifest) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
668 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
669 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
670 def update_bundle_list(repo, new_bundles=(), del_bundles=()): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
671 """modify the repo's AUTO_GEN_FILE |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
672 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
673 This method also regenerates the clone bundle manifest when needed""" |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
674 with repo.clonebundles_lock(): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
675 bundles = read_auto_gen(repo) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
676 if del_bundles: |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
677 bundles = [b for b in bundles if b not in del_bundles] |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
678 new_bundles = [b for b in new_bundles if b not in bundles] |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
679 bundles.extend(new_bundles) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
680 write_auto_gen(repo, bundles) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
681 all_changed = [] |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
682 all_changed.extend(new_bundles) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
683 all_changed.extend(del_bundles) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
684 if any(b.ready for b in all_changed): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
685 update_ondisk_manifest(repo) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
686 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
687 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
688 def cleanup_tmp_bundle(repo, target): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
689 """remove a GeneratingBundle file and entry""" |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
690 assert not target.ready |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
691 with repo.clonebundles_lock(): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
692 repo.vfs.tryunlink(target.filepath) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
693 update_bundle_list(repo, del_bundles=[target]) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
694 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
695 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
696 def finalize_one_bundle(repo, target): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
697 """upload a generated bundle and advertise it in the clonebundles.manifest""" |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
698 with repo.clonebundles_lock(): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
699 bundles = read_auto_gen(repo) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
700 if target in bundles and target.valid_for(repo): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
701 result = upload_bundle(repo, target) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
702 update_bundle_list(repo, new_bundles=[result]) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
703 cleanup_tmp_bundle(repo, target) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
704 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
705 |
50431
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
706 def find_outdated_bundles(repo, bundles): |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
707 """finds outdated bundles""" |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
708 olds = [] |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
709 per_types = {} |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
710 for b in bundles: |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
711 if not b.valid_for(repo): |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
712 olds.append(b) |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
713 continue |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
714 l = per_types.setdefault(b.bundle_type, []) |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
715 l.append(b) |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
716 for key in sorted(per_types): |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
717 all = per_types[key] |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
718 if len(all) > 1: |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
719 all.sort(key=lambda b: b.revs, reverse=True) |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
720 olds.extend(all[1:]) |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
721 return olds |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
722 |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
723 |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
724 def collect_garbage(repo): |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
725 """finds outdated bundles and get them deleted""" |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
726 with repo.clonebundles_lock(): |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
727 bundles = read_auto_gen(repo) |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
728 olds = find_outdated_bundles(repo, bundles) |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
729 for o in olds: |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
730 delete_bundle(repo, o) |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
731 update_bundle_list(repo, del_bundles=olds) |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
732 |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
733 |
50430
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
734 def upload_bundle(repo, bundle): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
735 """upload the result of a GeneratingBundle and return a GeneratedBundle |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
736 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
737 The upload is done using the `clone-bundles.upload-command` |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
738 """ |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
739 cmd = repo.ui.config(b'clone-bundles', b'upload-command') |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
740 url = repo.ui.config(b'clone-bundles', b'url-template') |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
741 basename = repo.vfs.basename(bundle.filepath) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
742 filepath = procutil.shellquote(bundle.filepath) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
743 variables = { |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
744 b'HGCB_BUNDLE_PATH': filepath, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
745 b'HGCB_BUNDLE_BASENAME': basename, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
746 } |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
747 env = procutil.shellenviron(environ=variables) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
748 ret = repo.ui.system(cmd, environ=env) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
749 if ret: |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
750 raise error.Abort(b"command returned status %d: %s" % (ret, cmd)) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
751 url = ( |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
752 url.decode('utf8') |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
753 .format(basename=basename.decode('utf8')) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
754 .encode('utf8') |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
755 ) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
756 return bundle.uploaded(url, basename) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
757 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
758 |
50431
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
759 def delete_bundle(repo, bundle): |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
760 """delete a bundle from storage""" |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
761 assert bundle.ready |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
762 msg = b'clone-bundles: deleting bundle %s\n' |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
763 msg %= bundle.basename |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
764 if repo.ui.configbool(b'devel', b'debug.clonebundles'): |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
765 repo.ui.write(msg) |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
766 else: |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
767 repo.ui.debug(msg) |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
768 |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
769 cmd = repo.ui.config(b'clone-bundles', b'delete-command') |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
770 variables = { |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
771 b'HGCB_BUNDLE_URL': bundle.file_url, |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
772 b'HGCB_BASENAME': bundle.basename, |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
773 } |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
774 env = procutil.shellenviron(environ=variables) |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
775 ret = repo.ui.system(cmd, environ=env) |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
776 if ret: |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
777 raise error.Abort(b"command returned status %d: %s" % (ret, cmd)) |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
778 |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
779 |
50430
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
780 def auto_bundle_needed_actions(repo, bundles, op_id): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
781 """find the list of bundles that need action |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
782 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
783 returns a list of RequestedBundle objects that need to be generated and |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
784 uploaded.""" |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
785 create_bundles = [] |
50431
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
786 delete_bundles = [] |
50430
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
787 repo = repo.filtered(b"immutable") |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
788 targets = repo.ui.configlist(b'clone-bundles', b'auto-generate.formats') |
50432
5b70b9f5a2f9
clone-bundles: only regenerate the clone bundle when cached ration is low
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50431
diff
changeset
|
789 ratio = float( |
5b70b9f5a2f9
clone-bundles: only regenerate the clone bundle when cached ration is low
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50431
diff
changeset
|
790 repo.ui.config(b'clone-bundles', b'trigger.below-bundled-ratio') |
5b70b9f5a2f9
clone-bundles: only regenerate the clone bundle when cached ration is low
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50431
diff
changeset
|
791 ) |
50433
d611805e7374
clone-bundles: also control automation based on absolute number of revisions
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50432
diff
changeset
|
792 abs_revs = repo.ui.configint(b'clone-bundles', b'trigger.revs') |
50430
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
793 revs = len(repo.changelog) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
794 generic_data = { |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
795 'revs': revs, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
796 'head_revs': repo.changelog.headrevs(), |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
797 'tip_rev': repo.changelog.tiprev(), |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
798 'tip_node': node.hex(repo.changelog.tip()), |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
799 'op_id': op_id, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
800 } |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
801 for t in targets: |
50433
d611805e7374
clone-bundles: also control automation based on absolute number of revisions
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50432
diff
changeset
|
802 if new_bundle_needed(repo, bundles, ratio, abs_revs, t, revs): |
50432
5b70b9f5a2f9
clone-bundles: only regenerate the clone bundle when cached ration is low
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50431
diff
changeset
|
803 data = generic_data.copy() |
5b70b9f5a2f9
clone-bundles: only regenerate the clone bundle when cached ration is low
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50431
diff
changeset
|
804 data['bundle_type'] = t |
5b70b9f5a2f9
clone-bundles: only regenerate the clone bundle when cached ration is low
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50431
diff
changeset
|
805 b = RequestedBundle(**data) |
5b70b9f5a2f9
clone-bundles: only regenerate the clone bundle when cached ration is low
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50431
diff
changeset
|
806 create_bundles.append(b) |
50431
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
807 delete_bundles.extend(find_outdated_bundles(repo, bundles)) |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
808 return create_bundles, delete_bundles |
50430
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
809 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
810 |
50433
d611805e7374
clone-bundles: also control automation based on absolute number of revisions
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50432
diff
changeset
|
811 def new_bundle_needed(repo, bundles, ratio, abs_revs, bundle_type, revs): |
50432
5b70b9f5a2f9
clone-bundles: only regenerate the clone bundle when cached ration is low
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50431
diff
changeset
|
812 """consider the current cached content and trigger new bundles if needed""" |
50433
d611805e7374
clone-bundles: also control automation based on absolute number of revisions
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50432
diff
changeset
|
813 threshold = max((revs * ratio), (revs - abs_revs)) |
50432
5b70b9f5a2f9
clone-bundles: only regenerate the clone bundle when cached ration is low
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50431
diff
changeset
|
814 for b in bundles: |
5b70b9f5a2f9
clone-bundles: only regenerate the clone bundle when cached ration is low
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50431
diff
changeset
|
815 if not b.valid_for(repo) or b.bundle_type != bundle_type: |
5b70b9f5a2f9
clone-bundles: only regenerate the clone bundle when cached ration is low
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50431
diff
changeset
|
816 continue |
5b70b9f5a2f9
clone-bundles: only regenerate the clone bundle when cached ration is low
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50431
diff
changeset
|
817 if b.revs > threshold: |
5b70b9f5a2f9
clone-bundles: only regenerate the clone bundle when cached ration is low
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50431
diff
changeset
|
818 return False |
5b70b9f5a2f9
clone-bundles: only regenerate the clone bundle when cached ration is low
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50431
diff
changeset
|
819 return True |
5b70b9f5a2f9
clone-bundles: only regenerate the clone bundle when cached ration is low
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50431
diff
changeset
|
820 |
5b70b9f5a2f9
clone-bundles: only regenerate the clone bundle when cached ration is low
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50431
diff
changeset
|
821 |
50430
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
822 def start_one_bundle(repo, bundle): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
823 """start the generation of a single bundle file |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
824 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
825 the `bundle` argument should be a RequestedBundle object. |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
826 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
827 This data is passed to the `debugmakeclonebundles` "as is". |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
828 """ |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
829 data = util.pickle.dumps(bundle) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
830 cmd = [procutil.hgexecutable(), b'--cwd', repo.path, INTERNAL_CMD] |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
831 env = procutil.shellenviron() |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
832 msg = b'clone-bundles: starting bundle generation: %s\n' |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
833 stdout = None |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
834 stderr = None |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
835 waits = [] |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
836 record_wait = None |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
837 if repo.ui.configbool(b'devel', b'debug.clonebundles'): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
838 stdout = procutil.stdout |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
839 stderr = procutil.stderr |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
840 repo.ui.write(msg % bundle.bundle_type) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
841 record_wait = waits.append |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
842 else: |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
843 repo.ui.debug(msg % bundle.bundle_type) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
844 bg = procutil.runbgcommand |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
845 bg( |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
846 cmd, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
847 env, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
848 stdin_bytes=data, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
849 stdout=stdout, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
850 stderr=stderr, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
851 record_wait=record_wait, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
852 ) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
853 for f in waits: |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
854 f() |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
855 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
856 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
857 INTERNAL_CMD = b'debug::internal-make-clone-bundles' |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
858 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
859 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
860 @command(INTERNAL_CMD, [], b'') |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
861 def debugmakeclonebundles(ui, repo): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
862 """Internal command to auto-generate debug bundles""" |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
863 requested_bundle = util.pickle.load(procutil.stdin) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
864 procutil.stdin.close() |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
865 |
50431
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
866 collect_garbage(repo) |
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
867 |
50430
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
868 fname = requested_bundle.suggested_filename |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
869 fpath = repo.vfs.makedirs(b'tmp-bundles') |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
870 fpath = repo.vfs.join(b'tmp-bundles', fname) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
871 bundle = requested_bundle.generating(fpath) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
872 update_bundle_list(repo, new_bundles=[bundle]) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
873 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
874 requested_bundle.generate_bundle(repo, fpath) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
875 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
876 repo.invalidate() |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
877 finalize_one_bundle(repo, bundle) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
878 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
879 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
880 def make_auto_bundler(source_repo): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
881 reporef = weakref.ref(source_repo) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
882 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
883 def autobundle(tr): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
884 repo = reporef() |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
885 assert repo is not None |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
886 bundles = read_auto_gen(repo) |
50431
971dc2369b04
clone-bundles: garbage collect older bundle when generating new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50430
diff
changeset
|
887 new, __ = auto_bundle_needed_actions(repo, bundles, b"%d_txn" % id(tr)) |
50430
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
888 for data in new: |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
889 start_one_bundle(repo, data) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
890 return None |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
891 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
892 return autobundle |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
893 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
894 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
895 def reposetup(ui, repo): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
896 """install the two pieces needed for automatic clonebundle generation |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
897 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
898 - add a "post-close" hook that fires bundling when needed |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
899 - introduce a clone-bundle lock to let multiple processes meddle with the |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
900 state files. |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
901 """ |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
902 if not repo.local(): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
903 return |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
904 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
905 class autobundlesrepo(repo.__class__): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
906 def transaction(self, *args, **kwargs): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
907 tr = super(autobundlesrepo, self).transaction(*args, **kwargs) |
50434
65fb4cedd5ea
clone-bundles: add a configuration to control auto-generation on changes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50433
diff
changeset
|
908 enabled = repo.ui.configbool( |
65fb4cedd5ea
clone-bundles: add a configuration to control auto-generation on changes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50433
diff
changeset
|
909 b'clone-bundles', |
65fb4cedd5ea
clone-bundles: add a configuration to control auto-generation on changes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50433
diff
changeset
|
910 b'auto-generate.on-change', |
65fb4cedd5ea
clone-bundles: add a configuration to control auto-generation on changes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50433
diff
changeset
|
911 ) |
50430
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
912 targets = repo.ui.configlist( |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
913 b'clone-bundles', b'auto-generate.formats' |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
914 ) |
50434
65fb4cedd5ea
clone-bundles: add a configuration to control auto-generation on changes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50433
diff
changeset
|
915 if enabled and targets: |
50430
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
916 tr.addpostclose(CAT_POSTCLOSE, make_auto_bundler(self)) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
917 return tr |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
918 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
919 @localrepo.unfilteredmethod |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
920 def clonebundles_lock(self, wait=True): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
921 '''Lock the repository file related to clone bundles''' |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
922 if not util.safehasattr(self, '_cb_lock_ref'): |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
923 self._cb_lock_ref = None |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
924 l = self._currentlock(self._cb_lock_ref) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
925 if l is not None: |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
926 l.lock() |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
927 return l |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
928 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
929 l = self._lock( |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
930 vfs=self.vfs, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
931 lockname=b"clonebundleslock", |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
932 wait=wait, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
933 releasefn=None, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
934 acquirefn=None, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
935 desc=_(b'repository %s') % self.origroot, |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
936 ) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
937 self._cb_lock_ref = weakref.ref(l) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
938 return l |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
939 |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
940 repo._wlockfreeprefix.add(AUTO_GEN_FILE) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
941 repo._wlockfreeprefix.add(bundlecaches.CB_MANIFEST_FILE) |
5ae30ff79c76
clone-bundles: add a basic first version of automatic bundle generation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
48875
diff
changeset
|
942 repo.__class__ = autobundlesrepo |
50435
23db5f15cc0c
clone-bundles: introduce a command to refresh bundle
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50434
diff
changeset
|
943 |
23db5f15cc0c
clone-bundles: introduce a command to refresh bundle
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50434
diff
changeset
|
944 |
50438
3973b1dc3ee3
clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50437
diff
changeset
|
945 @command( |
3973b1dc3ee3
clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50437
diff
changeset
|
946 b'admin::clone-bundles-refresh', |
3973b1dc3ee3
clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50437
diff
changeset
|
947 [ |
3973b1dc3ee3
clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50437
diff
changeset
|
948 ( |
3973b1dc3ee3
clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50437
diff
changeset
|
949 b'', |
3973b1dc3ee3
clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50437
diff
changeset
|
950 b'background', |
3973b1dc3ee3
clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50437
diff
changeset
|
951 False, |
3973b1dc3ee3
clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50437
diff
changeset
|
952 _(b'start bundle generation in the background'), |
3973b1dc3ee3
clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50437
diff
changeset
|
953 ), |
3973b1dc3ee3
clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50437
diff
changeset
|
954 ], |
3973b1dc3ee3
clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50437
diff
changeset
|
955 b'', |
3973b1dc3ee3
clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50437
diff
changeset
|
956 ) |
3973b1dc3ee3
clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50437
diff
changeset
|
957 def cmd_admin_clone_bundles_refresh( |
3973b1dc3ee3
clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50437
diff
changeset
|
958 ui, |
3973b1dc3ee3
clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50437
diff
changeset
|
959 repo: localrepo.localrepository, |
3973b1dc3ee3
clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50437
diff
changeset
|
960 background=False, |
3973b1dc3ee3
clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50437
diff
changeset
|
961 ): |
50435
23db5f15cc0c
clone-bundles: introduce a command to refresh bundle
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50434
diff
changeset
|
962 """generate clone bundles according to the configuration |
23db5f15cc0c
clone-bundles: introduce a command to refresh bundle
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50434
diff
changeset
|
963 |
23db5f15cc0c
clone-bundles: introduce a command to refresh bundle
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50434
diff
changeset
|
964 This runs the logic for automatic generation, removing outdated bundles and |
23db5f15cc0c
clone-bundles: introduce a command to refresh bundle
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50434
diff
changeset
|
965 generating new ones if necessary. See :hg:`help -e clone-bundles` for |
23db5f15cc0c
clone-bundles: introduce a command to refresh bundle
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50434
diff
changeset
|
966 details about how to configure this feature. |
23db5f15cc0c
clone-bundles: introduce a command to refresh bundle
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50434
diff
changeset
|
967 """ |
23db5f15cc0c
clone-bundles: introduce a command to refresh bundle
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50434
diff
changeset
|
968 debug = repo.ui.configbool(b'devel', b'debug.clonebundles') |
23db5f15cc0c
clone-bundles: introduce a command to refresh bundle
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50434
diff
changeset
|
969 bundles = read_auto_gen(repo) |
23db5f15cc0c
clone-bundles: introduce a command to refresh bundle
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50434
diff
changeset
|
970 op_id = b"%d_acbr" % os.getpid() |
23db5f15cc0c
clone-bundles: introduce a command to refresh bundle
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50434
diff
changeset
|
971 create, delete = auto_bundle_needed_actions(repo, bundles, op_id) |
23db5f15cc0c
clone-bundles: introduce a command to refresh bundle
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50434
diff
changeset
|
972 |
50438
3973b1dc3ee3
clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50437
diff
changeset
|
973 # if some bundles are scheduled for creation in the background, they will |
3973b1dc3ee3
clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50437
diff
changeset
|
974 # deal with garbage collection too, so no need to synchroniously do it. |
3973b1dc3ee3
clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50437
diff
changeset
|
975 # |
3973b1dc3ee3
clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50437
diff
changeset
|
976 # However if no bundles are scheduled for creation, we need to explicitly do |
3973b1dc3ee3
clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50437
diff
changeset
|
977 # it here. |
3973b1dc3ee3
clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50437
diff
changeset
|
978 if not (background and create): |
3973b1dc3ee3
clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50437
diff
changeset
|
979 # we clean up outdated bundles before generating new ones to keep the |
3973b1dc3ee3
clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50437
diff
changeset
|
980 # last two versions of the bundle around for a while and avoid having to |
3973b1dc3ee3
clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50437
diff
changeset
|
981 # deal with clients that just got served a manifest. |
3973b1dc3ee3
clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50437
diff
changeset
|
982 for o in delete: |
3973b1dc3ee3
clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50437
diff
changeset
|
983 delete_bundle(repo, o) |
3973b1dc3ee3
clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50437
diff
changeset
|
984 update_bundle_list(repo, del_bundles=delete) |
50435
23db5f15cc0c
clone-bundles: introduce a command to refresh bundle
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50434
diff
changeset
|
985 |
23db5f15cc0c
clone-bundles: introduce a command to refresh bundle
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50434
diff
changeset
|
986 if create: |
23db5f15cc0c
clone-bundles: introduce a command to refresh bundle
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50434
diff
changeset
|
987 fpath = repo.vfs.makedirs(b'tmp-bundles') |
50438
3973b1dc3ee3
clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50437
diff
changeset
|
988 |
3973b1dc3ee3
clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50437
diff
changeset
|
989 if background: |
3973b1dc3ee3
clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50437
diff
changeset
|
990 for requested_bundle in create: |
3973b1dc3ee3
clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50437
diff
changeset
|
991 start_one_bundle(repo, requested_bundle) |
3973b1dc3ee3
clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50437
diff
changeset
|
992 else: |
3973b1dc3ee3
clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50437
diff
changeset
|
993 for requested_bundle in create: |
3973b1dc3ee3
clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50437
diff
changeset
|
994 if debug: |
3973b1dc3ee3
clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50437
diff
changeset
|
995 msg = b'clone-bundles: starting bundle generation: %s\n' |
3973b1dc3ee3
clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50437
diff
changeset
|
996 repo.ui.write(msg % requested_bundle.bundle_type) |
3973b1dc3ee3
clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50437
diff
changeset
|
997 fname = requested_bundle.suggested_filename |
3973b1dc3ee3
clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50437
diff
changeset
|
998 fpath = repo.vfs.join(b'tmp-bundles', fname) |
3973b1dc3ee3
clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50437
diff
changeset
|
999 generating_bundle = requested_bundle.generating(fpath) |
3973b1dc3ee3
clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50437
diff
changeset
|
1000 update_bundle_list(repo, new_bundles=[generating_bundle]) |
3973b1dc3ee3
clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50437
diff
changeset
|
1001 requested_bundle.generate_bundle(repo, fpath) |
3973b1dc3ee3
clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50437
diff
changeset
|
1002 result = upload_bundle(repo, generating_bundle) |
3973b1dc3ee3
clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50437
diff
changeset
|
1003 update_bundle_list(repo, new_bundles=[result]) |
3973b1dc3ee3
clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50437
diff
changeset
|
1004 update_ondisk_manifest(repo) |
3973b1dc3ee3
clone-bundles: add an option to generate bundles in the background
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50437
diff
changeset
|
1005 cleanup_tmp_bundle(repo, generating_bundle) |
50437
10318b88b1d6
clone-bundles: add a command to clear all bundles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50435
diff
changeset
|
1006 |
10318b88b1d6
clone-bundles: add a command to clear all bundles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50435
diff
changeset
|
1007 |
10318b88b1d6
clone-bundles: add a command to clear all bundles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50435
diff
changeset
|
1008 @command(b'admin::clone-bundles-clear', [], b'') |
10318b88b1d6
clone-bundles: add a command to clear all bundles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50435
diff
changeset
|
1009 def cmd_admin_clone_bundles_clear(ui, repo: localrepo.localrepository): |
10318b88b1d6
clone-bundles: add a command to clear all bundles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50435
diff
changeset
|
1010 """remove existing clone bundle caches |
10318b88b1d6
clone-bundles: add a command to clear all bundles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50435
diff
changeset
|
1011 |
10318b88b1d6
clone-bundles: add a command to clear all bundles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50435
diff
changeset
|
1012 See `hg help admin::clone-bundles-refresh` for details on how to regenerate |
10318b88b1d6
clone-bundles: add a command to clear all bundles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50435
diff
changeset
|
1013 them. |
10318b88b1d6
clone-bundles: add a command to clear all bundles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50435
diff
changeset
|
1014 |
10318b88b1d6
clone-bundles: add a command to clear all bundles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50435
diff
changeset
|
1015 This command will only affect bundles currently available, it will not |
10318b88b1d6
clone-bundles: add a command to clear all bundles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50435
diff
changeset
|
1016 affect bundles being asynchronously generated. |
10318b88b1d6
clone-bundles: add a command to clear all bundles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50435
diff
changeset
|
1017 """ |
10318b88b1d6
clone-bundles: add a command to clear all bundles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50435
diff
changeset
|
1018 bundles = read_auto_gen(repo) |
10318b88b1d6
clone-bundles: add a command to clear all bundles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50435
diff
changeset
|
1019 delete = [b for b in bundles if b.ready] |
10318b88b1d6
clone-bundles: add a command to clear all bundles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50435
diff
changeset
|
1020 for o in delete: |
10318b88b1d6
clone-bundles: add a command to clear all bundles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50435
diff
changeset
|
1021 delete_bundle(repo, o) |
10318b88b1d6
clone-bundles: add a command to clear all bundles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
50435
diff
changeset
|
1022 update_bundle_list(repo, del_bundles=delete) |