Mercurial > hg
annotate hgext/clonebundles.py @ 47084:27f1191b1305
sidedata: replace sidedata upgrade mechanism with the new one
Note: this is split into a separate change (like some other patches in this
series) because it's not easy to have all patches work 100% and this seemed
easier for reviewers.
When cloning or upgrading a repo, we may need to compute (or remove) sidedata.
This is the same mechanism that is used in exchange, so we re-use the new
system to simplify the code and fix the remaining issues (correctly dropping
flags and handling partial removal, etc.).
This also highlighted an issue with `test-copies-in-changeset.t` that kept
sidedata categories that are not relevant anymore. They should probably be
dropped entirely, but that would be for another patch.
Differential Revision: https://phab.mercurial-scm.org/D10359
author | Raphaël Gomès <rgomes@octobus.net> |
---|---|
date | Mon, 19 Apr 2021 11:22:24 +0200 |
parents | 80f32ec8653a |
children | 6000f5b25c9b |
rev | line source |
---|---|
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
1 # This software may be used and distributed according to the terms of the |
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
2 # GNU General Public License version 2 or any later version. |
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
3 |
27738
a0e783d26e81
exchange: make clone bundles non-experimental and enabled by default
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27737
diff
changeset
|
4 """advertise pre-generated bundles to seed clones |
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
5 |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
6 "clonebundles" is a server-side extension used to advertise the existence |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
7 of pre-generated, externally hosted bundle files to clients that are |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
8 cloning so that cloning can be faster, more reliable, and require less |
37498
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
9 resources on the server. "pullbundles" is a related feature for sending |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
10 pre-generated bundle files to clients as part of pull operations. |
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
11 |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
12 Cloning can be a CPU and I/O intensive operation on servers. Traditionally, |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
13 the server, in response to a client's request to clone, dynamically generates |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
14 a bundle containing the entire repository content and sends it to the client. |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
15 There is no caching on the server and the server will have to redundantly |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
16 generate the same outgoing bundle in response to each clone request. For |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
17 servers with large repositories or with high clone volume, the load from |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
18 clones can make scaling the server challenging and costly. |
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
19 |
37498
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
20 This extension provides server operators the ability to offload |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
21 potentially expensive clone load to an external service. Pre-generated |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
22 bundles also allow using more CPU intensive compression, reducing the |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
23 effective bandwidth requirements. |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
24 |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
25 Here's how clone bundles work: |
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
26 |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
27 1. A server operator establishes a mechanism for making bundle files available |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
28 on a hosting service where Mercurial clients can fetch them. |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
29 2. A manifest file listing available bundle URLs and some optional metadata |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
30 is added to the Mercurial repository on the server. |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
31 3. A client initiates a clone against a clone bundles aware server. |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
32 4. The client sees the server is advertising clone bundles and fetches the |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
33 manifest listing available bundles. |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
34 5. The client filters and sorts the available bundles based on what it |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
35 supports and prefers. |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
36 6. The client downloads and applies an available bundle from the |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
37 server-specified URL. |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
38 7. The client reconnects to the original server and performs the equivalent |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
39 of :hg:`pull` to retrieve all repository data not in the bundle. (The |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
40 repository could have been updated between when the bundle was created |
37498
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
41 and when the client started the clone.) This may use "pullbundles". |
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
42 |
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
43 Instead of the server generating full repository bundles for every clone |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
44 request, it generates full bundles once and they are subsequently reused to |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
45 bootstrap new clones. The server may still transfer data at clone time. |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
46 However, this is only data that has been added/changed since the bundle was |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
47 created. For large, established repositories, this can reduce server load for |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
48 clones to less than 1% of original. |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
49 |
37498
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
50 Here's how pullbundles work: |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
51 |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
52 1. A manifest file listing available bundles and describing the revisions |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
53 is added to the Mercurial repository on the server. |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
54 2. A new-enough client informs the server that it supports partial pulls |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
55 and initiates a pull. |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
56 3. If the server has pull bundles enabled and sees the client advertising |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
57 partial pulls, it checks for a matching pull bundle in the manifest. |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
58 A bundle matches if the format is supported by the client, the client |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
59 has the required revisions already and needs something from the bundle. |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
60 4. If there is at least one matching bundle, the server sends it to the client. |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
61 5. The client applies the bundle and notices that the server reply was |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
62 incomplete. It initiates another pull. |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
63 |
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
64 To work, this extension requires the following of server operators: |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
65 |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
66 * Generating bundle files of repository content (typically periodically, |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
67 such as once per day). |
37498
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
68 * Clone bundles: A file server that clients have network access to and that |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
69 Python knows how to talk to through its normal URL handling facility |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
70 (typically an HTTP/HTTPS server). |
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
71 * A process for keeping the bundles manifest in sync with available bundle |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
72 files. |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
73 |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
74 Strictly speaking, using a static file hosting server isn't required: a server |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
75 operator could use a dynamic service for retrieving bundle data. However, |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
76 static file hosting services are simple and scalable and should be sufficient |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
77 for most needs. |
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
78 |
26884
762bf510b42c
clonebundles: fix typo s/comand/command/
Javi Merino <merino.jav@gmail.com>
parents:
26857
diff
changeset
|
79 Bundle files can be generated with the :hg:`bundle` command. Typically |
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
80 :hg:`bundle --all` is used to produce a bundle of the entire repository. |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
81 |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
82 :hg:`debugcreatestreamclonebundle` can be used to produce a special |
37498
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
83 *streaming clonebundle*. These are bundle files that are extremely efficient |
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
84 to produce and consume (read: fast). However, they are larger than |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
85 traditional bundle formats and require that clients support the exact set |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
86 of repository data store formats in use by the repository that created them. |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
87 Typically, a newer server can serve data that is compatible with older clients. |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
88 However, *streaming clone bundles* don't have this guarantee. **Server |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
89 operators need to be aware that newer versions of Mercurial may produce |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
90 streaming clone bundles incompatible with older Mercurial versions.** |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
91 |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
92 A server operator is responsible for creating a ``.hg/clonebundles.manifest`` |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
93 file containing the list of available bundle files suitable for seeding |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
94 clones. If this file does not exist, the repository will not advertise the |
37498
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
95 existence of clone bundles when clients connect. For pull bundles, |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
96 ``.hg/pullbundles.manifest`` is used. |
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
97 |
32454
702af1ad3b18
clonebundles: fix missing newline character
Matt Harbison <matt_harbison@yahoo.com>
parents:
31146
diff
changeset
|
98 The manifest file contains a newline (\\n) delimited list of entries. |
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
99 |
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
100 Each line in this file defines an available bundle. Lines have the format: |
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
101 |
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
102 <URL> [<key>=<value>[ <key>=<value>]] |
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
103 |
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
104 That is, a URL followed by an optional, space-delimited list of key=value |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
105 pairs describing additional properties of this bundle. Both keys and values |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
106 are URI encoded. |
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
107 |
37498
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
108 For pull bundles, the URL is a path under the ``.hg`` directory of the |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
109 repository. |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
110 |
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
111 Keys in UPPERCASE are reserved for use by Mercurial and are defined below. |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
112 All non-uppercase keys can be used by site installations. An example use |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
113 for custom properties is to use the *datacenter* attribute to define which |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
114 data center a file is hosted in. Clients could then prefer a server in the |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
115 data center closest to them. |
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
116 |
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
117 The following reserved keys are currently defined: |
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
118 |
26644
74de1c59f71c
clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26623
diff
changeset
|
119 BUNDLESPEC |
74de1c59f71c
clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26623
diff
changeset
|
120 A "bundle specification" string that describes the type of the bundle. |
74de1c59f71c
clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26623
diff
changeset
|
121 |
74de1c59f71c
clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26623
diff
changeset
|
122 These are string values that are accepted by the "--type" argument of |
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
123 :hg:`bundle`. |
26644
74de1c59f71c
clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26623
diff
changeset
|
124 |
74de1c59f71c
clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26623
diff
changeset
|
125 The values are parsed in strict mode, which means they must be of the |
74de1c59f71c
clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26623
diff
changeset
|
126 "<compression>-<type>" form. See |
74de1c59f71c
clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26623
diff
changeset
|
127 mercurial.exchange.parsebundlespec() for more details. |
74de1c59f71c
clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26623
diff
changeset
|
128 |
27886
0288e63ea3be
clonebundles: improve BUNDLESPEC documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27738
diff
changeset
|
129 :hg:`debugbundle --spec` can be used to print the bundle specification |
0288e63ea3be
clonebundles: improve BUNDLESPEC documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27738
diff
changeset
|
130 string for a bundle file. The output of this command can be used verbatim |
0288e63ea3be
clonebundles: improve BUNDLESPEC documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27738
diff
changeset
|
131 for the value of ``BUNDLESPEC`` (it is already escaped). |
0288e63ea3be
clonebundles: improve BUNDLESPEC documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27738
diff
changeset
|
132 |
26644
74de1c59f71c
clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26623
diff
changeset
|
133 Clients will automatically filter out specifications that are unknown or |
74de1c59f71c
clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26623
diff
changeset
|
134 unsupported so they won't attempt to download something that likely won't |
74de1c59f71c
clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26623
diff
changeset
|
135 apply. |
74de1c59f71c
clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26623
diff
changeset
|
136 |
74de1c59f71c
clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26623
diff
changeset
|
137 The actual value doesn't impact client behavior beyond filtering: |
74de1c59f71c
clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26623
diff
changeset
|
138 clients will still sniff the bundle type from the header of downloaded |
74de1c59f71c
clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26623
diff
changeset
|
139 files. |
26645
2faa7671a4b3
clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26644
diff
changeset
|
140 |
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
141 **Use of this key is highly recommended**, as it allows clients to |
27886
0288e63ea3be
clonebundles: improve BUNDLESPEC documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27738
diff
changeset
|
142 easily skip unsupported bundles. If this key is not defined, an old |
0288e63ea3be
clonebundles: improve BUNDLESPEC documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27738
diff
changeset
|
143 client may attempt to apply a bundle that it is incapable of reading. |
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
144 |
26645
2faa7671a4b3
clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26644
diff
changeset
|
145 REQUIRESNI |
2faa7671a4b3
clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26644
diff
changeset
|
146 Whether Server Name Indication (SNI) is required to connect to the URL. |
2faa7671a4b3
clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26644
diff
changeset
|
147 SNI allows servers to use multiple certificates on the same IP. It is |
2faa7671a4b3
clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26644
diff
changeset
|
148 somewhat common in CDNs and other hosting providers. Older Python |
2faa7671a4b3
clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26644
diff
changeset
|
149 versions do not support SNI. Defining this attribute enables clients |
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
150 with older Python versions to filter this entry without experiencing |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
151 an opaque SSL failure at connection time. |
26645
2faa7671a4b3
clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26644
diff
changeset
|
152 |
2faa7671a4b3
clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26644
diff
changeset
|
153 If this is defined, it is important to advertise a non-SNI fallback |
2faa7671a4b3
clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26644
diff
changeset
|
154 URL or clients running old Python releases may not be able to clone |
2faa7671a4b3
clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26644
diff
changeset
|
155 with the clonebundles facility. |
2faa7671a4b3
clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26644
diff
changeset
|
156 |
2faa7671a4b3
clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26644
diff
changeset
|
157 Value should be "true". |
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
158 |
45252
20d110e6eea6
clonebundles: document REQUIREDRAM key
Gregory Szorc <gregory.szorc@gmail.com>
parents:
43077
diff
changeset
|
159 REQUIREDRAM |
20d110e6eea6
clonebundles: document REQUIREDRAM key
Gregory Szorc <gregory.szorc@gmail.com>
parents:
43077
diff
changeset
|
160 Value specifies expected memory requirements to decode the payload. |
20d110e6eea6
clonebundles: document REQUIREDRAM key
Gregory Szorc <gregory.szorc@gmail.com>
parents:
43077
diff
changeset
|
161 Values can have suffixes for common bytes sizes. e.g. "64MB". |
20d110e6eea6
clonebundles: document REQUIREDRAM key
Gregory Szorc <gregory.szorc@gmail.com>
parents:
43077
diff
changeset
|
162 |
20d110e6eea6
clonebundles: document REQUIREDRAM key
Gregory Szorc <gregory.szorc@gmail.com>
parents:
43077
diff
changeset
|
163 This key is often used with zstd-compressed bundles using a high |
20d110e6eea6
clonebundles: document REQUIREDRAM key
Gregory Szorc <gregory.szorc@gmail.com>
parents:
43077
diff
changeset
|
164 compression level / window size, which can require 100+ MB of memory |
20d110e6eea6
clonebundles: document REQUIREDRAM key
Gregory Szorc <gregory.szorc@gmail.com>
parents:
43077
diff
changeset
|
165 to decode. |
20d110e6eea6
clonebundles: document REQUIREDRAM key
Gregory Szorc <gregory.szorc@gmail.com>
parents:
43077
diff
changeset
|
166 |
37498
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
167 heads |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
168 Used for pull bundles. This contains the ``;`` separated changeset |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
169 hashes of the heads of the bundle content. |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
170 |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
171 bases |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
172 Used for pull bundles. This contains the ``;`` separated changeset |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
173 hashes of the roots of the bundle content. This can be skipped if |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
174 the bundle was created without ``--base``. |
aacfca6f9767
wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents:
32773
diff
changeset
|
175 |
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
176 Manifests can contain multiple entries. Assuming metadata is defined, clients |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
177 will filter entries from the manifest that they don't support. The remaining |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
178 entries are optionally sorted by client preferences |
32773
d25802b0eef5
clonebundles: reference correct config option
Gregory Szorc <gregory.szorc@gmail.com>
parents:
32454
diff
changeset
|
179 (``ui.clonebundleprefers`` config option). The client then attempts |
26762
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
180 to fetch the bundle at the first URL in the remaining list. |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
181 |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
182 **Errors when downloading a bundle will fail the entire clone operation: |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
183 clients do not automatically fall back to a traditional clone.** The reason |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
184 for this is that if a server is using clone bundles, it is probably doing so |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
185 because the feature is necessary to help it scale. In other words, there |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
186 is an assumption that clone load will be offloaded to another service and |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
187 that the Mercurial server isn't responsible for serving this clone load. |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
188 If that other service experiences issues and clients start mass falling back to |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
189 the original Mercurial server, the added clone load could overwhelm the server |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
190 due to unexpected load and effectively take it offline. Not having clients |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
191 automatically fall back to cloning from the original server mitigates this |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
192 scenario. |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
193 |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
194 Because there is no automatic Mercurial server fallback on failure of the |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
195 bundle hosting service, it is important for server operators to view the bundle |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
196 hosting service as an extension of the Mercurial server in terms of |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
197 availability and service level agreements: if the bundle hosting service goes |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
198 down, so does the ability for clients to clone. Note: clients will see a |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
199 message informing them how to bypass the clone bundles facility when a failure |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
200 occurs. So server operators should prepare for some people to follow these |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
201 instructions when a failure occurs, thus driving more load to the original |
26f622859288
clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26691
diff
changeset
|
202 Mercurial server when the bundle hosting service fails. |
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
203 """ |
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
204 |
28095
7fa139eaebb4
clonebundles: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27886
diff
changeset
|
205 from __future__ import absolute_import |
7fa139eaebb4
clonebundles: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27886
diff
changeset
|
206 |
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
207 from mercurial import ( |
45785
80f32ec8653a
clonebundle: move the manifest filename to a constant
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
45252
diff
changeset
|
208 bundlecaches, |
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
209 extensions, |
37785
b4d85bc122bd
wireproto: rename wireproto to wireprotov1server (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37498
diff
changeset
|
210 wireprotov1server, |
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
211 ) |
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
212 |
43077
687b865b95ad
formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents:
43076
diff
changeset
|
213 testedwith = b'ships-with-hg-core' |
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
214 |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
37785
diff
changeset
|
215 |
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
216 def capabilities(orig, repo, proto): |
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
217 caps = orig(repo, proto) |
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
218 |
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
219 # Only advertise if a manifest exists. This does add some I/O to requests. |
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
220 # But this should be cheaper than a wasted network round trip due to |
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
221 # missing file. |
45785
80f32ec8653a
clonebundle: move the manifest filename to a constant
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
45252
diff
changeset
|
222 if repo.vfs.exists(bundlecaches.CB_MANIFEST_FILE): |
43077
687b865b95ad
formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents:
43076
diff
changeset
|
223 caps.append(b'clonebundles') |
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
224 |
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
225 return caps |
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
226 |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
37785
diff
changeset
|
227 |
26623
5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
228 def extsetup(ui): |
43077
687b865b95ad
formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents:
43076
diff
changeset
|
229 extensions.wrapfunction(wireprotov1server, b'_capabilities', capabilities) |