annotate hgext/clonebundles.py @ 40910:4bb6195755b1

test: enable sparse-revlog for test-ssh-proto-unbundle.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5337
author Boris Feld <boris.feld@octobus.net>
date Mon, 12 Nov 2018 01:17:05 +0100
parents b4d85bc122bd
children 2372284d9457
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
26623
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
1 # This software may be used and distributed according to the terms of the
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
2 # GNU General Public License version 2 or any later version.
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
3
27738
a0e783d26e81 exchange: make clone bundles non-experimental and enabled by default
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27737
diff changeset
4 """advertise pre-generated bundles to seed clones
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
5
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
6 "clonebundles" is a server-side extension used to advertise the existence
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
7 of pre-generated, externally hosted bundle files to clients that are
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
8 cloning so that cloning can be faster, more reliable, and require less
37498
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
9 resources on the server. "pullbundles" is a related feature for sending
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
10 pre-generated bundle files to clients as part of pull operations.
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
11
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
12 Cloning can be a CPU and I/O intensive operation on servers. Traditionally,
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
13 the server, in response to a client's request to clone, dynamically generates
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
14 a bundle containing the entire repository content and sends it to the client.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
15 There is no caching on the server and the server will have to redundantly
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
16 generate the same outgoing bundle in response to each clone request. For
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
17 servers with large repositories or with high clone volume, the load from
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
18 clones can make scaling the server challenging and costly.
26623
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
19
37498
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
20 This extension provides server operators the ability to offload
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
21 potentially expensive clone load to an external service. Pre-generated
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
22 bundles also allow using more CPU intensive compression, reducing the
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
23 effective bandwidth requirements.
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
24
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
25 Here's how clone bundles work:
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
26
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
27 1. A server operator establishes a mechanism for making bundle files available
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
28 on a hosting service where Mercurial clients can fetch them.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
29 2. A manifest file listing available bundle URLs and some optional metadata
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
30 is added to the Mercurial repository on the server.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
31 3. A client initiates a clone against a clone bundles aware server.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
32 4. The client sees the server is advertising clone bundles and fetches the
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
33 manifest listing available bundles.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
34 5. The client filters and sorts the available bundles based on what it
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
35 supports and prefers.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
36 6. The client downloads and applies an available bundle from the
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
37 server-specified URL.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
38 7. The client reconnects to the original server and performs the equivalent
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
39 of :hg:`pull` to retrieve all repository data not in the bundle. (The
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
40 repository could have been updated between when the bundle was created
37498
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
41 and when the client started the clone.) This may use "pullbundles".
26623
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
42
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
43 Instead of the server generating full repository bundles for every clone
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
44 request, it generates full bundles once and they are subsequently reused to
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
45 bootstrap new clones. The server may still transfer data at clone time.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
46 However, this is only data that has been added/changed since the bundle was
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
47 created. For large, established repositories, this can reduce server load for
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
48 clones to less than 1% of original.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
49
37498
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
50 Here's how pullbundles work:
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
51
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
52 1. A manifest file listing available bundles and describing the revisions
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
53 is added to the Mercurial repository on the server.
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
54 2. A new-enough client informs the server that it supports partial pulls
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
55 and initiates a pull.
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
56 3. If the server has pull bundles enabled and sees the client advertising
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
57 partial pulls, it checks for a matching pull bundle in the manifest.
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
58 A bundle matches if the format is supported by the client, the client
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
59 has the required revisions already and needs something from the bundle.
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
60 4. If there is at least one matching bundle, the server sends it to the client.
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
61 5. The client applies the bundle and notices that the server reply was
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
62 incomplete. It initiates another pull.
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
63
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
64 To work, this extension requires the following of server operators:
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
65
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
66 * Generating bundle files of repository content (typically periodically,
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
67 such as once per day).
37498
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
68 * Clone bundles: A file server that clients have network access to and that
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
69 Python knows how to talk to through its normal URL handling facility
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
70 (typically an HTTP/HTTPS server).
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
71 * A process for keeping the bundles manifest in sync with available bundle
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
72 files.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
73
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
74 Strictly speaking, using a static file hosting server isn't required: a server
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
75 operator could use a dynamic service for retrieving bundle data. However,
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
76 static file hosting services are simple and scalable and should be sufficient
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
77 for most needs.
26623
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
78
26884
762bf510b42c clonebundles: fix typo s/comand/command/
Javi Merino <merino.jav@gmail.com>
parents: 26857
diff changeset
79 Bundle files can be generated with the :hg:`bundle` command. Typically
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
80 :hg:`bundle --all` is used to produce a bundle of the entire repository.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
81
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
82 :hg:`debugcreatestreamclonebundle` can be used to produce a special
37498
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
83 *streaming clonebundle*. These are bundle files that are extremely efficient
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
84 to produce and consume (read: fast). However, they are larger than
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
85 traditional bundle formats and require that clients support the exact set
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
86 of repository data store formats in use by the repository that created them.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
87 Typically, a newer server can serve data that is compatible with older clients.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
88 However, *streaming clone bundles* don't have this guarantee. **Server
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
89 operators need to be aware that newer versions of Mercurial may produce
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
90 streaming clone bundles incompatible with older Mercurial versions.**
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
91
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
92 A server operator is responsible for creating a ``.hg/clonebundles.manifest``
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
93 file containing the list of available bundle files suitable for seeding
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
94 clones. If this file does not exist, the repository will not advertise the
37498
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
95 existence of clone bundles when clients connect. For pull bundles,
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
96 ``.hg/pullbundles.manifest`` is used.
26623
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
97
32454
702af1ad3b18 clonebundles: fix missing newline character
Matt Harbison <matt_harbison@yahoo.com>
parents: 31146
diff changeset
98 The manifest file contains a newline (\\n) delimited list of entries.
26623
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
99
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
100 Each line in this file defines an available bundle. Lines have the format:
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
101
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
102 <URL> [<key>=<value>[ <key>=<value>]]
26623
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
103
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
104 That is, a URL followed by an optional, space-delimited list of key=value
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
105 pairs describing additional properties of this bundle. Both keys and values
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
106 are URI encoded.
26623
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
107
37498
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
108 For pull bundles, the URL is a path under the ``.hg`` directory of the
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
109 repository.
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
110
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
111 Keys in UPPERCASE are reserved for use by Mercurial and are defined below.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
112 All non-uppercase keys can be used by site installations. An example use
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
113 for custom properties is to use the *datacenter* attribute to define which
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
114 data center a file is hosted in. Clients could then prefer a server in the
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
115 data center closest to them.
26623
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
116
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
117 The following reserved keys are currently defined:
26623
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
118
26644
74de1c59f71c clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26623
diff changeset
119 BUNDLESPEC
74de1c59f71c clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26623
diff changeset
120 A "bundle specification" string that describes the type of the bundle.
74de1c59f71c clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26623
diff changeset
121
74de1c59f71c clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26623
diff changeset
122 These are string values that are accepted by the "--type" argument of
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
123 :hg:`bundle`.
26644
74de1c59f71c clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26623
diff changeset
124
74de1c59f71c clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26623
diff changeset
125 The values are parsed in strict mode, which means they must be of the
74de1c59f71c clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26623
diff changeset
126 "<compression>-<type>" form. See
74de1c59f71c clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26623
diff changeset
127 mercurial.exchange.parsebundlespec() for more details.
74de1c59f71c clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26623
diff changeset
128
27886
0288e63ea3be clonebundles: improve BUNDLESPEC documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27738
diff changeset
129 :hg:`debugbundle --spec` can be used to print the bundle specification
0288e63ea3be clonebundles: improve BUNDLESPEC documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27738
diff changeset
130 string for a bundle file. The output of this command can be used verbatim
0288e63ea3be clonebundles: improve BUNDLESPEC documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27738
diff changeset
131 for the value of ``BUNDLESPEC`` (it is already escaped).
0288e63ea3be clonebundles: improve BUNDLESPEC documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27738
diff changeset
132
26644
74de1c59f71c clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26623
diff changeset
133 Clients will automatically filter out specifications that are unknown or
74de1c59f71c clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26623
diff changeset
134 unsupported so they won't attempt to download something that likely won't
74de1c59f71c clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26623
diff changeset
135 apply.
74de1c59f71c clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26623
diff changeset
136
74de1c59f71c clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26623
diff changeset
137 The actual value doesn't impact client behavior beyond filtering:
74de1c59f71c clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26623
diff changeset
138 clients will still sniff the bundle type from the header of downloaded
74de1c59f71c clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26623
diff changeset
139 files.
26645
2faa7671a4b3 clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26644
diff changeset
140
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
141 **Use of this key is highly recommended**, as it allows clients to
27886
0288e63ea3be clonebundles: improve BUNDLESPEC documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27738
diff changeset
142 easily skip unsupported bundles. If this key is not defined, an old
0288e63ea3be clonebundles: improve BUNDLESPEC documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27738
diff changeset
143 client may attempt to apply a bundle that it is incapable of reading.
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
144
26645
2faa7671a4b3 clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26644
diff changeset
145 REQUIRESNI
2faa7671a4b3 clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26644
diff changeset
146 Whether Server Name Indication (SNI) is required to connect to the URL.
2faa7671a4b3 clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26644
diff changeset
147 SNI allows servers to use multiple certificates on the same IP. It is
2faa7671a4b3 clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26644
diff changeset
148 somewhat common in CDNs and other hosting providers. Older Python
2faa7671a4b3 clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26644
diff changeset
149 versions do not support SNI. Defining this attribute enables clients
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
150 with older Python versions to filter this entry without experiencing
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
151 an opaque SSL failure at connection time.
26645
2faa7671a4b3 clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26644
diff changeset
152
2faa7671a4b3 clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26644
diff changeset
153 If this is defined, it is important to advertise a non-SNI fallback
2faa7671a4b3 clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26644
diff changeset
154 URL or clients running old Python releases may not be able to clone
2faa7671a4b3 clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26644
diff changeset
155 with the clonebundles facility.
2faa7671a4b3 clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26644
diff changeset
156
2faa7671a4b3 clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26644
diff changeset
157 Value should be "true".
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
158
37498
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
159 heads
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
160 Used for pull bundles. This contains the ``;`` separated changeset
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
161 hashes of the heads of the bundle content.
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
162
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
163 bases
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
164 Used for pull bundles. This contains the ``;`` separated changeset
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
165 hashes of the roots of the bundle content. This can be skipped if
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
166 the bundle was created without ``--base``.
aacfca6f9767 wireproto: support for pullbundles
Joerg Sonnenberger <joerg@bec.de>
parents: 32773
diff changeset
167
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
168 Manifests can contain multiple entries. Assuming metadata is defined, clients
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
169 will filter entries from the manifest that they don't support. The remaining
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
170 entries are optionally sorted by client preferences
32773
d25802b0eef5 clonebundles: reference correct config option
Gregory Szorc <gregory.szorc@gmail.com>
parents: 32454
diff changeset
171 (``ui.clonebundleprefers`` config option). The client then attempts
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
172 to fetch the bundle at the first URL in the remaining list.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
173
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
174 **Errors when downloading a bundle will fail the entire clone operation:
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
175 clients do not automatically fall back to a traditional clone.** The reason
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
176 for this is that if a server is using clone bundles, it is probably doing so
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
177 because the feature is necessary to help it scale. In other words, there
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
178 is an assumption that clone load will be offloaded to another service and
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
179 that the Mercurial server isn't responsible for serving this clone load.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
180 If that other service experiences issues and clients start mass falling back to
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
181 the original Mercurial server, the added clone load could overwhelm the server
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
182 due to unexpected load and effectively take it offline. Not having clients
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
183 automatically fall back to cloning from the original server mitigates this
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
184 scenario.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
185
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
186 Because there is no automatic Mercurial server fallback on failure of the
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
187 bundle hosting service, it is important for server operators to view the bundle
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
188 hosting service as an extension of the Mercurial server in terms of
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
189 availability and service level agreements: if the bundle hosting service goes
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
190 down, so does the ability for clients to clone. Note: clients will see a
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
191 message informing them how to bypass the clone bundles facility when a failure
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
192 occurs. So server operators should prepare for some people to follow these
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
193 instructions when a failure occurs, thus driving more load to the original
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
194 Mercurial server when the bundle hosting service fails.
26623
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
195 """
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
196
28095
7fa139eaebb4 clonebundles: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27886
diff changeset
197 from __future__ import absolute_import
7fa139eaebb4 clonebundles: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27886
diff changeset
198
26623
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
199 from mercurial import (
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
200 extensions,
37785
b4d85bc122bd wireproto: rename wireproto to wireprotov1server (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37498
diff changeset
201 wireprotov1server,
26623
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
202 )
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
203
29841
d5883fd055c6 extensions: change magic "shipped with hg" string
Augie Fackler <augie@google.com>
parents: 28095
diff changeset
204 testedwith = 'ships-with-hg-core'
26623
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
205
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
206 def capabilities(orig, repo, proto):
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
207 caps = orig(repo, proto)
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
208
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
209 # Only advertise if a manifest exists. This does add some I/O to requests.
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
210 # But this should be cheaper than a wasted network round trip due to
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
211 # missing file.
31146
16d8bec0177d clonebundle: use 'repo.vfs' instead of 'repo.opener'
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 29841
diff changeset
212 if repo.vfs.exists('clonebundles.manifest'):
26623
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
213 caps.append('clonebundles')
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
214
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
215 return caps
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
216
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
217 def extsetup(ui):
37785
b4d85bc122bd wireproto: rename wireproto to wireprotov1server (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37498
diff changeset
218 extensions.wrapfunction(wireprotov1server, '_capabilities', capabilities)