author | Siddharth Agarwal <sid0@fb.com> |
Thu, 19 Sep 2013 09:45:00 -0700 | |
changeset 19753 | 2c9645c0a582 |
parent 19071 | 64ea454e7d76 |
child 19779 | fb6e87d93948 |
permissions | -rw-r--r-- |
15168 | 1 |
# Copyright 2009-2010 Gregory P. Ward |
2 |
# Copyright 2009-2010 Intelerad Medical Systems Incorporated |
|
3 |
# Copyright 2010-2011 Fog Creek Software |
|
4 |
# Copyright 2010-2011 Unity Technologies |
|
5 |
# |
|
6 |
# This software may be used and distributed according to the terms of the |
|
7 |
# GNU General Public License version 2 or any later version. |
|
8 |
||
9 |
'''track large binary files |
|
10 |
||
15230 | 11 |
Large binary files tend to be not very compressible, not very |
12 |
diffable, and not at all mergeable. Such files are not handled |
|
13 |
efficiently by Mercurial's storage format (revlog), which is based on |
|
14 |
compressed binary deltas; storing large binary files as regular |
|
15 |
Mercurial files wastes bandwidth and disk space and increases |
|
16 |
Mercurial's memory usage. The largefiles extension addresses these |
|
17 |
problems by adding a centralized client-server layer on top of |
|
18 |
Mercurial: largefiles live in a *central store* out on the network |
|
19 |
somewhere, and you only fetch the revisions that you need when you |
|
20 |
need them. |
|
21 |
||
22 |
largefiles works by maintaining a "standin file" in .hglf/ for each |
|
23 |
largefile. The standins are small (41 bytes: an SHA-1 hash plus |
|
24 |
newline) and are tracked by Mercurial. Largefile revisions are |
|
25 |
identified by the SHA-1 hash of their contents, which is written to |
|
26 |
the standin. largefiles uses that revision ID to get/put largefile |
|
27 |
revisions from/to the central store. This saves both disk space and |
|
28 |
bandwidth, since you don't need to retrieve all historical revisions |
|
29 |
of large files when you clone or pull. |
|
30 |
||
31 |
To start a new repository or add new large binary files, just add |
|
15352
b74f74b482d8
largefiles: improve markup in module help text
Martin Geisler <mg@aragost.com>
parents:
15304
diff
changeset
|
32 |
--large to your :hg:`add` command. For example:: |
15230 | 33 |
|
34 |
$ dd if=/dev/urandom of=randomdata count=2000 |
|
35 |
$ hg add --large randomdata |
|
36 |
$ hg commit -m 'add randomdata as a largefile' |
|
37 |
||
38 |
When you push a changeset that adds/modifies largefiles to a remote |
|
39 |
repository, its largefile revisions will be uploaded along with it. |
|
40 |
Note that the remote Mercurial must also have the largefiles extension |
|
41 |
enabled for this to work. |
|
15168 | 42 |
|
15230 | 43 |
When you pull a changeset that affects largefiles from a remote |
18975
aa8205a9f51a
largefiles: update help
Mads Kiilerich <madski@unity3d.com>
parents:
18704
diff
changeset
|
44 |
repository, the largefiles for the changeset will by default not be |
aa8205a9f51a
largefiles: update help
Mads Kiilerich <madski@unity3d.com>
parents:
18704
diff
changeset
|
45 |
pulled down. However, when you update to such a revision, any |
aa8205a9f51a
largefiles: update help
Mads Kiilerich <madski@unity3d.com>
parents:
18704
diff
changeset
|
46 |
largefiles needed by that revision are downloaded and cached (if |
aa8205a9f51a
largefiles: update help
Mads Kiilerich <madski@unity3d.com>
parents:
18704
diff
changeset
|
47 |
they have never been downloaded before). One way to pull largefiles |
aa8205a9f51a
largefiles: update help
Mads Kiilerich <madski@unity3d.com>
parents:
18704
diff
changeset
|
48 |
when pulling is thus to use --update, which will update your working |
aa8205a9f51a
largefiles: update help
Mads Kiilerich <madski@unity3d.com>
parents:
18704
diff
changeset
|
49 |
copy to the latest pulled revision (and thereby downloading any new |
aa8205a9f51a
largefiles: update help
Mads Kiilerich <madski@unity3d.com>
parents:
18704
diff
changeset
|
50 |
largefiles). |
18704
d69585a5c5c0
largefiles: don't cache largefiles for pulled heads by default
Na'Tosha Bard <natosha@unity3d.com>
parents:
18599
diff
changeset
|
51 |
|
18976
6734951e2d24
largefiles: introduce lfpull command for pulling missing largefiles
Mads Kiilerich <madski@unity3d.com>
parents:
18975
diff
changeset
|
52 |
If you want to pull largefiles you don't need for update yet, then |
18978
8abaadab9abb
largefiles: introduce pull --lfrev option
Mads Kiilerich <madski@unity3d.com>
parents:
18976
diff
changeset
|
53 |
you can use pull with the `--lfrev` option or the :hg:`lfpull` command. |
18976
6734951e2d24
largefiles: introduce lfpull command for pulling missing largefiles
Mads Kiilerich <madski@unity3d.com>
parents:
18975
diff
changeset
|
54 |
|
19071
64ea454e7d76
largefiles: fix typos in documentation
Wagner Bruna <wbruna@softwareexpress.com.br>
parents:
18980
diff
changeset
|
55 |
If you know you are pulling from a non-default location and want to |
64ea454e7d76
largefiles: fix typos in documentation
Wagner Bruna <wbruna@softwareexpress.com.br>
parents:
18980
diff
changeset
|
56 |
download all the largefiles that correspond to the new changesets at |
18979
1176832fc757
largefiles: introduce pulled() revset expression for use in --lfrev
Mads Kiilerich <madski@unity3d.com>
parents:
18978
diff
changeset
|
57 |
the same time, then you can pull with `--lfrev "pulled()"`. |
1176832fc757
largefiles: introduce pulled() revset expression for use in --lfrev
Mads Kiilerich <madski@unity3d.com>
parents:
18978
diff
changeset
|
58 |
|
18975
aa8205a9f51a
largefiles: update help
Mads Kiilerich <madski@unity3d.com>
parents:
18704
diff
changeset
|
59 |
If you just want to ensure that you will have the largefiles needed to |
aa8205a9f51a
largefiles: update help
Mads Kiilerich <madski@unity3d.com>
parents:
18704
diff
changeset
|
60 |
merge or rebase with new heads that you are pulling, then you can pull |
18979
1176832fc757
largefiles: introduce pulled() revset expression for use in --lfrev
Mads Kiilerich <madski@unity3d.com>
parents:
18978
diff
changeset
|
61 |
with `--lfrev "head(pulled())"` flag to pre-emptively download any largefiles |
18704
d69585a5c5c0
largefiles: don't cache largefiles for pulled heads by default
Na'Tosha Bard <natosha@unity3d.com>
parents:
18599
diff
changeset
|
62 |
that are new in the heads you are pulling. |
18599
5cd1dbf4c5d2
largefiles: document behavior of caching largefiles for new heads
Na'Tosha Bard <natosha@unity3d.com>
parents:
17233
diff
changeset
|
63 |
|
18975
aa8205a9f51a
largefiles: update help
Mads Kiilerich <madski@unity3d.com>
parents:
18704
diff
changeset
|
64 |
Keep in mind that network access may now be required to update to |
aa8205a9f51a
largefiles: update help
Mads Kiilerich <madski@unity3d.com>
parents:
18704
diff
changeset
|
65 |
changesets that you have not previously updated to. The nature of the |
aa8205a9f51a
largefiles: update help
Mads Kiilerich <madski@unity3d.com>
parents:
18704
diff
changeset
|
66 |
largefiles extension means that updating is no longer guaranteed to |
aa8205a9f51a
largefiles: update help
Mads Kiilerich <madski@unity3d.com>
parents:
18704
diff
changeset
|
67 |
be a local-only operation. |
15230 | 68 |
|
69 |
If you already have large files tracked by Mercurial without the |
|
70 |
largefiles extension, you will need to convert your repository in |
|
15352
b74f74b482d8
largefiles: improve markup in module help text
Martin Geisler <mg@aragost.com>
parents:
15304
diff
changeset
|
71 |
order to benefit from largefiles. This is done with the |
b74f74b482d8
largefiles: improve markup in module help text
Martin Geisler <mg@aragost.com>
parents:
15304
diff
changeset
|
72 |
:hg:`lfconvert` command:: |
15230 | 73 |
|
74 |
$ hg lfconvert --size 10 oldrepo newrepo |
|
15168 | 75 |
|
15230 | 76 |
In repositories that already have largefiles in them, any new file |
77 |
over 10MB will automatically be added as a largefile. To change this |
|
15304
9aa9d4bb3d88
largefiles: rename config setting 'size' to 'minsize'
Greg Ward <greg@gerg.ca>
parents:
15291
diff
changeset
|
78 |
threshold, set ``largefiles.minsize`` in your Mercurial config file |
9aa9d4bb3d88
largefiles: rename config setting 'size' to 'minsize'
Greg Ward <greg@gerg.ca>
parents:
15291
diff
changeset
|
79 |
to the minimum size in megabytes to track as a largefile, or use the |
15230 | 80 |
--lfsize option to the add command (also in megabytes):: |
81 |
||
82 |
[largefiles] |
|
15304
9aa9d4bb3d88
largefiles: rename config setting 'size' to 'minsize'
Greg Ward <greg@gerg.ca>
parents:
15291
diff
changeset
|
83 |
minsize = 2 |
15230 | 84 |
|
85 |
$ hg add --lfsize 2 |
|
86 |
||
87 |
The ``largefiles.patterns`` config option allows you to specify a list |
|
15352
b74f74b482d8
largefiles: improve markup in module help text
Martin Geisler <mg@aragost.com>
parents:
15304
diff
changeset
|
88 |
of filename patterns (see :hg:`help patterns`) that should always be |
15230 | 89 |
tracked as largefiles:: |
90 |
||
91 |
[largefiles] |
|
92 |
patterns = |
|
93 |
*.jpg |
|
94 |
re:.*\.(png|bmp)$ |
|
95 |
library.zip |
|
96 |
content/audio/* |
|
97 |
||
98 |
Files that match one of these patterns will be added as largefiles |
|
99 |
regardless of their size. |
|
15743
6266b1b970a5
largefiles: clarify help when options are ignored until first add is done
Michal Sznajder <michalsznajder@gmail.com>
parents:
15352
diff
changeset
|
100 |
|
6266b1b970a5
largefiles: clarify help when options are ignored until first add is done
Michal Sznajder <michalsznajder@gmail.com>
parents:
15352
diff
changeset
|
101 |
The ``largefiles.minsize`` and ``largefiles.patterns`` config options |
6266b1b970a5
largefiles: clarify help when options are ignored until first add is done
Michal Sznajder <michalsznajder@gmail.com>
parents:
15352
diff
changeset
|
102 |
will be ignored for any repositories not already containing a |
6266b1b970a5
largefiles: clarify help when options are ignored until first add is done
Michal Sznajder <michalsznajder@gmail.com>
parents:
15352
diff
changeset
|
103 |
largefile. To add the first largefile to a repository, you must |
6266b1b970a5
largefiles: clarify help when options are ignored until first add is done
Michal Sznajder <michalsznajder@gmail.com>
parents:
15352
diff
changeset
|
104 |
explicitly do so with the --large flag passed to the :hg:`add` |
6266b1b970a5
largefiles: clarify help when options are ignored until first add is done
Michal Sznajder <michalsznajder@gmail.com>
parents:
15352
diff
changeset
|
105 |
command. |
15168 | 106 |
''' |
107 |
||
108 |
from mercurial import commands |
|
109 |
||
110 |
import lfcommands |
|
111 |
import reposetup |
|
112 |
import uisetup |
|
113 |
||
17233
acea82757d8a
largefiles: mark as a first party extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
15743
diff
changeset
|
114 |
testedwith = 'internal' |
acea82757d8a
largefiles: mark as a first party extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
15743
diff
changeset
|
115 |
|
15168 | 116 |
reposetup = reposetup.reposetup |
117 |
uisetup = uisetup.uisetup |
|
118 |
||
119 |
commands.norepo += " lfconvert" |
|
120 |
||
121 |
cmdtable = lfcommands.cmdtable |