hgext/largefiles/__init__.py
author Siddharth Agarwal <sid0@fb.com>
Thu, 19 Sep 2013 09:45:00 -0700
changeset 19753 2c9645c0a582
parent 19071 64ea454e7d76
child 19779 fb6e87d93948
permissions -rw-r--r--
util.h: fix gcc version checking gcc doesn't have a predefined GCC_VERSION macro.
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
15168
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
     1
# Copyright 2009-2010 Gregory P. Ward
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
     2
# Copyright 2009-2010 Intelerad Medical Systems Incorporated
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
     3
# Copyright 2010-2011 Fog Creek Software
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
     4
# Copyright 2010-2011 Unity Technologies
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
     5
#
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
     6
# This software may be used and distributed according to the terms of the
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
     7
# GNU General Public License version 2 or any later version.
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
     8
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
     9
'''track large binary files
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
    10
15230
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    11
Large binary files tend to be not very compressible, not very
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    12
diffable, and not at all mergeable. Such files are not handled
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    13
efficiently by Mercurial's storage format (revlog), which is based on
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    14
compressed binary deltas; storing large binary files as regular
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    15
Mercurial files wastes bandwidth and disk space and increases
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    16
Mercurial's memory usage. The largefiles extension addresses these
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    17
problems by adding a centralized client-server layer on top of
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    18
Mercurial: largefiles live in a *central store* out on the network
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    19
somewhere, and you only fetch the revisions that you need when you
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    20
need them.
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    21
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    22
largefiles works by maintaining a "standin file" in .hglf/ for each
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    23
largefile. The standins are small (41 bytes: an SHA-1 hash plus
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    24
newline) and are tracked by Mercurial. Largefile revisions are
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    25
identified by the SHA-1 hash of their contents, which is written to
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    26
the standin. largefiles uses that revision ID to get/put largefile
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    27
revisions from/to the central store. This saves both disk space and
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    28
bandwidth, since you don't need to retrieve all historical revisions
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    29
of large files when you clone or pull.
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    30
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    31
To start a new repository or add new large binary files, just add
15352
b74f74b482d8 largefiles: improve markup in module help text
Martin Geisler <mg@aragost.com>
parents: 15304
diff changeset
    32
--large to your :hg:`add` command. For example::
15230
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    33
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    34
  $ dd if=/dev/urandom of=randomdata count=2000
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    35
  $ hg add --large randomdata
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    36
  $ hg commit -m 'add randomdata as a largefile'
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    37
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    38
When you push a changeset that adds/modifies largefiles to a remote
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    39
repository, its largefile revisions will be uploaded along with it.
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    40
Note that the remote Mercurial must also have the largefiles extension
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    41
enabled for this to work.
15168
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
    42
15230
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    43
When you pull a changeset that affects largefiles from a remote
18975
aa8205a9f51a largefiles: update help
Mads Kiilerich <madski@unity3d.com>
parents: 18704
diff changeset
    44
repository, the largefiles for the changeset will by default not be
aa8205a9f51a largefiles: update help
Mads Kiilerich <madski@unity3d.com>
parents: 18704
diff changeset
    45
pulled down. However, when you update to such a revision, any
aa8205a9f51a largefiles: update help
Mads Kiilerich <madski@unity3d.com>
parents: 18704
diff changeset
    46
largefiles needed by that revision are downloaded and cached (if
aa8205a9f51a largefiles: update help
Mads Kiilerich <madski@unity3d.com>
parents: 18704
diff changeset
    47
they have never been downloaded before). One way to pull largefiles
aa8205a9f51a largefiles: update help
Mads Kiilerich <madski@unity3d.com>
parents: 18704
diff changeset
    48
when pulling is thus to use --update, which will update your working
aa8205a9f51a largefiles: update help
Mads Kiilerich <madski@unity3d.com>
parents: 18704
diff changeset
    49
copy to the latest pulled revision (and thereby downloading any new
aa8205a9f51a largefiles: update help
Mads Kiilerich <madski@unity3d.com>
parents: 18704
diff changeset
    50
largefiles).
18704
d69585a5c5c0 largefiles: don't cache largefiles for pulled heads by default
Na'Tosha Bard <natosha@unity3d.com>
parents: 18599
diff changeset
    51
18976
6734951e2d24 largefiles: introduce lfpull command for pulling missing largefiles
Mads Kiilerich <madski@unity3d.com>
parents: 18975
diff changeset
    52
If you want to pull largefiles you don't need for update yet, then
18978
8abaadab9abb largefiles: introduce pull --lfrev option
Mads Kiilerich <madski@unity3d.com>
parents: 18976
diff changeset
    53
you can use pull with the `--lfrev` option or the :hg:`lfpull` command.
18976
6734951e2d24 largefiles: introduce lfpull command for pulling missing largefiles
Mads Kiilerich <madski@unity3d.com>
parents: 18975
diff changeset
    54
19071
64ea454e7d76 largefiles: fix typos in documentation
Wagner Bruna <wbruna@softwareexpress.com.br>
parents: 18980
diff changeset
    55
If you know you are pulling from a non-default location and want to
64ea454e7d76 largefiles: fix typos in documentation
Wagner Bruna <wbruna@softwareexpress.com.br>
parents: 18980
diff changeset
    56
download all the largefiles that correspond to the new changesets at
18979
1176832fc757 largefiles: introduce pulled() revset expression for use in --lfrev
Mads Kiilerich <madski@unity3d.com>
parents: 18978
diff changeset
    57
the same time, then you can pull with `--lfrev "pulled()"`.
1176832fc757 largefiles: introduce pulled() revset expression for use in --lfrev
Mads Kiilerich <madski@unity3d.com>
parents: 18978
diff changeset
    58
18975
aa8205a9f51a largefiles: update help
Mads Kiilerich <madski@unity3d.com>
parents: 18704
diff changeset
    59
If you just want to ensure that you will have the largefiles needed to
aa8205a9f51a largefiles: update help
Mads Kiilerich <madski@unity3d.com>
parents: 18704
diff changeset
    60
merge or rebase with new heads that you are pulling, then you can pull
18979
1176832fc757 largefiles: introduce pulled() revset expression for use in --lfrev
Mads Kiilerich <madski@unity3d.com>
parents: 18978
diff changeset
    61
with `--lfrev "head(pulled())"` flag to pre-emptively download any largefiles
18704
d69585a5c5c0 largefiles: don't cache largefiles for pulled heads by default
Na'Tosha Bard <natosha@unity3d.com>
parents: 18599
diff changeset
    62
that are new in the heads you are pulling.
18599
5cd1dbf4c5d2 largefiles: document behavior of caching largefiles for new heads
Na'Tosha Bard <natosha@unity3d.com>
parents: 17233
diff changeset
    63
18975
aa8205a9f51a largefiles: update help
Mads Kiilerich <madski@unity3d.com>
parents: 18704
diff changeset
    64
Keep in mind that network access may now be required to update to
aa8205a9f51a largefiles: update help
Mads Kiilerich <madski@unity3d.com>
parents: 18704
diff changeset
    65
changesets that you have not previously updated to. The nature of the
aa8205a9f51a largefiles: update help
Mads Kiilerich <madski@unity3d.com>
parents: 18704
diff changeset
    66
largefiles extension means that updating is no longer guaranteed to
aa8205a9f51a largefiles: update help
Mads Kiilerich <madski@unity3d.com>
parents: 18704
diff changeset
    67
be a local-only operation.
15230
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    68
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    69
If you already have large files tracked by Mercurial without the
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    70
largefiles extension, you will need to convert your repository in
15352
b74f74b482d8 largefiles: improve markup in module help text
Martin Geisler <mg@aragost.com>
parents: 15304
diff changeset
    71
order to benefit from largefiles. This is done with the
b74f74b482d8 largefiles: improve markup in module help text
Martin Geisler <mg@aragost.com>
parents: 15304
diff changeset
    72
:hg:`lfconvert` command::
15230
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    73
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    74
  $ hg lfconvert --size 10 oldrepo newrepo
15168
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
    75
15230
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    76
In repositories that already have largefiles in them, any new file
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    77
over 10MB will automatically be added as a largefile. To change this
15304
9aa9d4bb3d88 largefiles: rename config setting 'size' to 'minsize'
Greg Ward <greg@gerg.ca>
parents: 15291
diff changeset
    78
threshold, set ``largefiles.minsize`` in your Mercurial config file
9aa9d4bb3d88 largefiles: rename config setting 'size' to 'minsize'
Greg Ward <greg@gerg.ca>
parents: 15291
diff changeset
    79
to the minimum size in megabytes to track as a largefile, or use the
15230
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    80
--lfsize option to the add command (also in megabytes)::
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    81
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    82
  [largefiles]
15304
9aa9d4bb3d88 largefiles: rename config setting 'size' to 'minsize'
Greg Ward <greg@gerg.ca>
parents: 15291
diff changeset
    83
  minsize = 2
15230
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    84
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    85
  $ hg add --lfsize 2
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    86
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    87
The ``largefiles.patterns`` config option allows you to specify a list
15352
b74f74b482d8 largefiles: improve markup in module help text
Martin Geisler <mg@aragost.com>
parents: 15304
diff changeset
    88
of filename patterns (see :hg:`help patterns`) that should always be
15230
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    89
tracked as largefiles::
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    90
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    91
  [largefiles]
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    92
  patterns =
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    93
    *.jpg
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    94
    re:.*\.(png|bmp)$
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    95
    library.zip
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    96
    content/audio/*
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    97
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    98
Files that match one of these patterns will be added as largefiles
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    99
regardless of their size.
15743
6266b1b970a5 largefiles: clarify help when options are ignored until first add is done
Michal Sznajder <michalsznajder@gmail.com>
parents: 15352
diff changeset
   100
6266b1b970a5 largefiles: clarify help when options are ignored until first add is done
Michal Sznajder <michalsznajder@gmail.com>
parents: 15352
diff changeset
   101
The ``largefiles.minsize`` and ``largefiles.patterns`` config options
6266b1b970a5 largefiles: clarify help when options are ignored until first add is done
Michal Sznajder <michalsznajder@gmail.com>
parents: 15352
diff changeset
   102
will be ignored for any repositories not already containing a
6266b1b970a5 largefiles: clarify help when options are ignored until first add is done
Michal Sznajder <michalsznajder@gmail.com>
parents: 15352
diff changeset
   103
largefile. To add the first largefile to a repository, you must
6266b1b970a5 largefiles: clarify help when options are ignored until first add is done
Michal Sznajder <michalsznajder@gmail.com>
parents: 15352
diff changeset
   104
explicitly do so with the --large flag passed to the :hg:`add`
6266b1b970a5 largefiles: clarify help when options are ignored until first add is done
Michal Sznajder <michalsznajder@gmail.com>
parents: 15352
diff changeset
   105
command.
15168
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
   106
'''
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
   107
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
   108
from mercurial import commands
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
   109
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
   110
import lfcommands
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
   111
import reposetup
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
   112
import uisetup
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
   113
17233
acea82757d8a largefiles: mark as a first party extension
Matt Harbison <matt_harbison@yahoo.com>
parents: 15743
diff changeset
   114
testedwith = 'internal'
acea82757d8a largefiles: mark as a first party extension
Matt Harbison <matt_harbison@yahoo.com>
parents: 15743
diff changeset
   115
15168
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
   116
reposetup = reposetup.reposetup
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
   117
uisetup = uisetup.uisetup
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
   118
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
   119
commands.norepo += " lfconvert"
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
   120
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
   121
cmdtable = lfcommands.cmdtable