hgext/largefiles/__init__.py
author Matt Mackall <mpm@selenic.com>
Fri, 15 May 2015 10:00:46 -0500
changeset 25089 c6427cd45760
parent 21770 15d434bee41c
child 25186 80c5b2666a96
permissions -rw-r--r--
setup: use try/except/finally This will raise a syntax error for people who attempt to use Py2.4, but that's already going to fail and we have no way to keep other 2.6isms from creeping in since we've removed the check-code rules and the buildbot.
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
15168
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
     1
# Copyright 2009-2010 Gregory P. Ward
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
     2
# Copyright 2009-2010 Intelerad Medical Systems Incorporated
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
     3
# Copyright 2010-2011 Fog Creek Software
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
     4
# Copyright 2010-2011 Unity Technologies
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
     5
#
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
     6
# This software may be used and distributed according to the terms of the
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
     7
# GNU General Public License version 2 or any later version.
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
     8
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
     9
'''track large binary files
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
    10
15230
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    11
Large binary files tend to be not very compressible, not very
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    12
diffable, and not at all mergeable. Such files are not handled
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    13
efficiently by Mercurial's storage format (revlog), which is based on
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    14
compressed binary deltas; storing large binary files as regular
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    15
Mercurial files wastes bandwidth and disk space and increases
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    16
Mercurial's memory usage. The largefiles extension addresses these
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    17
problems by adding a centralized client-server layer on top of
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    18
Mercurial: largefiles live in a *central store* out on the network
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    19
somewhere, and you only fetch the revisions that you need when you
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    20
need them.
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    21
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    22
largefiles works by maintaining a "standin file" in .hglf/ for each
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    23
largefile. The standins are small (41 bytes: an SHA-1 hash plus
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    24
newline) and are tracked by Mercurial. Largefile revisions are
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    25
identified by the SHA-1 hash of their contents, which is written to
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    26
the standin. largefiles uses that revision ID to get/put largefile
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    27
revisions from/to the central store. This saves both disk space and
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    28
bandwidth, since you don't need to retrieve all historical revisions
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    29
of large files when you clone or pull.
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    30
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    31
To start a new repository or add new large binary files, just add
15352
b74f74b482d8 largefiles: improve markup in module help text
Martin Geisler <mg@aragost.com>
parents: 15304
diff changeset
    32
--large to your :hg:`add` command. For example::
15230
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    33
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    34
  $ dd if=/dev/urandom of=randomdata count=2000
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    35
  $ hg add --large randomdata
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    36
  $ hg commit -m 'add randomdata as a largefile'
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    37
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    38
When you push a changeset that adds/modifies largefiles to a remote
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    39
repository, its largefile revisions will be uploaded along with it.
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    40
Note that the remote Mercurial must also have the largefiles extension
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    41
enabled for this to work.
15168
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
    42
15230
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    43
When you pull a changeset that affects largefiles from a remote
18975
aa8205a9f51a largefiles: update help
Mads Kiilerich <madski@unity3d.com>
parents: 18704
diff changeset
    44
repository, the largefiles for the changeset will by default not be
aa8205a9f51a largefiles: update help
Mads Kiilerich <madski@unity3d.com>
parents: 18704
diff changeset
    45
pulled down. However, when you update to such a revision, any
aa8205a9f51a largefiles: update help
Mads Kiilerich <madski@unity3d.com>
parents: 18704
diff changeset
    46
largefiles needed by that revision are downloaded and cached (if
aa8205a9f51a largefiles: update help
Mads Kiilerich <madski@unity3d.com>
parents: 18704
diff changeset
    47
they have never been downloaded before). One way to pull largefiles
aa8205a9f51a largefiles: update help
Mads Kiilerich <madski@unity3d.com>
parents: 18704
diff changeset
    48
when pulling is thus to use --update, which will update your working
aa8205a9f51a largefiles: update help
Mads Kiilerich <madski@unity3d.com>
parents: 18704
diff changeset
    49
copy to the latest pulled revision (and thereby downloading any new
aa8205a9f51a largefiles: update help
Mads Kiilerich <madski@unity3d.com>
parents: 18704
diff changeset
    50
largefiles).
18704
d69585a5c5c0 largefiles: don't cache largefiles for pulled heads by default
Na'Tosha Bard <natosha@unity3d.com>
parents: 18599
diff changeset
    51
18976
6734951e2d24 largefiles: introduce lfpull command for pulling missing largefiles
Mads Kiilerich <madski@unity3d.com>
parents: 18975
diff changeset
    52
If you want to pull largefiles you don't need for update yet, then
18978
8abaadab9abb largefiles: introduce pull --lfrev option
Mads Kiilerich <madski@unity3d.com>
parents: 18976
diff changeset
    53
you can use pull with the `--lfrev` option or the :hg:`lfpull` command.
18976
6734951e2d24 largefiles: introduce lfpull command for pulling missing largefiles
Mads Kiilerich <madski@unity3d.com>
parents: 18975
diff changeset
    54
19071
64ea454e7d76 largefiles: fix typos in documentation
Wagner Bruna <wbruna@softwareexpress.com.br>
parents: 18980
diff changeset
    55
If you know you are pulling from a non-default location and want to
64ea454e7d76 largefiles: fix typos in documentation
Wagner Bruna <wbruna@softwareexpress.com.br>
parents: 18980
diff changeset
    56
download all the largefiles that correspond to the new changesets at
18979
1176832fc757 largefiles: introduce pulled() revset expression for use in --lfrev
Mads Kiilerich <madski@unity3d.com>
parents: 18978
diff changeset
    57
the same time, then you can pull with `--lfrev "pulled()"`.
1176832fc757 largefiles: introduce pulled() revset expression for use in --lfrev
Mads Kiilerich <madski@unity3d.com>
parents: 18978
diff changeset
    58
18975
aa8205a9f51a largefiles: update help
Mads Kiilerich <madski@unity3d.com>
parents: 18704
diff changeset
    59
If you just want to ensure that you will have the largefiles needed to
aa8205a9f51a largefiles: update help
Mads Kiilerich <madski@unity3d.com>
parents: 18704
diff changeset
    60
merge or rebase with new heads that you are pulling, then you can pull
18979
1176832fc757 largefiles: introduce pulled() revset expression for use in --lfrev
Mads Kiilerich <madski@unity3d.com>
parents: 18978
diff changeset
    61
with `--lfrev "head(pulled())"` flag to pre-emptively download any largefiles
18704
d69585a5c5c0 largefiles: don't cache largefiles for pulled heads by default
Na'Tosha Bard <natosha@unity3d.com>
parents: 18599
diff changeset
    62
that are new in the heads you are pulling.
18599
5cd1dbf4c5d2 largefiles: document behavior of caching largefiles for new heads
Na'Tosha Bard <natosha@unity3d.com>
parents: 17233
diff changeset
    63
18975
aa8205a9f51a largefiles: update help
Mads Kiilerich <madski@unity3d.com>
parents: 18704
diff changeset
    64
Keep in mind that network access may now be required to update to
aa8205a9f51a largefiles: update help
Mads Kiilerich <madski@unity3d.com>
parents: 18704
diff changeset
    65
changesets that you have not previously updated to. The nature of the
aa8205a9f51a largefiles: update help
Mads Kiilerich <madski@unity3d.com>
parents: 18704
diff changeset
    66
largefiles extension means that updating is no longer guaranteed to
aa8205a9f51a largefiles: update help
Mads Kiilerich <madski@unity3d.com>
parents: 18704
diff changeset
    67
be a local-only operation.
15230
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    68
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    69
If you already have large files tracked by Mercurial without the
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    70
largefiles extension, you will need to convert your repository in
15352
b74f74b482d8 largefiles: improve markup in module help text
Martin Geisler <mg@aragost.com>
parents: 15304
diff changeset
    71
order to benefit from largefiles. This is done with the
b74f74b482d8 largefiles: improve markup in module help text
Martin Geisler <mg@aragost.com>
parents: 15304
diff changeset
    72
:hg:`lfconvert` command::
15230
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    73
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    74
  $ hg lfconvert --size 10 oldrepo newrepo
15168
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
    75
15230
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    76
In repositories that already have largefiles in them, any new file
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    77
over 10MB will automatically be added as a largefile. To change this
15304
9aa9d4bb3d88 largefiles: rename config setting 'size' to 'minsize'
Greg Ward <greg@gerg.ca>
parents: 15291
diff changeset
    78
threshold, set ``largefiles.minsize`` in your Mercurial config file
9aa9d4bb3d88 largefiles: rename config setting 'size' to 'minsize'
Greg Ward <greg@gerg.ca>
parents: 15291
diff changeset
    79
to the minimum size in megabytes to track as a largefile, or use the
15230
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    80
--lfsize option to the add command (also in megabytes)::
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    81
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    82
  [largefiles]
15304
9aa9d4bb3d88 largefiles: rename config setting 'size' to 'minsize'
Greg Ward <greg@gerg.ca>
parents: 15291
diff changeset
    83
  minsize = 2
15230
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    84
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    85
  $ hg add --lfsize 2
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    86
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    87
The ``largefiles.patterns`` config option allows you to specify a list
15352
b74f74b482d8 largefiles: improve markup in module help text
Martin Geisler <mg@aragost.com>
parents: 15304
diff changeset
    88
of filename patterns (see :hg:`help patterns`) that should always be
15230
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    89
tracked as largefiles::
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    90
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    91
  [largefiles]
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    92
  patterns =
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    93
    *.jpg
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    94
    re:.*\.(png|bmp)$
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    95
    library.zip
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    96
    content/audio/*
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    97
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    98
Files that match one of these patterns will be added as largefiles
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    99
regardless of their size.
15743
6266b1b970a5 largefiles: clarify help when options are ignored until first add is done
Michal Sznajder <michalsznajder@gmail.com>
parents: 15352
diff changeset
   100
6266b1b970a5 largefiles: clarify help when options are ignored until first add is done
Michal Sznajder <michalsznajder@gmail.com>
parents: 15352
diff changeset
   101
The ``largefiles.minsize`` and ``largefiles.patterns`` config options
6266b1b970a5 largefiles: clarify help when options are ignored until first add is done
Michal Sznajder <michalsznajder@gmail.com>
parents: 15352
diff changeset
   102
will be ignored for any repositories not already containing a
6266b1b970a5 largefiles: clarify help when options are ignored until first add is done
Michal Sznajder <michalsznajder@gmail.com>
parents: 15352
diff changeset
   103
largefile. To add the first largefile to a repository, you must
6266b1b970a5 largefiles: clarify help when options are ignored until first add is done
Michal Sznajder <michalsznajder@gmail.com>
parents: 15352
diff changeset
   104
explicitly do so with the --large flag passed to the :hg:`add`
6266b1b970a5 largefiles: clarify help when options are ignored until first add is done
Michal Sznajder <michalsznajder@gmail.com>
parents: 15352
diff changeset
   105
command.
15168
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
   106
'''
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
   107
21770
15d434bee41c largefiles: define norepo in command decorator
Gregory Szorc <gregory.szorc@gmail.com>
parents: 20858
diff changeset
   108
from mercurial import hg, localrepo
15168
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
   109
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
   110
import lfcommands
20858
bc56ec9e64df hg: introduce "wirepeersetupfuncs" to setup wire peer by extensions (issue4109)
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 19928
diff changeset
   111
import proto
15168
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
   112
import reposetup
19779
fb6e87d93948 largefiles: setup "largefiles" feature in each repositories individually
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 19071
diff changeset
   113
import uisetup as uisetupmod
15168
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
   114
17233
acea82757d8a largefiles: mark as a first party extension
Matt Harbison <matt_harbison@yahoo.com>
parents: 15743
diff changeset
   115
testedwith = 'internal'
acea82757d8a largefiles: mark as a first party extension
Matt Harbison <matt_harbison@yahoo.com>
parents: 15743
diff changeset
   116
15168
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
   117
reposetup = reposetup.reposetup
19779
fb6e87d93948 largefiles: setup "largefiles" feature in each repositories individually
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 19071
diff changeset
   118
fb6e87d93948 largefiles: setup "largefiles" feature in each repositories individually
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 19071
diff changeset
   119
def featuresetup(ui, supported):
19928
d1ac3790e10a localrepo: invoke only feature setup functions for enabled extensions
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 19779
diff changeset
   120
    # don't die on seeing a repo with the largefiles requirement
d1ac3790e10a localrepo: invoke only feature setup functions for enabled extensions
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 19779
diff changeset
   121
    supported |= set(['largefiles'])
19779
fb6e87d93948 largefiles: setup "largefiles" feature in each repositories individually
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 19071
diff changeset
   122
fb6e87d93948 largefiles: setup "largefiles" feature in each repositories individually
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 19071
diff changeset
   123
def uisetup(ui):
fb6e87d93948 largefiles: setup "largefiles" feature in each repositories individually
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 19071
diff changeset
   124
    localrepo.localrepository.featuresetupfuncs.add(featuresetup)
20858
bc56ec9e64df hg: introduce "wirepeersetupfuncs" to setup wire peer by extensions (issue4109)
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 19928
diff changeset
   125
    hg.wirepeersetupfuncs.append(proto.wirereposetup)
19779
fb6e87d93948 largefiles: setup "largefiles" feature in each repositories individually
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 19071
diff changeset
   126
    uisetupmod.uisetup(ui)
15168
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
   127
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
   128
cmdtable = lfcommands.cmdtable