view hgext/largefiles/__init__.py @ 40811:e13ab4acf555

rust: peek_mut optim for lazy ancestors

This is one of the two optimizations that are also
present in the Python code: replacing pairs of pop/push
on the BinaryHeap by single updates, hence having it
under the hood maintain its consistency (sift) only once.

On Mozilla central, the measured gain (see details below)
is around 7%.

Creating the PeekMut object by calling peek_mut() right away
instead of peek() first is less efficient (gain is only 4%, stats
not included).
Our interpretation is that its creation has a cost which is vasted
in the cases where it ends by droping the value (Peekmut::pop()
just does self.heap.pop() anyway). On the other hand, the immutable
peek() is very fast: it's just taking a reference in the
underlying vector.

The Python version still has another optimization:
if parent(current) == current-1, then the heap doesn't need
to maintain its consistency, since we already know that
it's bigger than all the others in the heap.
Rust's BinaryHeap doesn't allow us to mutate its biggest
element with no housekeeping, but we tried it anyway, with a
copy of the BinaryHeap implementation with a dedicaded added
method: it's not worth the technical debt in our opinion
(we measured only a further 1.6% improvement).
One possible explanation would be that the sift is really fast
anyway in that case, whereas it's not in the case of Python,
because it's at least partly done in slow Python code.
Still it's possible that replacing BinaryHeap by something more
dedicated to discrete ordered types could be faster.

Measurements on mozilla-central:

Three runs of 'hg perfancestors' on the parent changeset:

Moyenne des médianes: 0.100587
! wall 0.100062 comb 0.100000 user 0.100000 sys 0.000000 (best of 98)
! wall 0.135804 comb 0.130000 user 0.130000 sys 0.000000 (max of 98)
! wall 0.102864 comb 0.102755 user 0.099286 sys 0.003469 (avg of 98)
! wall 0.101486 comb 0.110000 user 0.110000 sys 0.000000 (median of 98)

! wall 0.096804 comb 0.090000 user 0.090000 sys 0.000000 (best of 100)
! wall 0.132235 comb 0.130000 user 0.120000 sys 0.010000 (max of 100)
! wall 0.100258 comb 0.100300 user 0.096000 sys 0.004300 (avg of 100)
! wall 0.098384 comb 0.100000 user 0.100000 sys 0.000000 (median of 100)

! wall 0.099925 comb 0.100000 user 0.100000 sys 0.000000 (best of 98)
! wall 0.133518 comb 0.140000 user 0.130000 sys 0.010000 (max of 98)
! wall 0.102381 comb 0.102449 user 0.098265 sys 0.004184 (avg of 98)
! wall 0.101891 comb 0.090000 user 0.090000 sys 0.000000 (median of 98)

Mean of the medians: 0.100587

On the present changeset:

! wall 0.091344 comb 0.090000 user 0.090000 sys 0.000000 (best of 100)
! wall 0.122728 comb 0.120000 user 0.110000 sys 0.010000 (max of 100)
! wall 0.093268 comb 0.093300 user 0.089300 sys 0.004000 (avg of 100)
! wall 0.092567 comb 0.100000 user 0.090000 sys 0.010000 (median of 100)

! wall 0.093294 comb 0.080000 user 0.080000 sys 0.000000 (best of 100)
! wall 0.144887 comb 0.150000 user 0.140000 sys 0.010000 (max of 100)
! wall 0.097708 comb 0.097700 user 0.093400 sys 0.004300 (avg of 100)
! wall 0.094980 comb 0.100000 user 0.090000 sys 0.010000 (median of 100)

! wall 0.091262 comb 0.090000 user 0.080000 sys 0.010000 (best of 100)
! wall 0.123772 comb 0.130000 user 0.120000 sys 0.010000 (max of 100)
! wall 0.093188 comb 0.093200 user 0.089300 sys 0.003900 (avg of 100)
! wall 0.092364 comb 0.100000 user 0.090000 sys 0.010000 (median of 100)

Mean of the medians is 0.0933

Differential Revision: https://phab.mercurial-scm.org/D5358

# Copyright 2009-2010 Gregory P. Ward
# Copyright 2009-2010 Intelerad Medical Systems Incorporated
# Copyright 2010-2011 Fog Creek Software
# Copyright 2010-2011 Unity Technologies
#
# This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version.

'''track large binary files

Large binary files tend to be not very compressible, not very
diffable, and not at all mergeable. Such files are not handled
efficiently by Mercurial's storage format (revlog), which is based on
compressed binary deltas; storing large binary files as regular
Mercurial files wastes bandwidth and disk space and increases
Mercurial's memory usage. The largefiles extension addresses these
problems by adding a centralized client-server layer on top of
Mercurial: largefiles live in a *central store* out on the network
somewhere, and you only fetch the revisions that you need when you
need them.

largefiles works by maintaining a "standin file" in .hglf/ for each
largefile. The standins are small (41 bytes: an SHA-1 hash plus
newline) and are tracked by Mercurial. Largefile revisions are
identified by the SHA-1 hash of their contents, which is written to
the standin. largefiles uses that revision ID to get/put largefile
revisions from/to the central store. This saves both disk space and
bandwidth, since you don't need to retrieve all historical revisions
of large files when you clone or pull.

To start a new repository or add new large binary files, just add
--large to your :hg:`add` command. For example::

  $ dd if=/dev/urandom of=randomdata count=2000
  $ hg add --large randomdata
  $ hg commit -m "add randomdata as a largefile"

When you push a changeset that adds/modifies largefiles to a remote
repository, its largefile revisions will be uploaded along with it.
Note that the remote Mercurial must also have the largefiles extension
enabled for this to work.

When you pull a changeset that affects largefiles from a remote
repository, the largefiles for the changeset will by default not be
pulled down. However, when you update to such a revision, any
largefiles needed by that revision are downloaded and cached (if
they have never been downloaded before). One way to pull largefiles
when pulling is thus to use --update, which will update your working
copy to the latest pulled revision (and thereby downloading any new
largefiles).

If you want to pull largefiles you don't need for update yet, then
you can use pull with the `--lfrev` option or the :hg:`lfpull` command.

If you know you are pulling from a non-default location and want to
download all the largefiles that correspond to the new changesets at
the same time, then you can pull with `--lfrev "pulled()"`.

If you just want to ensure that you will have the largefiles needed to
merge or rebase with new heads that you are pulling, then you can pull
with `--lfrev "head(pulled())"` flag to pre-emptively download any largefiles
that are new in the heads you are pulling.

Keep in mind that network access may now be required to update to
changesets that you have not previously updated to. The nature of the
largefiles extension means that updating is no longer guaranteed to
be a local-only operation.

If you already have large files tracked by Mercurial without the
largefiles extension, you will need to convert your repository in
order to benefit from largefiles. This is done with the
:hg:`lfconvert` command::

  $ hg lfconvert --size 10 oldrepo newrepo

In repositories that already have largefiles in them, any new file
over 10MB will automatically be added as a largefile. To change this
threshold, set ``largefiles.minsize`` in your Mercurial config file
to the minimum size in megabytes to track as a largefile, or use the
--lfsize option to the add command (also in megabytes)::

  [largefiles]
  minsize = 2

  $ hg add --lfsize 2

The ``largefiles.patterns`` config option allows you to specify a list
of filename patterns (see :hg:`help patterns`) that should always be
tracked as largefiles::

  [largefiles]
  patterns =
    *.jpg
    re:.*\\.(png|bmp)$
    library.zip
    content/audio/*

Files that match one of these patterns will be added as largefiles
regardless of their size.

The ``largefiles.minsize`` and ``largefiles.patterns`` config options
will be ignored for any repositories not already containing a
largefile. To add the first largefile to a repository, you must
explicitly do so with the --large flag passed to the :hg:`add`
command.
'''
from __future__ import absolute_import

from mercurial import (
    hg,
    localrepo,
    registrar,
)

from . import (
    lfcommands,
    overrides,
    proto,
    reposetup,
    uisetup as uisetupmod,
)

# Note for extension authors: ONLY specify testedwith = 'ships-with-hg-core' for
# extensions which SHIP WITH MERCURIAL. Non-mainline extensions should
# be specifying the version(s) of Mercurial they are tested with, or
# leave the attribute unspecified.
testedwith = 'ships-with-hg-core'

configtable = {}
configitem = registrar.configitem(configtable)

configitem('largefiles', 'minsize',
    default=configitem.dynamicdefault,
)
configitem('largefiles', 'patterns',
    default=list,
)
configitem('largefiles', 'usercache',
    default=None,
)

reposetup = reposetup.reposetup

def featuresetup(ui, supported):
    # don't die on seeing a repo with the largefiles requirement
    supported |= {'largefiles'}

def uisetup(ui):
    localrepo.featuresetupfuncs.add(featuresetup)
    hg.wirepeersetupfuncs.append(proto.wirereposetup)
    uisetupmod.uisetup(ui)

cmdtable = lfcommands.cmdtable
revsetpredicate = overrides.revsetpredicate