annotate hgext/remotefilelog/README.md @ 43301:90213d027154

sidedatacopies: only fetch information once for merge

Before this change, merge would result in reading the data from revlog twice.
With this change, we keep the information in memory until we encounter the other
parent.

When looking at pypy, I see about 1/3 of the changesets with copy information
being merge. Not doing duplicated fetch for them provide a significant speedup.

revision: large amount; added files: large amount; rename small amount; c3b14617fbd7 9ba6ab77fd29
before: ! wall 0.767042 comb 0.760000 user 0.750000 sys 0.010000 (median of 11)
after:  ! wall 0.671162 comb 0.670000 user 0.650000 sys 0.020000 (median of 13)
revision: large amount; added files: small amount; rename small amount; c3b14617fbd7 f650a9b140d2
before: ! wall 1.170169 comb 1.170000 user 1.130000 sys 0.040000 (median of 10)
after:  ! wall 1.030596 comb 1.040000 user 1.010000 sys 0.030000 (median of 10)
revision: large amount; added files: large amount; rename large amount; 08ea3258278e d9fa043f30c0
before: ! wall 0.209846 comb 0.200000 user 0.200000 sys 0.000000 (median of 46)
after:  ! wall 0.170981 comb 0.170000 user 0.170000 sys 0.000000 (median of 56)
revision: small amount; added files: large amount; rename large amount; df6f7a526b60 a83dc6a2d56f
before: ! wall 0.013248 comb 0.010000 user 0.010000 sys 0.000000 (median of 223)
after:  ! wall 0.013295 comb 0.020000 user 0.020000 sys 0.000000 (median of 222)
revision: small amount; added files: large amount; rename small amount; 4aa4e1f8e19a 169138063d63
before: ! wall 0.001672 comb 0.000000 user 0.000000 sys 0.000000 (median of 1000)
after:  ! wall 0.001666 comb 0.000000 user 0.000000 sys 0.000000 (median of 1000)
revision: small amount; added files: small amount; rename small amount; 4bc173b045a6 964879152e2e
before: ! wall 0.000119 comb 0.000000 user 0.000000 sys 0.000000 (median of 8010)
after:  ! wall 0.000119 comb 0.000000 user 0.000000 sys 0.000000 (median of 8007)
revision: medium amount; added files: large amount; rename medium amount; c95f1ced15f2 2c68e87c3efe
before: ! wall 0.168599 comb 0.160000 user 0.160000 sys 0.000000 (median of 58)
after:  ! wall 0.133316 comb 0.140000 user 0.140000 sys 0.000000 (median of 73)
revision: medium amount; added files: medium amount; rename small amount; d343da0c55a8 d7746d32bf9d
before: ! wall 0.036052 comb 0.030000 user 0.030000 sys 0.000000 (median of 100)
after:  ! wall 0.032558 comb 0.030000 user 0.030000 sys 0.000000 (median of 100)

Differential Revision: https://phab.mercurial-scm.org/D7127

remotefilelog
=============

The remotefilelog extension allows Mercurial to clone shallow copies of a repository such that all file contents are left on the server and only downloaded on demand by the client. This greatly speeds up clone and pull performance for repositories that have long histories or that are growing quickly.

In addition, the extension allows using a caching layer (such as memcache) to serve the file contents, thus providing better scalability and reducing server load.

Installing
==========

**NOTE:** See the limitations section below to check if remotefilelog will work for your use case.

remotefilelog can be installed like any other Mercurial extension. Download the source code and add the remotefilelog subdirectory to your `hgrc`:

    :::ini
    [extensions]
    remotefilelog=path/to/remotefilelog/remotefilelog

Configuring
-----------

**Server**

* `server` (required) - Set to 'True' to indicate that the server can serve shallow clones.
* `serverexpiration` - The server keeps a local cache of recently requested file revision blobs in .hg/remotefilelogcache. This setting specifies how many days they should be kept locally.  Defaults to 30.

An example server configuration:

    :::ini
    [remotefilelog]
    server = True
    serverexpiration = 14

**Client**

* `cachepath` (required) - the location to store locally cached file revisions
* `cachelimit` - the maximum size of the cachepath. By default it's 1000 GB.
* `cachegroup` - the default unix group for the cachepath. Useful on shared systems so multiple users can read and write to the same cache.
* `cacheprocess` - the external process that will handle the remote caching layer. If not set, all requests will go to the Mercurial server.
* `fallbackpath` - the Mercurial repo path to fetch file revisions from. By default it uses the paths.default repo. This setting is useful for cloning from shallow clones and still talking to the central server for file revisions.
* `includepattern` - a list of regex patterns matching files that should be kept remotely. Defaults to all files.
* `excludepattern` - a list of regex patterns matching files that should not be kept remotely and should always be downloaded.
* `pullprefetch` - a revset of commits whose file content should be prefetched after every pull. The most common value for this will be '(bookmark() + head()) & public()'. This is useful in environments where offline work is common, since it will enable offline updating to, rebasing to, and committing on every head and bookmark.

An example client configuration:

    :::ini
    [remotefilelog]
    cachepath = /dev/shm/hgcache
    cachelimit = 2 GB

Using as a largefiles replacement
---------------------------------

remotefilelog can theoretically be used as a replacement for the largefiles extension. You can use the `includepattern` setting to specify which directories or file types are considered large and they will be left on the server. Unlike the largefiles extension, this can be done without converting the server repository. Only the client configuration needs to specify the patterns.

The include/exclude settings haven't been extensively tested, so this feature is still considered experimental.

An example largefiles style client configuration:

    :::ini
    [remotefilelog]
    cachepath = /dev/shm/hgcache
    cachelimit = 2 GB
    includepattern = *.sql3
      bin/*

Usage
=====

Once you have configured the server, you can get a shallow clone by doing:

    :::bash
    hg clone --shallow ssh://server//path/repo

After that, all normal mercurial commands should work.

Occasionly the client or server caches may grow too big. Run `hg gc` to clean up the cache. It will remove cached files that appear to no longer be necessary, or any files that exceed the configured maximum size. This does not improve performance; it just frees up space.

Limitations
===========

1. The extension must be used with Mercurial 3.3 (commit d7d08337b3f6) or higher (earlier versions of the extension work with earlier versions of Mercurial though, up to Mercurial 2.7).

2. remotefilelog has only been tested on linux with case-sensitive filesystems. It should work on other unix systems but may have problems on case-insensitive filesystems.

3. remotefilelog only works with ssh based Mercurial repos. http based repos are currently not supported, though it shouldn't be too difficult for some motivated individual to implement.

4. Tags are not supported in completely shallow repos. If you use tags in your repo you will have to specify `excludepattern=.hgtags` in your client configuration to ensure that file is downloaded. The include/excludepattern settings are experimental at the moment and have yet to be deployed in a production environment.

5. A few commands will be slower. `hg log <filename>` will be much slower since it has to walk the entire commit history instead of just the filelog. Use `hg log -f <filename>` instead, which remains very fast.

Contributing
============

Patches are welcome as pull requests, though they will be collapsed and rebased to maintain a linear history.  Tests can be run via:

    :::bash
    cd tests
    ./run-tests --with-hg=path/to/hgrepo/hg

We (Facebook) have to ask for a "Contributor License Agreement" from someone who sends in a patch or code that we want to include in the codebase. This is a legal requirement; a similar situation applies to Apache and other ASF projects.

If we ask you to fill out a CLA we'll direct you to our [online CLA page](https://developers.facebook.com/opensource/cla) where you can complete it easily. We use the same form as the Apache CLA so that friction is minimal.

License
=======

remotefilelog is made available under the terms of the GNU General Public License version 2, or any later version. See the COPYING file that accompanies this distribution for the full text of the license.