merge revlogng with mpm tip

Fix transaction handling during interleaved file split.

merge 0.8.1 with revlogng

Additional appendfile fixes for interleaved data/index files The appendfile code was not passing default version info to the changelog or manifest classes, and so they were always being created as version 0. revlog.checkinlinesize had to be corrected to seek to the end of the index file when no index file was passed (only clone does this)

Speedup revlog.ancestors for the linear case revlog.ancestors can be expensive on big repos. This cuts down the overall time for hg update by ~19% by short cutting revlog.ancestors when one of the revisions is reachable from another.

Reduce index memory usage by storing the bare string instead of tuples Storing the tuple returned by struct.unpack significantly increases the memory required to store the entire index in ram. This patch uses struct.unpack on demand instead.

New lazy index code for revlogs. This tunes for large repositories. It does not read the whole index file in one big chunk, but tries to buffer reads in more reasonable chunks instead. Search speeds are improved in two ways. When trying to find a specific sha hash, it searches from the end of the file backward. More recent entries are more likely to be relevant, especially the tip. Also, this can load only the mapping of nodes to revlog index number. Loading the map uses less cpu (no struct.unpack) and much less memory than loading both the map and the index. This cuts down the time for hg tip on the 80,000 changeset kernel repo from 1.8s to 3.69s. Most commands the pull a single rev out of a big index get roughly the same benefit. Commands that read the whole index are not slower.

Fill in the uncompressed size during revlog.addgroup This uses code from Matt to calculate the size change that would result from applying a delta to keep an accurate running total of the text size during revlog.addgroup

Fix inlined revlogs to seek to eof after opening "a+"

Create an atomic opener that does not automatically rename on close The revlog.checkinlinesize() uses an atomic opener to replace the index file after converting it from inline to traditional .i and .d files. If this operation is interrupted, the atomic file class can overwrite a valid file with a partially written one. This patch introduces an atomic opener that does not automatically replace the destination file with the tempfile. This way an interrupted checkinlinesize() call turns into a noop.

Make the appendfile class inline-data index friendly The appendfile class needs a few changes to make it work with interleaved index files. It needs to support the tell() method, opening in a+ mode, and it needs to delay the checkinlinesize call until after the append file is written. Given that open(file, "a+") doesn't always seek to the end of the file, this adds seek operations to appendfile that understand whence args

Fix bundle repos to use an index tuple consistent with revlogng The bundle repo code was adding a field to the index tuple, which confused the revlogng changes. This creates a new dict instead to maintain the extra bundle info.

Implement data inlined with the index file This patch allows you to optionally inline data bytes with the revlog index file. It saves considerable space and checkout time by reducing the number of inodes, wasted partial blocks and system calls. To use the inline data add this to your .hgrc [revlog] # inline data only works with revlogng format=1 # inline is the only valid flag right now. flags=inline

Implement revlogng. revlogng results in smaller indexes, can address larger data files, and supports flags and version numbers. By default the original revlog format is used. To use the new format, use the following .hgrc field: [revlog] # format choices are 0 (classic revlog format) and 1 revlogng format=1