Merge with stable

chunkbuffer: use += rather than cStringIO to reduce memory footprint This significantly refactors the read() loop to use a queue of chunks. The queue is alternately filled to at least 256k and then emptied by concatenating onto the output buffer. For very large read sizes, += uses less memory because it can resize the target string in place.

httprepo: decompress stream incrementally to reduce memory usage

verify: initialize rp variable in case we hit out of memory

merge: drop reference to file contents after write This reduces memory usage on large consecutive gets.

revlog: drop cache after use to save memory footprint If we reconstruct back to back large versions, we need to drop the cache first to avoid doubling memory usage.

verify: reduce memory footprint when unpacking files By never holding a reference to the unpacked string, we avoid holding two consecutive large files in memory.

verify: report exceptions that don't have a str() method

color: fix test-diff-color breakage

hgcia/color: remove star imports This plays nicer with demandimport and allows pyflakes to detect undefined names.

hgfixes: added a fixer that makes bytes to be formatted correctly This patch implement a fixer that replaces all calls to the '%' when bytes arguments are used to a call to bytesformatter(), a function that knows how to format byte strings. As one can't be sure if a formatting call is done when only variables are used in a '%' call, these calls are also translated. The bytesformatter, in runtime, makes sure to return the "raw" % operation if that's what was intended.

py3kcompat: added a "compatibility layer" for py3k This patch adds some ugly constructs. The first of them is bytesformatter, a function that formats strings like when '%' is called. The main motivation for this function is py3k's strange behavior: >>> 'foo %s' % b'bar' "foo b'bar'" >>> b'foo %s' % b'bar' Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: unsupported operand type(s) for %: 'bytes' and 'bytes' >>> b'foo %s' % 'bar' Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: unsupported operand type(s) for %: 'bytes' and 'str' In other words, if we can't format bytes with bytes, and recall that all mercurial strings will be converted by a fixer, then things will break badly if we don't take a similar approach. The other addition with this patch is that the os.environ dictionary is monkeypatched to have bytes items. Hopefully this won't be needed in the future, as python 3.2 might get a os.environb dictionary that holds bytes items.

hgfixes: add a fixer to convert plain strings to bytestrings This patch implements a 2to3 fixer that converts all plain strings in a python source file to byte strings syntax. Example: foo = 'Normal string' would become foo = b'Normal string' The motivation behind this fixer can be found in http://selenic.com/pipermail/mercurial-devel/2010-June/022363.html or, in other words: the current hg source assumes that _most_ strings are "meant" to be byte sequences, so it makes sense to make the convertion implemented by this patch. As mentioned above, not all mercurial modules want to use strings as bytes, examples include i18n (which uses unicode), and demandimport (in py3k, module names are normal strings, thus unicode, and there's no need for a convertion). Therefore, these modules are blacklisted in the fixer. There are also a few functions that can take only unicode arguments, thus the convertion shouldn't be done for those.

revlog: add shallow header flag REVLOGSHALLOW header flag to mark revlog as shallow. The _shallow attribute of the revlog is used to check if the header flag is set.

revlog: add punched revision flag index flag to identify a revision as punched, i.e. it contains no data. REVIDX_PUNCHED_FLAG = 2, is used to mark a revision as punched. REVIDX_KNOWN_FLAGS is the accumulation of all index flags.