util: make chunkbuffer non-quadratic on Windows
The old str-based += collector performed very nicely on Linux, but
turns out to be quadratically expensive on Windows, causing
chunkbuffer to dominate in profiles.
This list-based version has been measured to significantly improve
performance with large chunks on Windows, with negligible overall
overhead on Linux (though microbenchmarks show it to be about 50% slower).
This may increase memory overhead where += didn't behave quadratically. If we
want to gather up 1G of data to join, we temporarily have 1G in our
list and 1G in our string.
--- a/mercurial/util.py Mon Nov 26 13:44:11 2012 -0600
+++ b/mercurial/util.py Mon Nov 26 15:42:52 2012 -0600
@@ -899,7 +899,7 @@
"""Read L bytes of data from the iterator of chunks of data.
Returns less than L bytes if the iterator runs dry."""
left = l
- buf = ''
+ buf = []
queue = self._queue
while left > 0:
# refill the queue
@@ -917,11 +917,11 @@
left -= len(chunk)
if left < 0:
queue.appendleft(chunk[left:])
- buf += chunk[:left]
+ buf.append(chunk[:left])
else:
- buf += chunk
+ buf.append(chunk)
- return buf
+ return ''.join(buf)
def filechunkiter(f, size=65536, limit=None):
"""Create a generator that produces the data in the file size