changeset 17962:4c29668ca316 stable

util: make chunkbuffer non-quadratic on Windows

The old str-based += collector performed very nicely on Linux, but
turns out to be quadratically expensive on Windows, causing
chunkbuffer to dominate in profiles.

This list-based version has been measured to significantly improve
performance with large chunks on Windows, with negligible overall
overhead on Linux (though microbenchmarks show it to be about 50% slower).

This may increase memory overhead where += didn't behave quadratically. If we
want to gather up 1G of data to join, we temporarily have 1G in our
list and 1G in our string.

--- a/mercurial/util.py	Mon Nov 26 13:44:11 2012 -0600
+++ b/mercurial/util.py	Mon Nov 26 15:42:52 2012 -0600
@@ -899,7 +899,7 @@
         """Read L bytes of data from the iterator of chunks of data.
         Returns less than L bytes if the iterator runs dry."""
         left = l
-        buf = ''
+        buf = []
         queue = self._queue
         while left > 0:
             # refill the queue
@@ -917,11 +917,11 @@
             left -= len(chunk)
             if left < 0:
                 queue.appendleft(chunk[left:])
-                buf += chunk[:left]
+                buf.append(chunk[:left])
             else:
-                buf += chunk
+                buf.append(chunk)
 
-        return buf
+        return ''.join(buf)
 
 def filechunkiter(f, size=65536, limit=None):
     """Create a generator that produces the data in the file size