Mercurial > hg
changeset 17962:4c29668ca316 stable
util: make chunkbuffer non-quadratic on Windows
The old str-based += collector performed very nicely on Linux, but
turns out to be quadratically expensive on Windows, causing
chunkbuffer to dominate in profiles.
This list-based version has been measured to significantly improve
performance with large chunks on Windows, with negligible overall
overhead on Linux (though microbenchmarks show it to be about 50% slower).
This may increase memory overhead where += didn't behave quadratically. If we
want to gather up 1G of data to join, we temporarily have 1G in our
list and 1G in our string.
| author | Matt Mackall <mpm@selenic.com> |
|---|---|
| date | Mon, 26 Nov 2012 15:42:52 -0600 |
| parents | b0affcb67cba |
| children | 6180dcb29ec5 |
| files | mercurial/util.py |
| diffstat | 1 files changed, 4 insertions(+), 4 deletions(-) [+] |
line wrap: on
line diff
--- a/mercurial/util.py Mon Nov 26 13:44:11 2012 -0600 +++ b/mercurial/util.py Mon Nov 26 15:42:52 2012 -0600 @@ -899,7 +899,7 @@ """Read L bytes of data from the iterator of chunks of data. Returns less than L bytes if the iterator runs dry.""" left = l - buf = '' + buf = [] queue = self._queue while left > 0: # refill the queue @@ -917,11 +917,11 @@ left -= len(chunk) if left < 0: queue.appendleft(chunk[left:]) - buf += chunk[:left] + buf.append(chunk[:left]) else: - buf += chunk + buf.append(chunk) - return buf + return ''.join(buf) def filechunkiter(f, size=65536, limit=None): """Create a generator that produces the data in the file size