comparison mercurial/util.py @ 17962:4c29668ca316 stable

util: make chunkbuffer non-quadratic on Windows

The old str-based += collector performed very nicely on Linux, but
turns out to be quadratically expensive on Windows, causing
chunkbuffer to dominate in profiles.

This list-based version has been measured to significantly improve
performance with large chunks on Windows, with negligible overall
overhead on Linux (though microbenchmarks show it to be about 50% slower).

This may increase memory overhead where += didn't behave quadratically. If we
want to gather up 1G of data to join, we temporarily have 1G in our
list and 1G in our string.

17961:b0affcb67cba

    def read(self, l):
        """Read L bytes of data from the iterator of chunks of data.
        Returns less than L bytes if the iterator runs dry."""
        left = l
        buf = ''
        queue = self._queue
        while left > 0:
            # refill the queue
            if not queue:
                target = 2**18

            chunk = queue.popleft()
            left -= len(chunk)
            if left < 0:
                queue.appendleft(chunk[left:])
                buf += chunk[:left]
            else:
                buf += chunk

        return buf

def filechunkiter(f, size=65536, limit=None):
    """Create a generator that produces the data in the file size
    (default 65536) bytes at a time, up to optional limit (default is
    to read all data).  Chunks may be less than size bytes if the

17962:4c29668ca316

    def read(self, l):
        """Read L bytes of data from the iterator of chunks of data.
        Returns less than L bytes if the iterator runs dry."""
        left = l
        buf = []
        queue = self._queue
        while left > 0:
            # refill the queue
            if not queue:
                target = 2**18

            chunk = queue.popleft()
            left -= len(chunk)
            if left < 0:
                queue.appendleft(chunk[left:])
                buf.append(chunk[:left])
            else:
                buf.append(chunk)

        return ''.join(buf)

def filechunkiter(f, size=65536, limit=None):
    """Create a generator that produces the data in the file size
    (default 65536) bytes at a time, up to optional limit (default is
    to read all data).  Chunks may be less than size bytes if the