comparison hgext/convert/p4.py @ 25882:97a9f7602014 stable

convert: when getting file from Perforce concatenate data at the end

As it turned out, even when getting relatively small files, concatenating
string data every time when new chunk is received is very inefficient.
Maintaining a string list of data chunks and concatenating everything in one go
at the end seems much more efficient - in my testing it made getting 40 MB file
7 times faster, whilst converting of a particularly big changelist with some big
files went down from 20 hours to 3 hours.

25881:9de443515f1d

        lasterror = None
        while True:
            stdout = util.popen(cmd, mode='rb')

            mode = None
            contents = ""
            keywords = None

            for d in loaditer(stdout):
                code = d["code"]
                data = d.get("data")

                            keywords = self.re_keywords_old
                        elif "k" in flags:
                            keywords = self.re_keywords

                elif code == "text" or code == "binary":
                    contents += data

                lasterror = None

            if not lasterror:
                break

        if mode is None:
            return None, None

        if keywords:
            contents = keywords.sub("$\\1$", contents)
        if mode == "l" and contents.endswith("\n"):
            contents = contents[:-1]

25882:97a9f7602014

        lasterror = None
        while True:
            stdout = util.popen(cmd, mode='rb')

            mode = None
            contents = []
            keywords = None

            for d in loaditer(stdout):
                code = d["code"]
                data = d.get("data")

                            keywords = self.re_keywords_old
                        elif "k" in flags:
                            keywords = self.re_keywords

                elif code == "text" or code == "binary":
                    contents.append(data)

                lasterror = None

            if not lasterror:
                break

        if mode is None:
            return None, None

        contents = ''.join(contents)

        if keywords:
            contents = keywords.sub("$\\1$", contents)
        if mode == "l" and contents.endswith("\n"):
            contents = contents[:-1]