lfs: avoid quadratic performance in processing server responses
authorMatt Harbison <matt_harbison@yahoo.com>
Tue, 14 Jan 2020 20:05:37 -0500
changeset 44086 ffac09da7a19
parent 44085 0ee0a3f6a990
child 44087 dc9b53482689
lfs: avoid quadratic performance in processing server responses This is also adapted from the Facebook repo[1]. Unlike there, we were already reading the download stream in chunks and immediately writing it to disk, so we basically avoided the problem on download. There shouldn't be a lot of data to read on upload, but it's better to get rid of this pattern. [1] https://github.com/facebookexperimental/eden/commit/82df66ffe97e21f3ee73dfec093c87500fc1f6a7 Differential Revision: https://phab.mercurial-scm.org/D7882
hgext/lfs/blobstore.py
--- a/hgext/lfs/blobstore.py	Tue Jan 14 19:42:24 2020 -0500
+++ b/hgext/lfs/blobstore.py	Tue Jan 14 20:05:37 2020 -0500
@@ -503,7 +503,6 @@
         for k, v in headers:
             request.add_header(pycompat.strurl(k), pycompat.strurl(v))
 
-        response = b''
         try:
             with contextlib.closing(self.urlopener.open(request)) as res:
                 contentlength = res.info().get(b"content-length")
@@ -520,11 +519,14 @@
                     # blobstore
                     localstore.download(oid, res, contentlength)
                 else:
+                    blocks = []
                     while True:
                         data = res.read(1048576)
                         if not data:
                             break
-                        response += data
+                        blocks.append(data)
+
+                    response = b"".join(blocks)
                     if response:
                         ui.debug(b'lfs %s response: %s' % (action, response))
         except util.urlerr.httperror as ex: