lfs: avoid quadratic performance in processing server responses
This is also adapted from the Facebook repo[1]. Unlike there, we were already
reading the download stream in chunks and immediately writing it to disk, so we
basically avoided the problem on download. There shouldn't be a lot of data to
read on upload, but it's better to get rid of this pattern.
[1] https://github.com/facebookexperimental/eden/commit/82df66ffe97e21f3ee73dfec093c87500fc1f6a7
Differential Revision: https://phab.mercurial-scm.org/D7882
--- a/hgext/lfs/blobstore.py Tue Jan 14 19:42:24 2020 -0500
+++ b/hgext/lfs/blobstore.py Tue Jan 14 20:05:37 2020 -0500
@@ -503,7 +503,6 @@
for k, v in headers:
request.add_header(pycompat.strurl(k), pycompat.strurl(v))
- response = b''
try:
with contextlib.closing(self.urlopener.open(request)) as res:
contentlength = res.info().get(b"content-length")
@@ -520,11 +519,14 @@
# blobstore
localstore.download(oid, res, contentlength)
else:
+ blocks = []
while True:
data = res.read(1048576)
if not data:
break
- response += data
+ blocks.append(data)
+
+ response = b"".join(blocks)
if response:
ui.debug(b'lfs %s response: %s' % (action, response))
except util.urlerr.httperror as ex: