changeset 35551:fa9dd53eb23e

lfs: introduce a localstore method for downloading from remote stores The current local.write() method requires the full data, which means concatenating file chunks in memory when downloading from a git server. The dedicated method downloads in chunks, verifies the content on the fly, and creates the usercache hardlink if successful. It can also be used for the file system based remotestore. An explicit division of labor between downloading from a remote store (which should be verified) and writing to the store because of a commit or similar (which doesn't need verification), seems clearer. I can't figure out how to make a similar function for upload, because for a file remote store, it's a simple open/read/write operation. For a gitremote store, it's open the file and a urlreq.request(), and process that.
author Matt Harbison <matt_harbison@yahoo.com>
date Thu, 21 Dec 2017 18:56:04 -0500
parents ed30934165c9
children fd610befc37f
files hgext/lfs/blobstore.py
diffstat 1 files changed, 20 insertions(+), 0 deletions(-) [+]
line wrap: on
line diff
--- a/hgext/lfs/blobstore.py	Tue Jan 02 12:14:08 2018 +0900
+++ b/hgext/lfs/blobstore.py	Thu Dec 21 18:56:04 2017 -0500
@@ -114,6 +114,26 @@
 
         return self.vfs(oid, 'rb')
 
+    def download(self, oid, src):
+        """Read the blob from the remote source in chunks, verify the content,
+        and write to this local blobstore."""
+        sha256 = hashlib.sha256()
+
+        with self.vfs(oid, 'wb', atomictemp=True) as fp:
+            for chunk in util.filechunkiter(src, size=1048576):
+                fp.write(chunk)
+                sha256.update(chunk)
+
+            realoid = sha256.hexdigest()
+            if realoid != oid:
+                raise error.Abort(_('corrupt remote lfs object: %s') % oid)
+
+        # XXX: should we verify the content of the cache, and hardlink back to
+        # the local store on success, but truncate, write and link on failure?
+        if not self.cachevfs.exists(oid):
+            self.ui.note(_('lfs: adding %s to the usercache\n') % oid)
+            lfutil.link(self.vfs.join(oid), self.cachevfs.join(oid))
+
     def write(self, oid, data, verify=True):
         """Write blob to local blobstore."""
         if verify: