revlog: enforce chunk slicing down to a certain size
authorBoris Feld <boris.feld@octobus.net>
Tue, 10 Jul 2018 12:20:57 +0200
changeset 38644 43d0619cec90
parent 38643 967fee55e8d9
child 38645 02850baddadd
revlog: enforce chunk slicing down to a certain size Limit maximum chunk size to 4x final size when reading a revision from a revlog. We only apply this logic when the target size is known from the revlog. Ideally, revlog's delta chain would be written in a way that does not trigger this extra slicing often. However, having this second guarantee that we won't read unexpectedly large amounts of memory in all cases is important for the future. Future delta chain building algorithms might have good reason to create delta chain with such characteristics. Including this code in core as soon as possible will make Mercurial 4.7 forward-compatible with such improvement.
mercurial/revlog.py
--- a/mercurial/revlog.py	Tue Jul 10 11:57:33 2018 +0200
+++ b/mercurial/revlog.py	Tue Jul 10 12:20:57 2018 +0200
@@ -1949,7 +1949,7 @@
         """
         return self.decompress(self._getsegmentforrevs(rev, rev, df=df)[1])
 
-    def _chunks(self, revs, df=None):
+    def _chunks(self, revs, df=None, targetsize=None):
         """Obtain decompressed chunks for the specified revisions.
 
         Accepts an iterable of numeric revisions that are assumed to be in
@@ -1976,7 +1976,7 @@
         if not self._withsparseread:
             slicedchunks = (revs,)
         else:
-            slicedchunks = _slicechunk(self, revs)
+            slicedchunks = _slicechunk(self, revs, targetsize)
 
         for revschunk in slicedchunks:
             firstrev = revschunk[0]
@@ -2079,7 +2079,12 @@
             # drop cache to save memory
             self._cache = None
 
-            bins = self._chunks(chain, df=_df)
+            targetsize = None
+            rawsize = self.index[rev][2]
+            if 0 <= rawsize:
+                targetsize = 4 * rawsize
+
+            bins = self._chunks(chain, df=_df, targetsize=targetsize)
             if rawtext is None:
                 rawtext = bytes(bins[0])
                 bins = bins[1:]