changeset 35012:d80380ba8e7d

changegroup: use any node, not min(), in treemanifest's generatemanifests This is fixing quadratic behavior, which is probably not noticeable in the common case, but if a very large directory gets added here, it can get pretty bad. This was noticed because we had some pushes that spent >25s in changegroup generation calling min() here, according to profiling. The original reasoning for min() being used in 829d369fc5a8 was that, at that point in the series, we were adding almost everything to tmfnodes during the first iteration through the loop , so we needed to avoid sending child directories before parents. Later changes made it so that the child directories were added only when we visited the parent directory (not all of them on the first iteration), so this is no longer necessary - there won't be any child directories in tmfnodes before the parents have been sent. This does mean that the manifests are now exchanged unordered, whereas previously we would essentially do [a, b, b/c, b/c/d, e], we now can send a, b, and e in any order; b/c must still follow b, and b/c/d must still follow b/c. Differential Revision: https://phab.mercurial-scm.org/D1351
author Kyle Lippincott <spectral@google.com>
date Wed, 08 Nov 2017 18:24:43 -0800
parents a2dfc723b6b5
children 36507048da0f
files mercurial/changegroup.py
diffstat 1 files changed, 4 insertions(+), 6 deletions(-) [+]
line wrap: on
line diff
--- a/mercurial/changegroup.py	Tue Nov 07 10:16:53 2017 -0800
+++ b/mercurial/changegroup.py	Wed Nov 08 18:24:43 2017 -0800
@@ -692,7 +692,7 @@
         # Callback for the manifest, used to collect linkrevs for filelog
         # revisions.
         # Returns the linkrev node (collected in lookupcl).
-        def makelookupmflinknode(dir):
+        def makelookupmflinknode(dir, nodes):
             if fastpathlinkrev:
                 assert not dir
                 return mfs.__getitem__
@@ -713,7 +713,7 @@
                 the client before you can trust the list of files and
                 treemanifests to send.
                 """
-                clnode = tmfnodes[dir][x]
+                clnode = nodes[x]
                 mdata = mfl.get(dir, x).readfast(shallow=True)
                 for p, n, fl in mdata.iterentries():
                     if fl == 't': # subdirectory manifest
@@ -733,15 +733,13 @@
 
         size = 0
         while tmfnodes:
-            dir = min(tmfnodes)
-            nodes = tmfnodes[dir]
+            dir, nodes = tmfnodes.popitem()
             prunednodes = self.prune(dirlog(dir), nodes, commonrevs)
             if not dir or prunednodes:
                 for x in self._packmanifests(dir, prunednodes,
-                                             makelookupmflinknode(dir)):
+                                             makelookupmflinknode(dir, nodes)):
                     size += len(x)
                     yield x
-            del tmfnodes[dir]
         self._verbosenote(_('%8.i (manifests)\n') % size)
         yield self._manifestsdone()