changegroup: sparsely populate fnodes stable
authorGregory Szorc <gregory.szorc@gmail.com>
Thu, 06 Nov 2014 22:48:20 -0800
branchstable
changeset 23226 5dcaed20b27c
parent 23225 bdf7b1ea1dae
child 23231 32dadb2637f4
changegroup: sparsely populate fnodes Previously, fnodes had a key and empty dict value for every element in changedfiles. This is somewhat wasteful. Empty dicts in CPython consume a lot more memory than you would expect - 280 bytes. On mozilla-central, which has ~190,000 files/fnodes keys, the previous loop populating fnodes allocated 91,924 KB of memory, most of that for the empty dicts. With this patch in place, our peak RSS during mozilla-central clone drops: before: 364,356 KB after: 326,008 KB delta: -38,348 KB When combined with the previous patch, total peak RSS decrease is now 190,116 KB.
mercurial/changegroup.py
--- a/mercurial/changegroup.py	Thu Nov 06 22:33:48 2014 -0800
+++ b/mercurial/changegroup.py	Thu Nov 06 22:48:20 2014 -0800
@@ -335,9 +335,6 @@
             yield chunk
         progress(msgbundling, None)
 
-        for f in changedfiles:
-            fnodes[f] = {}
-
         # Callback for the manifest, used to collect linkrevs for filelog
         # revisions.
         # Returns the linkrev node (collected in lookupcl).
@@ -349,7 +346,7 @@
                     if f in changedfiles:
                         # record the first changeset introducing this filelog
                         # version
-                        fnodes[f].setdefault(n, clnode)
+                        fnodes.setdefault(f, {}).setdefault(n, clnode)
             return clnode
 
         mfnodes = self.prune(mf, mfs, commonrevs, source)