changegroup: sparsely populate fnodes
Previously, fnodes had a key and empty dict value for every element in
changedfiles. This is somewhat wasteful. Empty dicts in CPython consume
a lot more memory than you would expect - 280 bytes.
On mozilla-central, which has ~190,000 files/fnodes keys, the previous
loop populating fnodes allocated 91,924 KB of memory, most of that for
the empty dicts.
With this patch in place, our peak RSS during mozilla-central clone
drops:
before: 364,356 KB
after: 326,008 KB
delta: -38,348 KB
When combined with the previous patch, total peak RSS decrease is now
190,116 KB.
--- a/mercurial/changegroup.py Thu Nov 06 22:33:48 2014 -0800
+++ b/mercurial/changegroup.py Thu Nov 06 22:48:20 2014 -0800
@@ -335,9 +335,6 @@
yield chunk
progress(msgbundling, None)
- for f in changedfiles:
- fnodes[f] = {}
-
# Callback for the manifest, used to collect linkrevs for filelog
# revisions.
# Returns the linkrev node (collected in lookupcl).
@@ -349,7 +346,7 @@
if f in changedfiles:
# record the first changeset introducing this filelog
# version
- fnodes[f].setdefault(n, clnode)
+ fnodes.setdefault(f, {}).setdefault(n, clnode)
return clnode
mfnodes = self.prune(mf, mfs, commonrevs, source)