Mercurial > hg
comparison mercurial/changegroup.py @ 23226:5dcaed20b27c stable
changegroup: sparsely populate fnodes
Previously, fnodes had a key and empty dict value for every element in
changedfiles. This is somewhat wasteful. Empty dicts in CPython consume
a lot more memory than you would expect - 280 bytes.
On mozilla-central, which has ~190,000 files/fnodes keys, the previous
loop populating fnodes allocated 91,924 KB of memory, most of that for
the empty dicts.
With this patch in place, our peak RSS during mozilla-central clone
drops:
before: 364,356 KB
after: 326,008 KB
delta: -38,348 KB
When combined with the previous patch, total peak RSS decrease is now
190,116 KB.
author | Gregory Szorc <gregory.szorc@gmail.com> |
---|---|
date | Thu, 06 Nov 2014 22:48:20 -0800 |
parents | bdf7b1ea1dae |
children | 18cc87e4375a cc0ff93d0c0c |
comparison
equal
deleted
inserted
replaced
23225:bdf7b1ea1dae | 23226:5dcaed20b27c |
---|---|
333 for chunk in self.group(clnodes, cl, lookupcl, units=_('changesets'), | 333 for chunk in self.group(clnodes, cl, lookupcl, units=_('changesets'), |
334 reorder=reorder): | 334 reorder=reorder): |
335 yield chunk | 335 yield chunk |
336 progress(msgbundling, None) | 336 progress(msgbundling, None) |
337 | 337 |
338 for f in changedfiles: | |
339 fnodes[f] = {} | |
340 | |
341 # Callback for the manifest, used to collect linkrevs for filelog | 338 # Callback for the manifest, used to collect linkrevs for filelog |
342 # revisions. | 339 # revisions. |
343 # Returns the linkrev node (collected in lookupcl). | 340 # Returns the linkrev node (collected in lookupcl). |
344 def lookupmf(x): | 341 def lookupmf(x): |
345 clnode = mfs[x] | 342 clnode = mfs[x] |
347 mdata = mf.readfast(x) | 344 mdata = mf.readfast(x) |
348 for f, n in mdata.iteritems(): | 345 for f, n in mdata.iteritems(): |
349 if f in changedfiles: | 346 if f in changedfiles: |
350 # record the first changeset introducing this filelog | 347 # record the first changeset introducing this filelog |
351 # version | 348 # version |
352 fnodes[f].setdefault(n, clnode) | 349 fnodes.setdefault(f, {}).setdefault(n, clnode) |
353 return clnode | 350 return clnode |
354 | 351 |
355 mfnodes = self.prune(mf, mfs, commonrevs, source) | 352 mfnodes = self.prune(mf, mfs, commonrevs, source) |
356 for chunk in self.group(mfnodes, mf, lookupmf, units=_('manifests'), | 353 for chunk in self.group(mfnodes, mf, lookupmf, units=_('manifests'), |
357 reorder=reorder): | 354 reorder=reorder): |