comparison mercurial/changegroup.py @ 23226:5dcaed20b27c stable

changegroup: sparsely populate fnodes

Previously, fnodes had a key and empty dict value for every element in
changedfiles. This is somewhat wasteful. Empty dicts in CPython consume
a lot more memory than you would expect - 280 bytes.

On mozilla-central, which has ~190,000 files/fnodes keys, the previous
loop populating fnodes allocated 91,924 KB of memory, most of that for
the empty dicts.

With this patch in place, our peak RSS during mozilla-central clone
drops:

before:  364,356 KB
after:   326,008 KB
delta:   -38,348 KB

When combined with the previous patch, total peak RSS decrease is now
190,116 KB.

23225:bdf7b1ea1dae

        for chunk in self.group(clnodes, cl, lookupcl, units=_('changesets'),
                                reorder=reorder):
            yield chunk
        progress(msgbundling, None)

        for f in changedfiles:
            fnodes[f] = {}

        # Callback for the manifest, used to collect linkrevs for filelog
        # revisions.
        # Returns the linkrev node (collected in lookupcl).
        def lookupmf(x):
            clnode = mfs[x]

                mdata = mf.readfast(x)
                for f, n in mdata.iteritems():
                    if f in changedfiles:
                        # record the first changeset introducing this filelog
                        # version
                        fnodes[f].setdefault(n, clnode)
            return clnode

        mfnodes = self.prune(mf, mfs, commonrevs, source)
        for chunk in self.group(mfnodes, mf, lookupmf, units=_('manifests'),
                                reorder=reorder):

23226:5dcaed20b27c

        for chunk in self.group(clnodes, cl, lookupcl, units=_('changesets'),
                                reorder=reorder):
            yield chunk
        progress(msgbundling, None)

        # Callback for the manifest, used to collect linkrevs for filelog
        # revisions.
        # Returns the linkrev node (collected in lookupcl).
        def lookupmf(x):
            clnode = mfs[x]

                mdata = mf.readfast(x)
                for f, n in mdata.iteritems():
                    if f in changedfiles:
                        # record the first changeset introducing this filelog
                        # version
                        fnodes.setdefault(f, {}).setdefault(n, clnode)
            return clnode

        mfnodes = self.prune(mf, mfs, commonrevs, source)
        for chunk in self.group(mfnodes, mf, lookupmf, units=_('manifests'),
                                reorder=reorder):