comparison mercurial/revlog.py @ 20180:969148b49fc6

revlog: allow tuning of the chunk cache size (via format.chunkcachesize)

Running perfmoonwalk on the Mercurial repo (with almost 20,000 changesets) on
Mac OS X with an SSD, before this change:

$ hg --config format.chunkcachesize=1024 perfmoonwalk
! wall 2.022021 comb 2.030000 user 1.970000 sys 0.060000 (best of 5)

(16,154 cache hits, 3,840 misses.)

$ hg --config format.chunkcachesize=4096 perfmoonwalk
! wall 1.901006 comb 1.900000 user 1.880000 sys 0.020000 (best of 6)

(19,003 hits, 991 misses.)

$ hg --config format.chunkcachesize=16384 perfmoonwalk
! wall 1.802775 comb 1.800000 user 1.800000 sys 0.000000 (best of 6)

(19,746 hits, 248 misses.)

$ hg --config format.chunkcachesize=32768 perfmoonwalk
! wall 1.818545 comb 1.810000 user 1.810000 sys 0.000000 (best of 6)

(19,870 hits, 124 misses.)

$ hg --config format.chunkcachesize=65536 perfmoonwalk
! wall 1.801350 comb 1.810000 user 1.800000 sys 0.010000 (best of 6)

(19,932 hits, 62 misses.)

$ hg --config format.chunkcachesize=131072 perfmoonwalk
! wall 1.805879 comb 1.820000 user 1.810000 sys 0.010000 (best of 6)

(19,963 hits, 31 misses.)

We may want to change the default size in the future based on testing and
user feedback.

20179:5bb3826bdac4

        self.datafile = indexfile[:-2] + ".d"
        self.opener = opener
        self._cache = None
        self._basecache = None
        self._chunkcache = (0, '')
        self.index = []
        self._pcache = {}
        self._nodecache = {nullid: nullrev}
        self._nodepos = None

            if 'revlogv1' in opts:
                if 'generaldelta' in opts:
                    v |= REVLOGGENERALDELTA
            else:
                v = 0

        i = ''
        self._initempty = True
        try:
            f = self.opener(self.indexfile)

            df = self.opener(self.datafile)

        # Cache data both forward and backward around the requested
        # data, in a fixed size window. This helps speed up operations
        # involving reading the revlog backwards.
        realoffset = offset & ~65535
        reallength = ((offset + length + 65536) & ~65535) - realoffset
        df.seek(realoffset)
        d = df.read(reallength)
        df.close()
        self._addchunk(realoffset, d)
        if offset != realoffset or reallength != length:

20180:969148b49fc6

        self.datafile = indexfile[:-2] + ".d"
        self.opener = opener
        self._cache = None
        self._basecache = None
        self._chunkcache = (0, '')
        self._chunkcachesize = 65536
        self.index = []
        self._pcache = {}
        self._nodecache = {nullid: nullrev}
        self._nodepos = None

            if 'revlogv1' in opts:
                if 'generaldelta' in opts:
                    v |= REVLOGGENERALDELTA
            else:
                v = 0
            if 'chunkcachesize' in opts:
                self._chunkcachesize = opts['chunkcachesize']

        if self._chunkcachesize <= 0:
            raise RevlogError(_('revlog chunk cache size %r is not greater '
                                'than 0') % self._chunkcachesize)
        elif self._chunkcachesize & (self._chunkcachesize - 1):
            raise RevlogError(_('revlog chunk cache size %r is not a power '
                                'of 2') % self._chunkcachesize)

        i = ''
        self._initempty = True
        try:
            f = self.opener(self.indexfile)

            df = self.opener(self.datafile)

        # Cache data both forward and backward around the requested
        # data, in a fixed size window. This helps speed up operations
        # involving reading the revlog backwards.
        cachesize = self._chunkcachesize
        realoffset = offset & ~(cachesize - 1)
        reallength = (((offset + length + cachesize) & ~(cachesize - 1))
                      - realoffset)
        df.seek(realoffset)
        d = df.read(reallength)
        df.close()
        self._addchunk(realoffset, d)
        if offset != realoffset or reallength != length: