comparison mercurial/wireproto.py @ 35787:a84dbc87dae9

exchange: send bundle2 stream clones uncompressed

Stream clones don't compress well. And compression undermines
a point of stream clones which is to trade significant CPU
reductions by increasing size.

Building upon our introduction of metadata to communicate bundle
information back to callers of exchange.getbundlechunks(), we add
an attribute to the bundler that communicates whether the bundle is
best left uncompressed. We return this attribute as part of the bundle
metadata. And the wire protocol honors it when determining whether
to compress the wire protocol response.

The added test demonstrates that the raw result from the wire
protocol is not compressed. It also demonstrates that the server
will serve stream responses when the feature isn't enabled. We'll
address that in another commit.

The effect of this change is that server-side CPU usage for bundle2
stream clones is significantly reduced by removing zstd compression.
For the mozilla-unified repository:

before: 37.69 user 8.01 system
after:  27.38 user 7.34 system

Assuming things are CPU bound, that ~10s reduction would translate to
faster clones on the client. zstd can decompress at >1 GB/s. So the
overhead from decompression on the client is small in the grand scheme
of things. But if zlib compression were being used, the overhead would
be much greater.

Differential Revision: https://phab.mercurial-scm.org/D1926

35786:8144f1b07e21

            if proto.name == 'http':
                return ooberror(bundle2required)
            raise error.Abort(bundle2requiredmain,
                              hint=bundle2requiredhint)

    preferuncompressed = False

    try:
        if repo.ui.configbool('server', 'disablefullbundle'):
            # Check to see if this is a full clone.
            clheads = set(repo.changelog.heads())

                    _('server has pull-based clones disabled'),
                    hint=_('remove --pull if specified or upgrade Mercurial'))

        info, chunks = exchange.getbundlechunks(repo, 'serve',
                                                **pycompat.strkwargs(opts))
    except error.Abort as exc:
        # cleanly forward Abort error to the client
        if not exchange.bundle2requested(opts.get('bundlecaps')):
            if proto.name == 'http':
                return ooberror(str(exc) + '\n')

        if exc.hint is not None:
            advargs.append(('hint', exc.hint))
        bundler.addpart(bundle2.bundlepart('error:abort',
                                           manargs, advargs))
        chunks = bundler.getchunks()
        preferuncompressed = True

    return streamres(gen=chunks, prefer_uncompressed=preferuncompressed)

@wireprotocommand('heads')
def heads(repo, proto):
    h = repo.heads()
    return encodelist(h) + "\n"

35787:a84dbc87dae9

            if proto.name == 'http':
                return ooberror(bundle2required)
            raise error.Abort(bundle2requiredmain,
                              hint=bundle2requiredhint)

    prefercompressed = True

    try:
        if repo.ui.configbool('server', 'disablefullbundle'):
            # Check to see if this is a full clone.
            clheads = set(repo.changelog.heads())

                    _('server has pull-based clones disabled'),
                    hint=_('remove --pull if specified or upgrade Mercurial'))

        info, chunks = exchange.getbundlechunks(repo, 'serve',
                                                **pycompat.strkwargs(opts))
        prefercompressed = info.get('prefercompressed', True)
    except error.Abort as exc:
        # cleanly forward Abort error to the client
        if not exchange.bundle2requested(opts.get('bundlecaps')):
            if proto.name == 'http':
                return ooberror(str(exc) + '\n')

        if exc.hint is not None:
            advargs.append(('hint', exc.hint))
        bundler.addpart(bundle2.bundlepart('error:abort',
                                           manargs, advargs))
        chunks = bundler.getchunks()
        prefercompressed = False

    return streamres(gen=chunks, prefer_uncompressed=not prefercompressed)

@wireprotocommand('heads')
def heads(repo, proto):
    h = repo.heads()
    return encodelist(h) + "\n"