wireprotoframing: buffer emitted data to reduce frame count
An upcoming commit introduces a wire protocol command that can emit
hundreds of thousands of small objects. Without a buffering layer,
we would emit a single, small frame for every object. Performance
profiling revealed this to be a source of significant overhead for
both client and server.
This commit introduces a very crude buffering layer so that we emit
fewer, bigger frames in such a scenario. This code will likely get
rewritten in the future to be part of the streams API, as we'll
need a similar strategy for compressing data. I don't want to think
about it too much at the moment though.
server
before: user 32.500+0.000 sys 1.160+0.000
after: user 20.230+0.010 sys 0.180+0.000
client
before: user 133.400+0.000 sys 93.120+0.000
after: user 68.370+0.000 sys 32.950+0.000
This appears to indicate we have significant overhead in the frame
processing code on both client and server. It might be worth profiling
that at some point...
Differential Revision: https://phab.mercurial-scm.org/D4473
from __future__ import absolute_import
import __builtin__
import os
from mercurial import (
util,
)
def lowerwrap(scope, funcname):
f = getattr(scope, funcname)
def wrap(fname, *args, **kwargs):
d, base = os.path.split(fname)
try:
files = os.listdir(d or '.')
except OSError:
files = []
if base in files:
return f(fname, *args, **kwargs)
for fn in files:
if fn.lower() == base.lower():
return f(os.path.join(d, fn), *args, **kwargs)
return f(fname, *args, **kwargs)
scope.__dict__[funcname] = wrap
def normcase(path):
return path.lower()
os.path.normcase = normcase
for f in 'file open'.split():
lowerwrap(__builtin__, f)
for f in "chmod chown open lstat stat remove unlink".split():
lowerwrap(os, f)
for f in "exists lexists".split():
lowerwrap(os.path, f)
lowerwrap(util, 'posixfile')