wireprotoframing: buffer emitted data to reduce frame count
An upcoming commit introduces a wire protocol command that can emit
hundreds of thousands of small objects. Without a buffering layer,
we would emit a single, small frame for every object. Performance
profiling revealed this to be a source of significant overhead for
both client and server.
This commit introduces a very crude buffering layer so that we emit
fewer, bigger frames in such a scenario. This code will likely get
rewritten in the future to be part of the streams API, as we'll
need a similar strategy for compressing data. I don't want to think
about it too much at the moment though.
server
before: user 32.500+0.000 sys 1.160+0.000
after: user 20.230+0.010 sys 0.180+0.000
client
before: user 133.400+0.000 sys 93.120+0.000
after: user 68.370+0.000 sys 32.950+0.000
This appears to indicate we have significant overhead in the frame
processing code on both client and server. It might be worth profiling
that at some point...
Differential Revision: https://phab.mercurial-scm.org/D4473
#!/usr/bin/env python
# Undump a dump from dumprevlog
# $ hg init
# $ undumprevlog < repo.dump
from __future__ import absolute_import, print_function
import sys
from mercurial import (
node,
revlog,
transaction,
vfs as vfsmod,
)
from mercurial.utils import (
procutil,
)
for fp in (sys.stdin, sys.stdout, sys.stderr):
procutil.setbinary(fp)
opener = vfsmod.vfs('.', False)
tr = transaction.transaction(sys.stderr.write, opener, {'store': opener},
"undump.journal")
while True:
l = sys.stdin.readline()
if not l:
break
if l.startswith("file:"):
f = l[6:-1]
r = revlog.revlog(opener, f)
print(f)
elif l.startswith("node:"):
n = node.bin(l[6:-1])
elif l.startswith("linkrev:"):
lr = int(l[9:-1])
elif l.startswith("parents:"):
p = l[9:-1].split()
p1 = node.bin(p[0])
p2 = node.bin(p[1])
elif l.startswith("length:"):
length = int(l[8:-1])
sys.stdin.readline() # start marker
d = sys.stdin.read(length)
sys.stdin.readline() # end marker
r.addrevision(d, tr, lr, p1, p2)
tr.close()