view tests/test-custom-filters.t @ 39560:84bf6ded9317

wireprotoframing: buffer emitted data to reduce frame count

An upcoming commit introduces a wire protocol command that can emit
hundreds of thousands of small objects. Without a buffering layer,
we would emit a single, small frame for every object. Performance
profiling revealed this to be a source of significant overhead for
both client and server.

This commit introduces a very crude buffering layer so that we emit
fewer, bigger frames in such a scenario. This code will likely get
rewritten in the future to be part of the streams API, as we'll
need a similar strategy for compressing data. I don't want to think
about it too much at the moment though.

server
before: user 32.500+0.000 sys 1.160+0.000
after:  user 20.230+0.010 sys 0.180+0.000

client
before: user 133.400+0.000 sys 93.120+0.000
after:  user  68.370+0.000 sys 32.950+0.000

This appears to indicate we have significant overhead in the frame
processing code on both client and server. It might be worth profiling
that at some point...

Differential Revision: https://phab.mercurial-scm.org/D4473

  $ hg init

  $ cat > .hg/hgrc <<EOF
  > [extensions]
  > prefixfilter = prefix.py
  > [encode]
  > *.txt = stripprefix: Copyright 2046, The Masters
  > [decode]
  > *.txt = insertprefix: Copyright 2046, The Masters
  > EOF

  $ cat > prefix.py <<EOF
  > from mercurial import error
  > def stripprefix(s, cmd, filename, **kwargs):
  >     header = b'%s\n' % cmd
  >     if s[:len(header)] != header:
  >         raise error.Abort(b'missing header "%s" in %s' % (cmd, filename))
  >     return s[len(header):]
  > def insertprefix(s, cmd):
  >     return b'%s\n%s' % (cmd, s)
  > def reposetup(ui, repo):
  >     repo.adddatafilter(b'stripprefix:', stripprefix)
  >     repo.adddatafilter(b'insertprefix:', insertprefix)
  > EOF

  $ cat > .hgignore <<EOF
  > .hgignore
  > prefix.py
  > prefix.pyc
  > EOF

  $ cat > stuff.txt <<EOF
  > Copyright 2046, The Masters
  > Some stuff to ponder very carefully.
  > EOF
  $ hg add stuff.txt
  $ hg ci -m stuff

Repository data:

  $ hg cat stuff.txt
  Some stuff to ponder very carefully.

Fresh checkout:

  $ rm stuff.txt
  $ hg up -C
  1 files updated, 0 files merged, 0 files removed, 0 files unresolved
  $ cat stuff.txt
  Copyright 2046, The Masters
  Some stuff to ponder very carefully.
  $ echo "Very very carefully." >> stuff.txt
  $ hg stat
  M stuff.txt

  $ echo "Unauthorized material subject to destruction." > morestuff.txt

Problem encoding:

  $ hg add morestuff.txt
  $ hg ci -m morestuff
  abort: missing header "Copyright 2046, The Masters" in morestuff.txt
  [255]
  $ hg stat
  M stuff.txt
  A morestuff.txt