cborutil: change buffering strategy
Profiling revealed that we were spending a lot of time on the
line that was concatenating the old buffer with the incoming data
when attempting to decode long byte strings, such as manifest
revisions.
Essentially, we were feeding N chunks of size len(X) << len(Y) into
decode() and continuously allocating a new, larger buffer to hold
the undecoded input. This created substantial memory churn and
slowed down execution.
Changing the code to aggregate pending chunks in a list until we
have enough data to fully decode the next atom makes things much
more efficient.
I don't have exact data, but I recall the old code spending >1s
on manifest fulltexts from the mozilla-unified repo. The new code
doesn't significantly appear in profile output.
Differential Revision: https://phab.mercurial-scm.org/D4854
[This file is here for historical purposes, all recent contributors
should appear in the changelog directly]
Andrea Arcangeli <andrea at suse.de>
Thomas Arendsen Hein <thomas at intevation.de>
Goffredo Baroncelli <kreijack at libero.it>
Muli Ben-Yehuda <mulix at mulix.org>
Mikael Berthe <mikael at lilotux.net>
Benoit Boissinot <bboissin at gmail.com>
Brendan Cully <brendan at kublai.com>
Vincent Danjean <vdanjean.ml at free.fr>
Jake Edge <jake at edge2.net>
Michael Fetterman <michael.fetterman at intel.com>
Edouard Gomez <ed.gomez at free.fr>
Eric Hopper <hopper at omnifarious.org>
Alecs King <alecsk at gmail.com>
Volker Kleinfeld <Volker.Kleinfeld at gmx.de>
Vadim Lebedev <vadim at mbdsys.com>
Christopher Li <hg at chrisli.org>
Chris Mason <mason at suse.com>
Colin McMillen <mcmillen at cs.cmu.edu>
Wojciech Milkowski <wmilkowski at interia.pl>
Chad Netzer <chad.netzer at gmail.com>
Bryan O'Sullivan <bos at serpentine.com>
Vicent Seguí Pascual <vseguip at gmail.com>
Sean Perry <shaleh at speakeasy.net>
Nguyen Anh Quynh <aquynh at gmail.com>
Ollivier Robert <roberto at keltia.freenix.fr>
Alexander Schremmer <alex at alexanderweb.de>
Arun Sharma <arun at sharma-home.net>
Josef "Jeff" Sipek <jeffpc at optonline.net>
Kevin Smith <yarcs at qualitycode.com>
TK Soh <teekaysoh at yahoo.com>
Radoslaw Szkodzinski <astralstorm at gorzow.mm.pl>
Samuel Tardieu <sam at rfc1149.net>
K Thananchayan <thananck at yahoo.com>
Andrew Thompson <andrewkt at aktzero.com>
Michael S. Tsirkin <mst at mellanox.co.il>
Rafael Villar Burke <pachi at mmn-arquitectos.com>
Tristan Wibberley <tristan at wibberley.org>
Mark Williamson <mark.williamson at cl.cam.ac.uk>