changeset 35793:4fb2bb61597c

bundle2: increase payload part chunk size to 32kb Bundle2 payload parts are framed chunks. Esentially, we obtain data in equal size chunks of size `preferedchunksize` and emit those to a generator. That generator is fed into a compressor (which can be the no-op compressor, which just re-emits the generator). And the output from the compressor likely goes to a file descriptor or socket. What this means is that small chunk sizes create more Python objects and Python function calls than larger chunk sizes. And as we know, Python object and function call overhead in performance sensitive code matters (at least with CPython). This commit increases the bundle2 part payload chunk size from 4k to 32k. Practically speaking, this means that the chunks we feed into a compressor (implemented in C code) or feed directly into a file handle or socket write() are larger. It's possible the chunks might be larger than what the receiver can handle in one logical operation. But at that point, we're in C code, which is much more efficient at dealing with splitting up the chunk and making multiple function calls than Python is. A downside to larger chunks is that the receiver has to wait for that much data to arrive (either raw or from a decompressor) before it can process the chunk. But 32kb still feels like a small buffer to have to wait for. And in many cases, the client will convert from 8 read(4096) to 1 read(32768). That's happening in Python land. So we cut down on the number of Python objects and function calls, making the client faster as well. I don't think there are any significant concerns to increasing the payload chunk size to 32kb. The impact of this change on performance significant. Using `curl` to obtain a stream clone bundle2 payload from a server on localhost serving the mozilla-unified repository: before: 20.78 user; 7.71 system; 80.5 MB/s after: 13.90 user; 3.51 system; 132 MB/s legacy: 9.72 user; 8.16 system; 132 MB/s bundle2 stream clone generation is still more resource intensive than legacy stream clone (that's likely because of the use of a util.chunkbuffer). But the throughput is the same. We might be in territory we're this is effectively a benchmark of the networking stack or Python's syscall throughput. From the client perspective, `hg clone -U --stream`: before: 33.50 user; 7.95 system; 53.3 MB/s after: 22.82 user; 7.33 system; 72.7 MB/s legacy: 29.96 user; 7.94 system; 58.0 MB/s And for `hg clone --stream` with a working directory update of ~230k files: after: 119.55 user; 26.47 system; 0:57.08 wall legacy: 126.98 user; 26.94 system; 1:05.56 wall So, it appears that bundle2's stream clone is now definitively faster than legacy stream clone! Differential Revision: https://phab.mercurial-scm.org/D1932
author Gregory Szorc <gregory.szorc@gmail.com>
date Sat, 20 Jan 2018 22:55:42 -0800
parents 1d118f9f4f57
children 27b6df1b5adb 34287c2e3fc4
files mercurial/bundle2.py tests/test-clone-uncompressed.t
diffstat 2 files changed, 3 insertions(+), 3 deletions(-) [+]
line wrap: on
line diff
--- a/mercurial/bundle2.py	Mon Jan 22 12:23:47 2018 -0800
+++ b/mercurial/bundle2.py	Sat Jan 20 22:55:42 2018 -0800
@@ -183,7 +183,7 @@
 _fpayloadsize = '>i'
 _fpartparamcount = '>BB'
 
-preferedchunksize = 4096
+preferedchunksize = 32768
 
 _parttypeforbidden = re.compile('[^a-zA-Z0-9_:-]')
 
--- a/tests/test-clone-uncompressed.t	Mon Jan 22 12:23:47 2018 -0800
+++ b/tests/test-clone-uncompressed.t	Sat Jan 20 22:55:42 2018 -0800
@@ -199,7 +199,7 @@
   
 
   $ f --size --hex --bytes 256 body
-  body: size=112318
+  body: size=112222
   0000: 04 6e 6f 6e 65 48 47 32 30 00 00 00 00 00 00 00 |.noneHG20.......|
   0010: 68 07 53 54 52 45 41 4d 32 00 00 00 00 03 00 09 |h.STREAM2.......|
   0020: 05 09 04 0c 2d 62 79 74 65 63 6f 75 6e 74 39 38 |....-bytecount98|
@@ -207,7 +207,7 @@
   0040: 72 65 71 75 69 72 65 6d 65 6e 74 73 64 6f 74 65 |requirementsdote|
   0050: 6e 63 6f 64 65 20 66 6e 63 61 63 68 65 20 67 65 |ncode fncache ge|
   0060: 6e 65 72 61 6c 64 65 6c 74 61 20 72 65 76 6c 6f |neraldelta revlo|
-  0070: 67 76 31 20 73 74 6f 72 65 00 00 10 00 73 08 42 |gv1 store....s.B|
+  0070: 67 76 31 20 73 74 6f 72 65 00 00 80 00 73 08 42 |gv1 store....s.B|
   0080: 64 61 74 61 2f 30 2e 69 00 03 00 01 00 00 00 00 |data/0.i........|
   0090: 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 01 |................|
   00a0: ff ff ff ff ff ff ff ff 80 29 63 a0 49 d3 23 87 |.........)c.I.#.|