wireproto: compress data from a generator
Currently, the "getbundle" wire protocol command obtains a generator of
data, converts it to a util.chunkbuffer, then converts it back to a
generator via the protocol's groupchunks() implementation. For the SSH
protocol, groupchunks() simply reads 4kb chunks then write()s the
data to a file descriptor. For the HTTP protocol, groupchunks() reads
32kb chunks, feeds those into a zlib compressor, emits compressed data
as it is available, and that is sent to the WSGI layer, where it is
likely turned into HTTP chunked transfer chunks as is or further
buffered and turned into a larger chunk.
For both the SSH and HTTP protocols, there is inefficiency from using
util.chunkbuffer.
For SSH, emitting consistent 4kb chunks sounds nice. However, the file
descriptor it is writing to is almost certainly buffered. That means
that a Python .write() probably doesn't translate into exactly what is
written to the I/O layer.
For HTTP, we're going through an intermediate layer to zlib compress
data. So all util.chunkbuffer is doing is ensuring that the chunks we
feed into the zlib compressor are of uniform size. This means more CPU
time in Python buffering and emitting chunks in util.chunkbuffer but
fewer function calls to zlib.
This patch introduces and implements a new wire protocol abstract
method: compresschunks(). It is like groupchunks() except it operates
on a generator instead of something with a .read(). The SSH
implementation simply proxies chunks. The HTTP implementation uses
zlib compression.
To avoid duplicate code, the HTTP groupchunks() has been reimplemented
in terms of compresschunks().
To prove this all works, the "getbundle" wire protocol command has been
switched to compresschunks(). This removes the util.chunkbuffer from
that command. Now, data essentially streams straight from the
changegroup emitter to the wire, possibly through a zlib compressor.
Generators all the way, baby.
There were slim to no performance changes on the server as measured
with the mozilla-central repository. This is likely because CPU
time is dominated by reading revlogs, producing the changegroup, and
zlib compressing the output stream. Still, this brings us a little
closer to our ideal of using generators everywhere.
Check that renames are correctly saved by a commit after a merge
Test with the merge on 3 having the rename on the local parent
$ hg init a
$ cd a
$ echo line1 > foo
$ hg add foo
$ hg ci -m '0: add foo'
$ echo line2 >> foo
$ hg ci -m '1: change foo'
$ hg up -C 0
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ hg mv foo bar
$ rm bar
$ echo line0 > bar
$ echo line1 >> bar
$ hg ci -m '2: mv foo bar; change bar'
created new head
$ hg merge 1
merging bar and foo to bar
0 files updated, 1 files merged, 0 files removed, 0 files unresolved
(branch merge, don't forget to commit)
$ cat bar
line0
line1
line2
$ hg ci -m '3: merge with local rename'
$ hg debugindex bar
rev offset length ..... linkrev nodeid p1 p2 (re)
0 0 77 ..... 2 d35118874825 000000000000 000000000000 (re)
1 77 76 ..... 3 5345f5ab8abd 000000000000 d35118874825 (re)
$ hg debugrename bar
bar renamed from foo:9e25c27b87571a1edee5ae4dddee5687746cc8e2
$ hg debugindex foo
rev offset length ..... linkrev nodeid p1 p2 (re)
0 0 7 ..... 0 690b295714ae 000000000000 000000000000 (re)
1 7 13 ..... 1 9e25c27b8757 690b295714ae 000000000000 (re)
Revert the content change from rev 2:
$ hg up -C 2
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ rm bar
$ echo line1 > bar
$ hg ci -m '4: revert content change from rev 2'
created new head
$ hg log --template '{rev}:{node|short} {parents}\n'
4:2263c1be0967 2:0f2ff26688b9
3:0555950ead28 2:0f2ff26688b9 1:5cd961e4045d
2:0f2ff26688b9 0:2665aaee66e9
1:5cd961e4045d
0:2665aaee66e9
This should use bar@rev2 as the ancestor:
$ hg --debug merge 3
searching for copies back to rev 1
resolving manifests
branchmerge: True, force: False, partial: False
ancestor: 0f2ff26688b9, local: 2263c1be0967+, remote: 0555950ead28
preserving bar for resolve of bar
starting 4 threads for background file closing (?)
bar: versions differ -> m (premerge)
picked tool ':merge' for bar (binary False symlink False changedelete False)
merging bar
my bar@2263c1be0967+ other bar@0555950ead28 ancestor bar@0f2ff26688b9
premerge successful
0 files updated, 1 files merged, 0 files removed, 0 files unresolved
(branch merge, don't forget to commit)
$ cat bar
line1
line2
$ hg ci -m '5: merge'
$ hg debugindex bar
rev offset length ..... linkrev nodeid p1 p2 (re)
0 0 77 ..... 2 d35118874825 000000000000 000000000000 (re)
1 77 76 ..... 3 5345f5ab8abd 000000000000 d35118874825 (re)
2 153 7 ..... 4 ff4b45017382 d35118874825 000000000000 (re)
3 160 13 ..... 5 3701b4893544 ff4b45017382 5345f5ab8abd (re)
Same thing, but with the merge on 3 having the rename
on the remote parent:
$ cd ..
$ hg clone -U -r 1 -r 2 a b
adding changesets
adding manifests
adding file changes
added 3 changesets with 3 changes to 2 files (+1 heads)
$ cd b
$ hg up -C 1
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ hg merge 2
merging foo and bar to bar
0 files updated, 1 files merged, 0 files removed, 0 files unresolved
(branch merge, don't forget to commit)
$ cat bar
line0
line1
line2
$ hg ci -m '3: merge with remote rename'
$ hg debugindex bar
rev offset length ..... linkrev nodeid p1 p2 (re)
0 0 77 ..... 2 d35118874825 000000000000 000000000000 (re)
1 77 76 ..... 3 5345f5ab8abd 000000000000 d35118874825 (re)
$ hg debugrename bar
bar renamed from foo:9e25c27b87571a1edee5ae4dddee5687746cc8e2
$ hg debugindex foo
rev offset length ..... linkrev nodeid p1 p2 (re)
0 0 7 ..... 0 690b295714ae 000000000000 000000000000 (re)
1 7 13 ..... 1 9e25c27b8757 690b295714ae 000000000000 (re)
Revert the content change from rev 2:
$ hg up -C 2
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ rm bar
$ echo line1 > bar
$ hg ci -m '4: revert content change from rev 2'
created new head
$ hg log --template '{rev}:{node|short} {parents}\n'
4:2263c1be0967 2:0f2ff26688b9
3:3ffa6b9e35f0 1:5cd961e4045d 2:0f2ff26688b9
2:0f2ff26688b9 0:2665aaee66e9
1:5cd961e4045d
0:2665aaee66e9
This should use bar@rev2 as the ancestor:
$ hg --debug merge 3
searching for copies back to rev 1
resolving manifests
branchmerge: True, force: False, partial: False
ancestor: 0f2ff26688b9, local: 2263c1be0967+, remote: 3ffa6b9e35f0
preserving bar for resolve of bar
starting 4 threads for background file closing (?)
bar: versions differ -> m (premerge)
picked tool ':merge' for bar (binary False symlink False changedelete False)
merging bar
my bar@2263c1be0967+ other bar@3ffa6b9e35f0 ancestor bar@0f2ff26688b9
premerge successful
0 files updated, 1 files merged, 0 files removed, 0 files unresolved
(branch merge, don't forget to commit)
$ cat bar
line1
line2
$ hg ci -m '5: merge'
$ hg debugindex bar
rev offset length ..... linkrev nodeid p1 p2 (re)
0 0 77 ..... 2 d35118874825 000000000000 000000000000 (re)
1 77 76 ..... 3 5345f5ab8abd 000000000000 d35118874825 (re)
2 153 7 ..... 4 ff4b45017382 d35118874825 000000000000 (re)
3 160 13 ..... 5 3701b4893544 ff4b45017382 5345f5ab8abd (re)
$ cd ..