Mercurial > hg
annotate setup_osutil_cffi.py @ 30206:d105195436c0
wireproto: compress data from a generator
Currently, the "getbundle" wire protocol command obtains a generator of
data, converts it to a util.chunkbuffer, then converts it back to a
generator via the protocol's groupchunks() implementation. For the SSH
protocol, groupchunks() simply reads 4kb chunks then write()s the
data to a file descriptor. For the HTTP protocol, groupchunks() reads
32kb chunks, feeds those into a zlib compressor, emits compressed data
as it is available, and that is sent to the WSGI layer, where it is
likely turned into HTTP chunked transfer chunks as is or further
buffered and turned into a larger chunk.
For both the SSH and HTTP protocols, there is inefficiency from using
util.chunkbuffer.
For SSH, emitting consistent 4kb chunks sounds nice. However, the file
descriptor it is writing to is almost certainly buffered. That means
that a Python .write() probably doesn't translate into exactly what is
written to the I/O layer.
For HTTP, we're going through an intermediate layer to zlib compress
data. So all util.chunkbuffer is doing is ensuring that the chunks we
feed into the zlib compressor are of uniform size. This means more CPU
time in Python buffering and emitting chunks in util.chunkbuffer but
fewer function calls to zlib.
This patch introduces and implements a new wire protocol abstract
method: compresschunks(). It is like groupchunks() except it operates
on a generator instead of something with a .read(). The SSH
implementation simply proxies chunks. The HTTP implementation uses
zlib compression.
To avoid duplicate code, the HTTP groupchunks() has been reimplemented
in terms of compresschunks().
To prove this all works, the "getbundle" wire protocol command has been
switched to compresschunks(). This removes the util.chunkbuffer from
that command. Now, data essentially streams straight from the
changegroup emitter to the wire, possibly through a zlib compressor.
Generators all the way, baby.
There were slim to no performance changes on the server as measured
with the mozilla-central repository. This is likely because CPU
time is dominated by reading revlogs, producing the changegroup, and
zlib compressing the output stream. Still, this brings us a little
closer to our ideal of using generators everywhere.
author | Gregory Szorc <gregory.szorc@gmail.com> |
---|---|
date | Sun, 16 Oct 2016 11:10:21 -0700 |
parents | a043c6d372db |
children |
rev | line source |
---|---|
29600
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
1 from __future__ import absolute_import |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
2 |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
3 import cffi |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
4 |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
5 ffi = cffi.FFI() |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
6 ffi.set_source("_osutil_cffi", """ |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
7 #include <sys/attr.h> |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
8 #include <sys/vnode.h> |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
9 #include <unistd.h> |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
10 #include <fcntl.h> |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
11 #include <time.h> |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
12 |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
13 typedef struct val_attrs { |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
14 uint32_t length; |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
15 attribute_set_t returned; |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
16 attrreference_t name_info; |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
17 fsobj_type_t obj_type; |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
18 struct timespec mtime; |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
19 uint32_t accessmask; |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
20 off_t datalength; |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
21 } __attribute__((aligned(4), packed)) val_attrs_t; |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
22 """, include_dirs=['mercurial']) |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
23 ffi.cdef(''' |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
24 |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
25 typedef uint32_t attrgroup_t; |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
26 |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
27 typedef struct attrlist { |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
28 uint16_t bitmapcount; /* number of attr. bit sets in list */ |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
29 uint16_t reserved; /* (to maintain 4-byte alignment) */ |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
30 attrgroup_t commonattr; /* common attribute group */ |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
31 attrgroup_t volattr; /* volume attribute group */ |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
32 attrgroup_t dirattr; /* directory attribute group */ |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
33 attrgroup_t fileattr; /* file attribute group */ |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
34 attrgroup_t forkattr; /* fork attribute group */ |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
35 ...; |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
36 }; |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
37 |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
38 typedef struct attribute_set { |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
39 ...; |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
40 } attribute_set_t; |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
41 |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
42 typedef struct attrreference { |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
43 int attr_dataoffset; |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
44 int attr_length; |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
45 ...; |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
46 } attrreference_t; |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
47 |
29910
a043c6d372db
osutil: fix the declaration to work on different sizes of off_t
Maciej Fijalkowski <fijall@gmail.com>
parents:
29600
diff
changeset
|
48 typedef int ... off_t; |
a043c6d372db
osutil: fix the declaration to work on different sizes of off_t
Maciej Fijalkowski <fijall@gmail.com>
parents:
29600
diff
changeset
|
49 |
29600
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
50 typedef struct val_attrs { |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
51 uint32_t length; |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
52 attribute_set_t returned; |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
53 attrreference_t name_info; |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
54 uint32_t obj_type; |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
55 struct timespec mtime; |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
56 uint32_t accessmask; |
29910
a043c6d372db
osutil: fix the declaration to work on different sizes of off_t
Maciej Fijalkowski <fijall@gmail.com>
parents:
29600
diff
changeset
|
57 off_t datalength; |
29600
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
58 ...; |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
59 } val_attrs_t; |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
60 |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
61 /* the exact layout of the above struct will be figured out during build time */ |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
62 |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
63 typedef int ... time_t; |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
64 |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
65 typedef struct timespec { |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
66 time_t tv_sec; |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
67 ...; |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
68 }; |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
69 |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
70 int getattrlist(const char* path, struct attrlist * attrList, void * attrBuf, |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
71 size_t attrBufSize, unsigned int options); |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
72 |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
73 int getattrlistbulk(int dirfd, struct attrlist * attrList, void * attrBuf, |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
74 size_t attrBufSize, uint64_t options); |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
75 |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
76 #define ATTR_BIT_MAP_COUNT ... |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
77 #define ATTR_CMN_NAME ... |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
78 #define ATTR_CMN_OBJTYPE ... |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
79 #define ATTR_CMN_MODTIME ... |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
80 #define ATTR_CMN_ACCESSMASK ... |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
81 #define ATTR_CMN_ERROR ... |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
82 #define ATTR_CMN_RETURNED_ATTRS ... |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
83 #define ATTR_FILE_DATALENGTH ... |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
84 |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
85 #define VREG ... |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
86 #define VDIR ... |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
87 #define VLNK ... |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
88 #define VBLK ... |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
89 #define VCHR ... |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
90 #define VFIFO ... |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
91 #define VSOCK ... |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
92 |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
93 #define S_IFMT ... |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
94 |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
95 int open(const char *path, int oflag, int perm); |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
96 int close(int); |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
97 |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
98 #define O_RDONLY ... |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
99 ''') |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
100 |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
101 if __name__ == '__main__': |
7a157639b8f2
osutil: add darwin-only version of os.listdir using cffi
Maciej Fijalkowski <fijall@gmail.com>
parents:
diff
changeset
|
102 ffi.compile() |