Mercurial > hg
annotate mercurial/httpclient/_readers.py @ 19038:36733ab7fa05
http2: sane readline
It turns out that it pays off to read more than a byte at a time with
a select in between :)
author | Brendan Cully <brendan@kublai.com> |
---|---|
date | Fri, 01 Feb 2013 15:00:23 -0800 |
parents | 1fde25ad9396 |
children | fae47ecaa952 |
rev | line source |
---|---|
16643
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
1 # Copyright 2011, Google Inc. |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
2 # All rights reserved. |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
3 # |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
4 # Redistribution and use in source and binary forms, with or without |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
5 # modification, are permitted provided that the following conditions are |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
6 # met: |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
7 # |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
8 # * Redistributions of source code must retain the above copyright |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
9 # notice, this list of conditions and the following disclaimer. |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
10 # * Redistributions in binary form must reproduce the above |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
11 # copyright notice, this list of conditions and the following disclaimer |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
12 # in the documentation and/or other materials provided with the |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
13 # distribution. |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
14 # * Neither the name of Google Inc. nor the names of its |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
15 # contributors may be used to endorse or promote products derived from |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
16 # this software without specific prior written permission. |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
17 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
18 # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
19 # "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
20 # LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
21 # A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
22 # OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
23 # SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
24 # LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
25 # DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
26 # THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
27 # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
28 # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
29 """Reader objects to abstract out different body response types. |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
30 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
31 This module is package-private. It is not expected that these will |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
32 have any clients outside of httpplus. |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
33 """ |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
34 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
35 import httplib |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
36 import itertools |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
37 import logging |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
38 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
39 logger = logging.getLogger(__name__) |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
40 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
41 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
42 class ReadNotReady(Exception): |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
43 """Raised when read() is attempted but not enough data is loaded.""" |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
44 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
45 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
46 class HTTPRemoteClosedError(httplib.HTTPException): |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
47 """The server closed the remote socket in the middle of a response.""" |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
48 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
49 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
50 class AbstractReader(object): |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
51 """Abstract base class for response readers. |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
52 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
53 Subclasses must implement _load, and should implement _close if |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
54 it's not an error for the server to close their socket without |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
55 some termination condition being detected during _load. |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
56 """ |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
57 def __init__(self): |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
58 self._finished = False |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
59 self._done_chunks = [] |
19036
19d1cc30e7a3
http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents:
17424
diff
changeset
|
60 self.available_data = 0 |
16643
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
61 |
19036
19d1cc30e7a3
http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents:
17424
diff
changeset
|
62 def addchunk(self, data): |
19d1cc30e7a3
http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents:
17424
diff
changeset
|
63 self._done_chunks.append(data) |
19d1cc30e7a3
http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents:
17424
diff
changeset
|
64 self.available_data += len(data) |
19d1cc30e7a3
http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents:
17424
diff
changeset
|
65 |
19d1cc30e7a3
http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents:
17424
diff
changeset
|
66 def pushchunk(self, data): |
19d1cc30e7a3
http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents:
17424
diff
changeset
|
67 self._done_chunks.insert(0, data) |
19d1cc30e7a3
http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents:
17424
diff
changeset
|
68 self.available_data += len(data) |
19d1cc30e7a3
http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents:
17424
diff
changeset
|
69 |
19d1cc30e7a3
http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents:
17424
diff
changeset
|
70 def popchunk(self): |
19d1cc30e7a3
http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents:
17424
diff
changeset
|
71 b = self._done_chunks.pop(0) |
19d1cc30e7a3
http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents:
17424
diff
changeset
|
72 self.available_data -= len(b) |
19d1cc30e7a3
http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents:
17424
diff
changeset
|
73 |
19d1cc30e7a3
http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents:
17424
diff
changeset
|
74 return b |
16643
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
75 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
76 def done(self): |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
77 return self._finished |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
78 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
79 def read(self, amt): |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
80 if self.available_data < amt and not self._finished: |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
81 raise ReadNotReady() |
19037
1fde25ad9396
http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents:
19036
diff
changeset
|
82 blocks = [] |
1fde25ad9396
http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents:
19036
diff
changeset
|
83 need = amt |
1fde25ad9396
http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents:
19036
diff
changeset
|
84 while self._done_chunks: |
1fde25ad9396
http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents:
19036
diff
changeset
|
85 b = self.popchunk() |
1fde25ad9396
http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents:
19036
diff
changeset
|
86 if len(b) > need: |
1fde25ad9396
http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents:
19036
diff
changeset
|
87 nb = b[:need] |
1fde25ad9396
http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents:
19036
diff
changeset
|
88 self.pushchunk(b[need:]) |
1fde25ad9396
http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents:
19036
diff
changeset
|
89 b = nb |
1fde25ad9396
http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents:
19036
diff
changeset
|
90 blocks.append(b) |
1fde25ad9396
http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents:
19036
diff
changeset
|
91 need -= len(b) |
1fde25ad9396
http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents:
19036
diff
changeset
|
92 if need == 0: |
1fde25ad9396
http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents:
19036
diff
changeset
|
93 break |
16643
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
94 result = ''.join(blocks) |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
95 assert len(result) == amt or (self._finished and len(result) < amt) |
19037
1fde25ad9396
http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents:
19036
diff
changeset
|
96 |
16643
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
97 return result |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
98 |
19038 | 99 def readto(self, delimstr, blocks = None): |
100 """return available data chunks up to the first one in which delimstr | |
101 occurs. No data will be returned after delimstr -- the chunk in which | |
102 it occurs will be split and the remainder pushed back onto the available | |
103 data queue. If blocks is supplied chunks will be added to blocks, otherwise | |
104 a new list will be allocated. | |
105 """ | |
106 if blocks is None: | |
107 blocks = [] | |
108 | |
109 while self._done_chunks: | |
110 b = self.popchunk() | |
111 i = b.find(delimstr) + len(delimstr) | |
112 if i: | |
113 if i < len(b): | |
114 self.pushchunk(b[i:]) | |
115 blocks.append(b[:i]) | |
116 break | |
117 else: | |
118 blocks.append(b) | |
119 | |
120 return blocks | |
121 | |
16643
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
122 def _load(self, data): # pragma: no cover |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
123 """Subclasses must implement this. |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
124 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
125 As data is available to be read out of this object, it should |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
126 be placed into the _done_chunks list. Subclasses should not |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
127 rely on data remaining in _done_chunks forever, as it may be |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
128 reaped if the client is parsing data as it comes in. |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
129 """ |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
130 raise NotImplementedError |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
131 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
132 def _close(self): |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
133 """Default implementation of close. |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
134 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
135 The default implementation assumes that the reader will mark |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
136 the response as finished on the _finished attribute once the |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
137 entire response body has been read. In the event that this is |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
138 not true, the subclass should override the implementation of |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
139 close (for example, close-is-end responses have to set |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
140 self._finished in the close handler.) |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
141 """ |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
142 if not self._finished: |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
143 raise HTTPRemoteClosedError( |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
144 'server appears to have closed the socket mid-response') |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
145 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
146 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
147 class AbstractSimpleReader(AbstractReader): |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
148 """Abstract base class for simple readers that require no response decoding. |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
149 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
150 Examples of such responses are Connection: Close (close-is-end) |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
151 and responses that specify a content length. |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
152 """ |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
153 def _load(self, data): |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
154 if data: |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
155 assert not self._finished, ( |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
156 'tried to add data (%r) to a closed reader!' % data) |
17424
e7cfe3587ea4
fix trivial spelling errors
Mads Kiilerich <mads@kiilerich.com>
parents:
16643
diff
changeset
|
157 logger.debug('%s read an additional %d data', self.name, len(data)) |
19036
19d1cc30e7a3
http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents:
17424
diff
changeset
|
158 self.addchunk(data) |
16643
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
159 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
160 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
161 class CloseIsEndReader(AbstractSimpleReader): |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
162 """Reader for responses that specify Connection: Close for length.""" |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
163 name = 'close-is-end' |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
164 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
165 def _close(self): |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
166 logger.info('Marking close-is-end reader as closed.') |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
167 self._finished = True |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
168 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
169 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
170 class ContentLengthReader(AbstractSimpleReader): |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
171 """Reader for responses that specify an exact content length.""" |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
172 name = 'content-length' |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
173 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
174 def __init__(self, amount): |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
175 AbstractReader.__init__(self) |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
176 self._amount = amount |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
177 if amount == 0: |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
178 self._finished = True |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
179 self._amount_seen = 0 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
180 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
181 def _load(self, data): |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
182 AbstractSimpleReader._load(self, data) |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
183 self._amount_seen += len(data) |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
184 if self._amount_seen >= self._amount: |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
185 self._finished = True |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
186 logger.debug('content-length read complete') |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
187 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
188 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
189 class ChunkedReader(AbstractReader): |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
190 """Reader for chunked transfer encoding responses.""" |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
191 def __init__(self, eol): |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
192 AbstractReader.__init__(self) |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
193 self._eol = eol |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
194 self._leftover_skip_amt = 0 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
195 self._leftover_data = '' |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
196 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
197 def _load(self, data): |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
198 assert not self._finished, 'tried to add data to a closed reader!' |
17424
e7cfe3587ea4
fix trivial spelling errors
Mads Kiilerich <mads@kiilerich.com>
parents:
16643
diff
changeset
|
199 logger.debug('chunked read an additional %d data', len(data)) |
16643
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
200 position = 0 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
201 if self._leftover_data: |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
202 logger.debug('chunked reader trying to finish block from leftover data') |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
203 # TODO: avoid this string concatenation if possible |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
204 data = self._leftover_data + data |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
205 position = self._leftover_skip_amt |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
206 self._leftover_data = '' |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
207 self._leftover_skip_amt = 0 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
208 datalen = len(data) |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
209 while position < datalen: |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
210 split = data.find(self._eol, position) |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
211 if split == -1: |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
212 self._leftover_data = data |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
213 self._leftover_skip_amt = position |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
214 return |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
215 amt = int(data[position:split], base=16) |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
216 block_start = split + len(self._eol) |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
217 # If the whole data chunk plus the eol trailer hasn't |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
218 # loaded, we'll wait for the next load. |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
219 if block_start + amt + len(self._eol) > len(data): |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
220 self._leftover_data = data |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
221 self._leftover_skip_amt = position |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
222 return |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
223 if amt == 0: |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
224 self._finished = True |
17424
e7cfe3587ea4
fix trivial spelling errors
Mads Kiilerich <mads@kiilerich.com>
parents:
16643
diff
changeset
|
225 logger.debug('closing chunked reader due to chunk of length 0') |
16643
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
226 return |
19036
19d1cc30e7a3
http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents:
17424
diff
changeset
|
227 self.addchunk(data[block_start:block_start + amt]) |
16643
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
228 position = block_start + amt + len(self._eol) |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
229 # no-check-code |