Mercurial > hg
annotate mercurial/httpclient/_readers.py @ 19037:1fde25ad9396
http2: make read use pushchunk/popchunk, eschew itertools
The itertools approach was showing up high in the profile output.
author | Brendan Cully <brendan@kublai.com> |
---|---|
date | Fri, 01 Feb 2013 14:41:35 -0800 |
parents | 19d1cc30e7a3 |
children | 36733ab7fa05 |
rev | line source |
---|---|
16643
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
1 # Copyright 2011, Google Inc. |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
2 # All rights reserved. |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
3 # |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
4 # Redistribution and use in source and binary forms, with or without |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
5 # modification, are permitted provided that the following conditions are |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
6 # met: |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
7 # |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
8 # * Redistributions of source code must retain the above copyright |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
9 # notice, this list of conditions and the following disclaimer. |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
10 # * Redistributions in binary form must reproduce the above |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
11 # copyright notice, this list of conditions and the following disclaimer |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
12 # in the documentation and/or other materials provided with the |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
13 # distribution. |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
14 # * Neither the name of Google Inc. nor the names of its |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
15 # contributors may be used to endorse or promote products derived from |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
16 # this software without specific prior written permission. |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
17 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
18 # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
19 # "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
20 # LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
21 # A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
22 # OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
23 # SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
24 # LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
25 # DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
26 # THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
27 # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
28 # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
29 """Reader objects to abstract out different body response types. |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
30 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
31 This module is package-private. It is not expected that these will |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
32 have any clients outside of httpplus. |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
33 """ |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
34 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
35 import httplib |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
36 import itertools |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
37 import logging |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
38 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
39 logger = logging.getLogger(__name__) |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
40 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
41 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
42 class ReadNotReady(Exception): |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
43 """Raised when read() is attempted but not enough data is loaded.""" |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
44 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
45 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
46 class HTTPRemoteClosedError(httplib.HTTPException): |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
47 """The server closed the remote socket in the middle of a response.""" |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
48 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
49 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
50 class AbstractReader(object): |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
51 """Abstract base class for response readers. |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
52 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
53 Subclasses must implement _load, and should implement _close if |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
54 it's not an error for the server to close their socket without |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
55 some termination condition being detected during _load. |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
56 """ |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
57 def __init__(self): |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
58 self._finished = False |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
59 self._done_chunks = [] |
19036
19d1cc30e7a3
http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents:
17424
diff
changeset
|
60 self.available_data = 0 |
16643
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
61 |
19036
19d1cc30e7a3
http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents:
17424
diff
changeset
|
62 def addchunk(self, data): |
19d1cc30e7a3
http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents:
17424
diff
changeset
|
63 self._done_chunks.append(data) |
19d1cc30e7a3
http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents:
17424
diff
changeset
|
64 self.available_data += len(data) |
19d1cc30e7a3
http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents:
17424
diff
changeset
|
65 |
19d1cc30e7a3
http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents:
17424
diff
changeset
|
66 def pushchunk(self, data): |
19d1cc30e7a3
http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents:
17424
diff
changeset
|
67 self._done_chunks.insert(0, data) |
19d1cc30e7a3
http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents:
17424
diff
changeset
|
68 self.available_data += len(data) |
19d1cc30e7a3
http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents:
17424
diff
changeset
|
69 |
19d1cc30e7a3
http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents:
17424
diff
changeset
|
70 def popchunk(self): |
19d1cc30e7a3
http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents:
17424
diff
changeset
|
71 b = self._done_chunks.pop(0) |
19d1cc30e7a3
http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents:
17424
diff
changeset
|
72 self.available_data -= len(b) |
19d1cc30e7a3
http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents:
17424
diff
changeset
|
73 |
19d1cc30e7a3
http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents:
17424
diff
changeset
|
74 return b |
16643
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
75 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
76 def done(self): |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
77 return self._finished |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
78 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
79 def read(self, amt): |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
80 if self.available_data < amt and not self._finished: |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
81 raise ReadNotReady() |
19037
1fde25ad9396
http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents:
19036
diff
changeset
|
82 blocks = [] |
1fde25ad9396
http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents:
19036
diff
changeset
|
83 need = amt |
1fde25ad9396
http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents:
19036
diff
changeset
|
84 while self._done_chunks: |
1fde25ad9396
http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents:
19036
diff
changeset
|
85 b = self.popchunk() |
1fde25ad9396
http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents:
19036
diff
changeset
|
86 if len(b) > need: |
1fde25ad9396
http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents:
19036
diff
changeset
|
87 nb = b[:need] |
1fde25ad9396
http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents:
19036
diff
changeset
|
88 self.pushchunk(b[need:]) |
1fde25ad9396
http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents:
19036
diff
changeset
|
89 b = nb |
1fde25ad9396
http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents:
19036
diff
changeset
|
90 blocks.append(b) |
1fde25ad9396
http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents:
19036
diff
changeset
|
91 need -= len(b) |
1fde25ad9396
http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents:
19036
diff
changeset
|
92 if need == 0: |
1fde25ad9396
http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents:
19036
diff
changeset
|
93 break |
16643
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
94 result = ''.join(blocks) |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
95 assert len(result) == amt or (self._finished and len(result) < amt) |
19037
1fde25ad9396
http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents:
19036
diff
changeset
|
96 |
16643
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
97 return result |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
98 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
99 def _load(self, data): # pragma: no cover |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
100 """Subclasses must implement this. |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
101 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
102 As data is available to be read out of this object, it should |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
103 be placed into the _done_chunks list. Subclasses should not |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
104 rely on data remaining in _done_chunks forever, as it may be |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
105 reaped if the client is parsing data as it comes in. |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
106 """ |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
107 raise NotImplementedError |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
108 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
109 def _close(self): |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
110 """Default implementation of close. |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
111 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
112 The default implementation assumes that the reader will mark |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
113 the response as finished on the _finished attribute once the |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
114 entire response body has been read. In the event that this is |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
115 not true, the subclass should override the implementation of |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
116 close (for example, close-is-end responses have to set |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
117 self._finished in the close handler.) |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
118 """ |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
119 if not self._finished: |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
120 raise HTTPRemoteClosedError( |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
121 'server appears to have closed the socket mid-response') |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
122 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
123 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
124 class AbstractSimpleReader(AbstractReader): |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
125 """Abstract base class for simple readers that require no response decoding. |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
126 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
127 Examples of such responses are Connection: Close (close-is-end) |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
128 and responses that specify a content length. |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
129 """ |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
130 def _load(self, data): |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
131 if data: |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
132 assert not self._finished, ( |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
133 'tried to add data (%r) to a closed reader!' % data) |
17424
e7cfe3587ea4
fix trivial spelling errors
Mads Kiilerich <mads@kiilerich.com>
parents:
16643
diff
changeset
|
134 logger.debug('%s read an additional %d data', self.name, len(data)) |
19036
19d1cc30e7a3
http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents:
17424
diff
changeset
|
135 self.addchunk(data) |
16643
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
136 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
137 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
138 class CloseIsEndReader(AbstractSimpleReader): |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
139 """Reader for responses that specify Connection: Close for length.""" |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
140 name = 'close-is-end' |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
141 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
142 def _close(self): |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
143 logger.info('Marking close-is-end reader as closed.') |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
144 self._finished = True |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
145 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
146 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
147 class ContentLengthReader(AbstractSimpleReader): |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
148 """Reader for responses that specify an exact content length.""" |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
149 name = 'content-length' |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
150 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
151 def __init__(self, amount): |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
152 AbstractReader.__init__(self) |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
153 self._amount = amount |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
154 if amount == 0: |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
155 self._finished = True |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
156 self._amount_seen = 0 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
157 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
158 def _load(self, data): |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
159 AbstractSimpleReader._load(self, data) |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
160 self._amount_seen += len(data) |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
161 if self._amount_seen >= self._amount: |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
162 self._finished = True |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
163 logger.debug('content-length read complete') |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
164 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
165 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
166 class ChunkedReader(AbstractReader): |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
167 """Reader for chunked transfer encoding responses.""" |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
168 def __init__(self, eol): |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
169 AbstractReader.__init__(self) |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
170 self._eol = eol |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
171 self._leftover_skip_amt = 0 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
172 self._leftover_data = '' |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
173 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
174 def _load(self, data): |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
175 assert not self._finished, 'tried to add data to a closed reader!' |
17424
e7cfe3587ea4
fix trivial spelling errors
Mads Kiilerich <mads@kiilerich.com>
parents:
16643
diff
changeset
|
176 logger.debug('chunked read an additional %d data', len(data)) |
16643
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
177 position = 0 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
178 if self._leftover_data: |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
179 logger.debug('chunked reader trying to finish block from leftover data') |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
180 # TODO: avoid this string concatenation if possible |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
181 data = self._leftover_data + data |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
182 position = self._leftover_skip_amt |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
183 self._leftover_data = '' |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
184 self._leftover_skip_amt = 0 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
185 datalen = len(data) |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
186 while position < datalen: |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
187 split = data.find(self._eol, position) |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
188 if split == -1: |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
189 self._leftover_data = data |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
190 self._leftover_skip_amt = position |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
191 return |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
192 amt = int(data[position:split], base=16) |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
193 block_start = split + len(self._eol) |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
194 # If the whole data chunk plus the eol trailer hasn't |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
195 # loaded, we'll wait for the next load. |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
196 if block_start + amt + len(self._eol) > len(data): |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
197 self._leftover_data = data |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
198 self._leftover_skip_amt = position |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
199 return |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
200 if amt == 0: |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
201 self._finished = True |
17424
e7cfe3587ea4
fix trivial spelling errors
Mads Kiilerich <mads@kiilerich.com>
parents:
16643
diff
changeset
|
202 logger.debug('closing chunked reader due to chunk of length 0') |
16643
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
203 return |
19036
19d1cc30e7a3
http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents:
17424
diff
changeset
|
204 self.addchunk(data[block_start:block_start + amt]) |
16643
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
205 position = block_start + amt + len(self._eol) |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
206 # no-check-code |