Mercurial > hg
annotate mercurial/httpclient/_readers.py @ 28071:261324dd5f5b
merge: introduce method to minimize merge regions
In the next diff, we will use this to trim down the start and end of conflict
regions where the A and B sides both made the same changes.
author | Ryan McElroy <rmcelroy@fb.com> |
---|---|
date | Wed, 10 Feb 2016 08:25:03 -0800 |
parents | 1ad9da968a2e |
children | 8a66eda46c98 |
rev | line source |
---|---|
16643
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
1 # Copyright 2011, Google Inc. |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
2 # All rights reserved. |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
3 # |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
4 # Redistribution and use in source and binary forms, with or without |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
5 # modification, are permitted provided that the following conditions are |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
6 # met: |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
7 # |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
8 # * Redistributions of source code must retain the above copyright |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
9 # notice, this list of conditions and the following disclaimer. |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
10 # * Redistributions in binary form must reproduce the above |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
11 # copyright notice, this list of conditions and the following disclaimer |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
12 # in the documentation and/or other materials provided with the |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
13 # distribution. |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
14 # * Neither the name of Google Inc. nor the names of its |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
15 # contributors may be used to endorse or promote products derived from |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
16 # this software without specific prior written permission. |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
17 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
18 # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
19 # "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
20 # LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
21 # A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
22 # OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
23 # SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
24 # LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
25 # DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
26 # THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
27 # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
28 # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
29 """Reader objects to abstract out different body response types. |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
30 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
31 This module is package-private. It is not expected that these will |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
32 have any clients outside of httpplus. |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
33 """ |
27601
1ad9da968a2e
httpclient: update to 938f2107d6e2 of httpplus
Augie Fackler <augie@google.com>
parents:
19182
diff
changeset
|
34 from __future__ import absolute_import |
16643
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
35 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
36 import httplib |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
37 import logging |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
38 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
39 logger = logging.getLogger(__name__) |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
40 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
41 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
42 class ReadNotReady(Exception): |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
43 """Raised when read() is attempted but not enough data is loaded.""" |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
44 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
45 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
46 class HTTPRemoteClosedError(httplib.HTTPException): |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
47 """The server closed the remote socket in the middle of a response.""" |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
48 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
49 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
50 class AbstractReader(object): |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
51 """Abstract base class for response readers. |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
52 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
53 Subclasses must implement _load, and should implement _close if |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
54 it's not an error for the server to close their socket without |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
55 some termination condition being detected during _load. |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
56 """ |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
57 def __init__(self): |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
58 self._finished = False |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
59 self._done_chunks = [] |
19036
19d1cc30e7a3
http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents:
17424
diff
changeset
|
60 self.available_data = 0 |
16643
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
61 |
19182
fae47ecaa952
httpclient: upgrade to fe8c09e4db64 of httpplus
Augie Fackler <raf@durin42.com>
parents:
19038
diff
changeset
|
62 def _addchunk(self, data): |
19036
19d1cc30e7a3
http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents:
17424
diff
changeset
|
63 self._done_chunks.append(data) |
19d1cc30e7a3
http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents:
17424
diff
changeset
|
64 self.available_data += len(data) |
19d1cc30e7a3
http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents:
17424
diff
changeset
|
65 |
19182
fae47ecaa952
httpclient: upgrade to fe8c09e4db64 of httpplus
Augie Fackler <raf@durin42.com>
parents:
19038
diff
changeset
|
66 def _pushchunk(self, data): |
19036
19d1cc30e7a3
http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents:
17424
diff
changeset
|
67 self._done_chunks.insert(0, data) |
19d1cc30e7a3
http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents:
17424
diff
changeset
|
68 self.available_data += len(data) |
19d1cc30e7a3
http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents:
17424
diff
changeset
|
69 |
19182
fae47ecaa952
httpclient: upgrade to fe8c09e4db64 of httpplus
Augie Fackler <raf@durin42.com>
parents:
19038
diff
changeset
|
70 def _popchunk(self): |
19036
19d1cc30e7a3
http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents:
17424
diff
changeset
|
71 b = self._done_chunks.pop(0) |
19d1cc30e7a3
http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents:
17424
diff
changeset
|
72 self.available_data -= len(b) |
19d1cc30e7a3
http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents:
17424
diff
changeset
|
73 |
19d1cc30e7a3
http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents:
17424
diff
changeset
|
74 return b |
16643
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
75 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
76 def done(self): |
19182
fae47ecaa952
httpclient: upgrade to fe8c09e4db64 of httpplus
Augie Fackler <raf@durin42.com>
parents:
19038
diff
changeset
|
77 """Returns true if the response body is entirely read.""" |
16643
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
78 return self._finished |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
79 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
80 def read(self, amt): |
19182
fae47ecaa952
httpclient: upgrade to fe8c09e4db64 of httpplus
Augie Fackler <raf@durin42.com>
parents:
19038
diff
changeset
|
81 """Read amt bytes from the response body.""" |
16643
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
82 if self.available_data < amt and not self._finished: |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
83 raise ReadNotReady() |
19037
1fde25ad9396
http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents:
19036
diff
changeset
|
84 blocks = [] |
1fde25ad9396
http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents:
19036
diff
changeset
|
85 need = amt |
1fde25ad9396
http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents:
19036
diff
changeset
|
86 while self._done_chunks: |
19182
fae47ecaa952
httpclient: upgrade to fe8c09e4db64 of httpplus
Augie Fackler <raf@durin42.com>
parents:
19038
diff
changeset
|
87 b = self._popchunk() |
19037
1fde25ad9396
http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents:
19036
diff
changeset
|
88 if len(b) > need: |
1fde25ad9396
http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents:
19036
diff
changeset
|
89 nb = b[:need] |
19182
fae47ecaa952
httpclient: upgrade to fe8c09e4db64 of httpplus
Augie Fackler <raf@durin42.com>
parents:
19038
diff
changeset
|
90 self._pushchunk(b[need:]) |
19037
1fde25ad9396
http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents:
19036
diff
changeset
|
91 b = nb |
1fde25ad9396
http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents:
19036
diff
changeset
|
92 blocks.append(b) |
1fde25ad9396
http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents:
19036
diff
changeset
|
93 need -= len(b) |
1fde25ad9396
http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents:
19036
diff
changeset
|
94 if need == 0: |
1fde25ad9396
http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents:
19036
diff
changeset
|
95 break |
16643
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
96 result = ''.join(blocks) |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
97 assert len(result) == amt or (self._finished and len(result) < amt) |
19037
1fde25ad9396
http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents:
19036
diff
changeset
|
98 |
16643
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
99 return result |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
100 |
19038 | 101 def readto(self, delimstr, blocks = None): |
27601
1ad9da968a2e
httpclient: update to 938f2107d6e2 of httpplus
Augie Fackler <augie@google.com>
parents:
19182
diff
changeset
|
102 """return available data chunks up to the first one in which |
1ad9da968a2e
httpclient: update to 938f2107d6e2 of httpplus
Augie Fackler <augie@google.com>
parents:
19182
diff
changeset
|
103 delimstr occurs. No data will be returned after delimstr -- |
1ad9da968a2e
httpclient: update to 938f2107d6e2 of httpplus
Augie Fackler <augie@google.com>
parents:
19182
diff
changeset
|
104 the chunk in which it occurs will be split and the remainder |
1ad9da968a2e
httpclient: update to 938f2107d6e2 of httpplus
Augie Fackler <augie@google.com>
parents:
19182
diff
changeset
|
105 pushed back onto the available data queue. If blocks is |
1ad9da968a2e
httpclient: update to 938f2107d6e2 of httpplus
Augie Fackler <augie@google.com>
parents:
19182
diff
changeset
|
106 supplied chunks will be added to blocks, otherwise a new list |
1ad9da968a2e
httpclient: update to 938f2107d6e2 of httpplus
Augie Fackler <augie@google.com>
parents:
19182
diff
changeset
|
107 will be allocated. |
19038 | 108 """ |
109 if blocks is None: | |
110 blocks = [] | |
111 | |
112 while self._done_chunks: | |
19182
fae47ecaa952
httpclient: upgrade to fe8c09e4db64 of httpplus
Augie Fackler <raf@durin42.com>
parents:
19038
diff
changeset
|
113 b = self._popchunk() |
19038 | 114 i = b.find(delimstr) + len(delimstr) |
115 if i: | |
116 if i < len(b): | |
19182
fae47ecaa952
httpclient: upgrade to fe8c09e4db64 of httpplus
Augie Fackler <raf@durin42.com>
parents:
19038
diff
changeset
|
117 self._pushchunk(b[i:]) |
19038 | 118 blocks.append(b[:i]) |
119 break | |
120 else: | |
121 blocks.append(b) | |
122 | |
123 return blocks | |
124 | |
16643
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
125 def _load(self, data): # pragma: no cover |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
126 """Subclasses must implement this. |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
127 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
128 As data is available to be read out of this object, it should |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
129 be placed into the _done_chunks list. Subclasses should not |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
130 rely on data remaining in _done_chunks forever, as it may be |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
131 reaped if the client is parsing data as it comes in. |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
132 """ |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
133 raise NotImplementedError |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
134 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
135 def _close(self): |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
136 """Default implementation of close. |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
137 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
138 The default implementation assumes that the reader will mark |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
139 the response as finished on the _finished attribute once the |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
140 entire response body has been read. In the event that this is |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
141 not true, the subclass should override the implementation of |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
142 close (for example, close-is-end responses have to set |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
143 self._finished in the close handler.) |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
144 """ |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
145 if not self._finished: |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
146 raise HTTPRemoteClosedError( |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
147 'server appears to have closed the socket mid-response') |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
148 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
149 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
150 class AbstractSimpleReader(AbstractReader): |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
151 """Abstract base class for simple readers that require no response decoding. |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
152 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
153 Examples of such responses are Connection: Close (close-is-end) |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
154 and responses that specify a content length. |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
155 """ |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
156 def _load(self, data): |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
157 if data: |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
158 assert not self._finished, ( |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
159 'tried to add data (%r) to a closed reader!' % data) |
19182
fae47ecaa952
httpclient: upgrade to fe8c09e4db64 of httpplus
Augie Fackler <raf@durin42.com>
parents:
19038
diff
changeset
|
160 logger.debug('%s read an additional %d data', |
fae47ecaa952
httpclient: upgrade to fe8c09e4db64 of httpplus
Augie Fackler <raf@durin42.com>
parents:
19038
diff
changeset
|
161 self.name, len(data)) # pylint: disable=E1101 |
fae47ecaa952
httpclient: upgrade to fe8c09e4db64 of httpplus
Augie Fackler <raf@durin42.com>
parents:
19038
diff
changeset
|
162 self._addchunk(data) |
16643
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
163 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
164 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
165 class CloseIsEndReader(AbstractSimpleReader): |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
166 """Reader for responses that specify Connection: Close for length.""" |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
167 name = 'close-is-end' |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
168 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
169 def _close(self): |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
170 logger.info('Marking close-is-end reader as closed.') |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
171 self._finished = True |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
172 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
173 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
174 class ContentLengthReader(AbstractSimpleReader): |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
175 """Reader for responses that specify an exact content length.""" |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
176 name = 'content-length' |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
177 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
178 def __init__(self, amount): |
19182
fae47ecaa952
httpclient: upgrade to fe8c09e4db64 of httpplus
Augie Fackler <raf@durin42.com>
parents:
19038
diff
changeset
|
179 AbstractSimpleReader.__init__(self) |
16643
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
180 self._amount = amount |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
181 if amount == 0: |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
182 self._finished = True |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
183 self._amount_seen = 0 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
184 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
185 def _load(self, data): |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
186 AbstractSimpleReader._load(self, data) |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
187 self._amount_seen += len(data) |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
188 if self._amount_seen >= self._amount: |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
189 self._finished = True |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
190 logger.debug('content-length read complete') |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
191 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
192 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
193 class ChunkedReader(AbstractReader): |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
194 """Reader for chunked transfer encoding responses.""" |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
195 def __init__(self, eol): |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
196 AbstractReader.__init__(self) |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
197 self._eol = eol |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
198 self._leftover_skip_amt = 0 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
199 self._leftover_data = '' |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
200 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
201 def _load(self, data): |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
202 assert not self._finished, 'tried to add data to a closed reader!' |
17424
e7cfe3587ea4
fix trivial spelling errors
Mads Kiilerich <mads@kiilerich.com>
parents:
16643
diff
changeset
|
203 logger.debug('chunked read an additional %d data', len(data)) |
16643
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
204 position = 0 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
205 if self._leftover_data: |
19182
fae47ecaa952
httpclient: upgrade to fe8c09e4db64 of httpplus
Augie Fackler <raf@durin42.com>
parents:
19038
diff
changeset
|
206 logger.debug( |
fae47ecaa952
httpclient: upgrade to fe8c09e4db64 of httpplus
Augie Fackler <raf@durin42.com>
parents:
19038
diff
changeset
|
207 'chunked reader trying to finish block from leftover data') |
16643
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
208 # TODO: avoid this string concatenation if possible |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
209 data = self._leftover_data + data |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
210 position = self._leftover_skip_amt |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
211 self._leftover_data = '' |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
212 self._leftover_skip_amt = 0 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
213 datalen = len(data) |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
214 while position < datalen: |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
215 split = data.find(self._eol, position) |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
216 if split == -1: |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
217 self._leftover_data = data |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
218 self._leftover_skip_amt = position |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
219 return |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
220 amt = int(data[position:split], base=16) |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
221 block_start = split + len(self._eol) |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
222 # If the whole data chunk plus the eol trailer hasn't |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
223 # loaded, we'll wait for the next load. |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
224 if block_start + amt + len(self._eol) > len(data): |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
225 self._leftover_data = data |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
226 self._leftover_skip_amt = position |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
227 return |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
228 if amt == 0: |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
229 self._finished = True |
17424
e7cfe3587ea4
fix trivial spelling errors
Mads Kiilerich <mads@kiilerich.com>
parents:
16643
diff
changeset
|
230 logger.debug('closing chunked reader due to chunk of length 0') |
16643
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
231 return |
19182
fae47ecaa952
httpclient: upgrade to fe8c09e4db64 of httpplus
Augie Fackler <raf@durin42.com>
parents:
19038
diff
changeset
|
232 self._addchunk(data[block_start:block_start + amt]) |
16643
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
233 position = block_start + amt + len(self._eol) |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
234 # no-check-code |