Mercurial > hg
annotate mercurial/httpclient/_readers.py @ 29089:222b8170d69e stable
progress: stop excessive clearing (issue4801)
The progress bar was being cleared on every write(), regardless of
whether it was currently displayed. This could foul up the display of
any writes that didn't include a linebreak.
In particular, the win32 mode of the color extension was turning
single prompt string writes into two writes, and the resulting
clear/write/clear/write pattern was making the prompt invisible.
We fix this by insisting that we have shown a progress bar and haven't
just cleared it (setting lastprint to 0).
Conveniently, the test suite already had instances of duplicate
clears.. that are now cleared up.
author | Matt Mackall <mpm@selenic.com> |
---|---|
date | Fri, 06 May 2016 17:53:06 -0500 |
parents | 1ad9da968a2e |
children | 8a66eda46c98 |
rev | line source |
---|---|
16643
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
1 # Copyright 2011, Google Inc. |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
2 # All rights reserved. |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
3 # |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
4 # Redistribution and use in source and binary forms, with or without |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
5 # modification, are permitted provided that the following conditions are |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
6 # met: |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
7 # |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
8 # * Redistributions of source code must retain the above copyright |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
9 # notice, this list of conditions and the following disclaimer. |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
10 # * Redistributions in binary form must reproduce the above |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
11 # copyright notice, this list of conditions and the following disclaimer |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
12 # in the documentation and/or other materials provided with the |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
13 # distribution. |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
14 # * Neither the name of Google Inc. nor the names of its |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
15 # contributors may be used to endorse or promote products derived from |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
16 # this software without specific prior written permission. |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
17 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
18 # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
19 # "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
20 # LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
21 # A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
22 # OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
23 # SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
24 # LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
25 # DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
26 # THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
27 # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
28 # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
29 """Reader objects to abstract out different body response types. |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
30 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
31 This module is package-private. It is not expected that these will |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
32 have any clients outside of httpplus. |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
33 """ |
27601
1ad9da968a2e
httpclient: update to 938f2107d6e2 of httpplus
Augie Fackler <augie@google.com>
parents:
19182
diff
changeset
|
34 from __future__ import absolute_import |
16643
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
35 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
36 import httplib |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
37 import logging |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
38 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
39 logger = logging.getLogger(__name__) |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
40 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
41 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
42 class ReadNotReady(Exception): |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
43 """Raised when read() is attempted but not enough data is loaded.""" |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
44 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
45 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
46 class HTTPRemoteClosedError(httplib.HTTPException): |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
47 """The server closed the remote socket in the middle of a response.""" |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
48 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
49 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
50 class AbstractReader(object): |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
51 """Abstract base class for response readers. |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
52 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
53 Subclasses must implement _load, and should implement _close if |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
54 it's not an error for the server to close their socket without |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
55 some termination condition being detected during _load. |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
56 """ |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
57 def __init__(self): |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
58 self._finished = False |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
59 self._done_chunks = [] |
19036
19d1cc30e7a3
http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents:
17424
diff
changeset
|
60 self.available_data = 0 |
16643
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
61 |
19182
fae47ecaa952
httpclient: upgrade to fe8c09e4db64 of httpplus
Augie Fackler <raf@durin42.com>
parents:
19038
diff
changeset
|
62 def _addchunk(self, data): |
19036
19d1cc30e7a3
http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents:
17424
diff
changeset
|
63 self._done_chunks.append(data) |
19d1cc30e7a3
http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents:
17424
diff
changeset
|
64 self.available_data += len(data) |
19d1cc30e7a3
http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents:
17424
diff
changeset
|
65 |
19182
fae47ecaa952
httpclient: upgrade to fe8c09e4db64 of httpplus
Augie Fackler <raf@durin42.com>
parents:
19038
diff
changeset
|
66 def _pushchunk(self, data): |
19036
19d1cc30e7a3
http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents:
17424
diff
changeset
|
67 self._done_chunks.insert(0, data) |
19d1cc30e7a3
http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents:
17424
diff
changeset
|
68 self.available_data += len(data) |
19d1cc30e7a3
http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents:
17424
diff
changeset
|
69 |
19182
fae47ecaa952
httpclient: upgrade to fe8c09e4db64 of httpplus
Augie Fackler <raf@durin42.com>
parents:
19038
diff
changeset
|
70 def _popchunk(self): |
19036
19d1cc30e7a3
http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents:
17424
diff
changeset
|
71 b = self._done_chunks.pop(0) |
19d1cc30e7a3
http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents:
17424
diff
changeset
|
72 self.available_data -= len(b) |
19d1cc30e7a3
http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents:
17424
diff
changeset
|
73 |
19d1cc30e7a3
http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents:
17424
diff
changeset
|
74 return b |
16643
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
75 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
76 def done(self): |
19182
fae47ecaa952
httpclient: upgrade to fe8c09e4db64 of httpplus
Augie Fackler <raf@durin42.com>
parents:
19038
diff
changeset
|
77 """Returns true if the response body is entirely read.""" |
16643
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
78 return self._finished |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
79 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
80 def read(self, amt): |
19182
fae47ecaa952
httpclient: upgrade to fe8c09e4db64 of httpplus
Augie Fackler <raf@durin42.com>
parents:
19038
diff
changeset
|
81 """Read amt bytes from the response body.""" |
16643
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
82 if self.available_data < amt and not self._finished: |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
83 raise ReadNotReady() |
19037
1fde25ad9396
http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents:
19036
diff
changeset
|
84 blocks = [] |
1fde25ad9396
http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents:
19036
diff
changeset
|
85 need = amt |
1fde25ad9396
http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents:
19036
diff
changeset
|
86 while self._done_chunks: |
19182
fae47ecaa952
httpclient: upgrade to fe8c09e4db64 of httpplus
Augie Fackler <raf@durin42.com>
parents:
19038
diff
changeset
|
87 b = self._popchunk() |
19037
1fde25ad9396
http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents:
19036
diff
changeset
|
88 if len(b) > need: |
1fde25ad9396
http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents:
19036
diff
changeset
|
89 nb = b[:need] |
19182
fae47ecaa952
httpclient: upgrade to fe8c09e4db64 of httpplus
Augie Fackler <raf@durin42.com>
parents:
19038
diff
changeset
|
90 self._pushchunk(b[need:]) |
19037
1fde25ad9396
http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents:
19036
diff
changeset
|
91 b = nb |
1fde25ad9396
http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents:
19036
diff
changeset
|
92 blocks.append(b) |
1fde25ad9396
http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents:
19036
diff
changeset
|
93 need -= len(b) |
1fde25ad9396
http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents:
19036
diff
changeset
|
94 if need == 0: |
1fde25ad9396
http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents:
19036
diff
changeset
|
95 break |
16643
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
96 result = ''.join(blocks) |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
97 assert len(result) == amt or (self._finished and len(result) < amt) |
19037
1fde25ad9396
http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents:
19036
diff
changeset
|
98 |
16643
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
99 return result |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
100 |
19038 | 101 def readto(self, delimstr, blocks = None): |
27601
1ad9da968a2e
httpclient: update to 938f2107d6e2 of httpplus
Augie Fackler <augie@google.com>
parents:
19182
diff
changeset
|
102 """return available data chunks up to the first one in which |
1ad9da968a2e
httpclient: update to 938f2107d6e2 of httpplus
Augie Fackler <augie@google.com>
parents:
19182
diff
changeset
|
103 delimstr occurs. No data will be returned after delimstr -- |
1ad9da968a2e
httpclient: update to 938f2107d6e2 of httpplus
Augie Fackler <augie@google.com>
parents:
19182
diff
changeset
|
104 the chunk in which it occurs will be split and the remainder |
1ad9da968a2e
httpclient: update to 938f2107d6e2 of httpplus
Augie Fackler <augie@google.com>
parents:
19182
diff
changeset
|
105 pushed back onto the available data queue. If blocks is |
1ad9da968a2e
httpclient: update to 938f2107d6e2 of httpplus
Augie Fackler <augie@google.com>
parents:
19182
diff
changeset
|
106 supplied chunks will be added to blocks, otherwise a new list |
1ad9da968a2e
httpclient: update to 938f2107d6e2 of httpplus
Augie Fackler <augie@google.com>
parents:
19182
diff
changeset
|
107 will be allocated. |
19038 | 108 """ |
109 if blocks is None: | |
110 blocks = [] | |
111 | |
112 while self._done_chunks: | |
19182
fae47ecaa952
httpclient: upgrade to fe8c09e4db64 of httpplus
Augie Fackler <raf@durin42.com>
parents:
19038
diff
changeset
|
113 b = self._popchunk() |
19038 | 114 i = b.find(delimstr) + len(delimstr) |
115 if i: | |
116 if i < len(b): | |
19182
fae47ecaa952
httpclient: upgrade to fe8c09e4db64 of httpplus
Augie Fackler <raf@durin42.com>
parents:
19038
diff
changeset
|
117 self._pushchunk(b[i:]) |
19038 | 118 blocks.append(b[:i]) |
119 break | |
120 else: | |
121 blocks.append(b) | |
122 | |
123 return blocks | |
124 | |
16643
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
125 def _load(self, data): # pragma: no cover |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
126 """Subclasses must implement this. |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
127 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
128 As data is available to be read out of this object, it should |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
129 be placed into the _done_chunks list. Subclasses should not |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
130 rely on data remaining in _done_chunks forever, as it may be |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
131 reaped if the client is parsing data as it comes in. |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
132 """ |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
133 raise NotImplementedError |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
134 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
135 def _close(self): |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
136 """Default implementation of close. |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
137 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
138 The default implementation assumes that the reader will mark |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
139 the response as finished on the _finished attribute once the |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
140 entire response body has been read. In the event that this is |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
141 not true, the subclass should override the implementation of |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
142 close (for example, close-is-end responses have to set |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
143 self._finished in the close handler.) |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
144 """ |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
145 if not self._finished: |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
146 raise HTTPRemoteClosedError( |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
147 'server appears to have closed the socket mid-response') |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
148 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
149 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
150 class AbstractSimpleReader(AbstractReader): |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
151 """Abstract base class for simple readers that require no response decoding. |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
152 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
153 Examples of such responses are Connection: Close (close-is-end) |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
154 and responses that specify a content length. |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
155 """ |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
156 def _load(self, data): |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
157 if data: |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
158 assert not self._finished, ( |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
159 'tried to add data (%r) to a closed reader!' % data) |
19182
fae47ecaa952
httpclient: upgrade to fe8c09e4db64 of httpplus
Augie Fackler <raf@durin42.com>
parents:
19038
diff
changeset
|
160 logger.debug('%s read an additional %d data', |
fae47ecaa952
httpclient: upgrade to fe8c09e4db64 of httpplus
Augie Fackler <raf@durin42.com>
parents:
19038
diff
changeset
|
161 self.name, len(data)) # pylint: disable=E1101 |
fae47ecaa952
httpclient: upgrade to fe8c09e4db64 of httpplus
Augie Fackler <raf@durin42.com>
parents:
19038
diff
changeset
|
162 self._addchunk(data) |
16643
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
163 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
164 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
165 class CloseIsEndReader(AbstractSimpleReader): |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
166 """Reader for responses that specify Connection: Close for length.""" |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
167 name = 'close-is-end' |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
168 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
169 def _close(self): |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
170 logger.info('Marking close-is-end reader as closed.') |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
171 self._finished = True |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
172 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
173 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
174 class ContentLengthReader(AbstractSimpleReader): |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
175 """Reader for responses that specify an exact content length.""" |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
176 name = 'content-length' |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
177 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
178 def __init__(self, amount): |
19182
fae47ecaa952
httpclient: upgrade to fe8c09e4db64 of httpplus
Augie Fackler <raf@durin42.com>
parents:
19038
diff
changeset
|
179 AbstractSimpleReader.__init__(self) |
16643
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
180 self._amount = amount |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
181 if amount == 0: |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
182 self._finished = True |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
183 self._amount_seen = 0 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
184 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
185 def _load(self, data): |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
186 AbstractSimpleReader._load(self, data) |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
187 self._amount_seen += len(data) |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
188 if self._amount_seen >= self._amount: |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
189 self._finished = True |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
190 logger.debug('content-length read complete') |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
191 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
192 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
193 class ChunkedReader(AbstractReader): |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
194 """Reader for chunked transfer encoding responses.""" |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
195 def __init__(self, eol): |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
196 AbstractReader.__init__(self) |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
197 self._eol = eol |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
198 self._leftover_skip_amt = 0 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
199 self._leftover_data = '' |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
200 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
201 def _load(self, data): |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
202 assert not self._finished, 'tried to add data to a closed reader!' |
17424
e7cfe3587ea4
fix trivial spelling errors
Mads Kiilerich <mads@kiilerich.com>
parents:
16643
diff
changeset
|
203 logger.debug('chunked read an additional %d data', len(data)) |
16643
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
204 position = 0 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
205 if self._leftover_data: |
19182
fae47ecaa952
httpclient: upgrade to fe8c09e4db64 of httpplus
Augie Fackler <raf@durin42.com>
parents:
19038
diff
changeset
|
206 logger.debug( |
fae47ecaa952
httpclient: upgrade to fe8c09e4db64 of httpplus
Augie Fackler <raf@durin42.com>
parents:
19038
diff
changeset
|
207 'chunked reader trying to finish block from leftover data') |
16643
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
208 # TODO: avoid this string concatenation if possible |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
209 data = self._leftover_data + data |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
210 position = self._leftover_skip_amt |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
211 self._leftover_data = '' |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
212 self._leftover_skip_amt = 0 |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
213 datalen = len(data) |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
214 while position < datalen: |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
215 split = data.find(self._eol, position) |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
216 if split == -1: |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
217 self._leftover_data = data |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
218 self._leftover_skip_amt = position |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
219 return |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
220 amt = int(data[position:split], base=16) |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
221 block_start = split + len(self._eol) |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
222 # If the whole data chunk plus the eol trailer hasn't |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
223 # loaded, we'll wait for the next load. |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
224 if block_start + amt + len(self._eol) > len(data): |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
225 self._leftover_data = data |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
226 self._leftover_skip_amt = position |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
227 return |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
228 if amt == 0: |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
229 self._finished = True |
17424
e7cfe3587ea4
fix trivial spelling errors
Mads Kiilerich <mads@kiilerich.com>
parents:
16643
diff
changeset
|
230 logger.debug('closing chunked reader due to chunk of length 0') |
16643
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
231 return |
19182
fae47ecaa952
httpclient: upgrade to fe8c09e4db64 of httpplus
Augie Fackler <raf@durin42.com>
parents:
19038
diff
changeset
|
232 self._addchunk(data[block_start:block_start + amt]) |
16643
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
233 position = block_start + amt + len(self._eol) |
24dbef11f477
httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
234 # no-check-code |