mercurial/httpclient/_readers.py
author Augie Fackler <raf@durin42.com>
Fri, 04 May 2012 16:00:33 -0500
changeset 16643 24dbef11f477
child 17424 e7cfe3587ea4
permissions -rw-r--r--
httpclient: update to revision 892730fe7f46 of httpplus
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
16643
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
     1
# Copyright 2011, Google Inc.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
     2
# All rights reserved.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
     3
#
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
     4
# Redistribution and use in source and binary forms, with or without
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
     5
# modification, are permitted provided that the following conditions are
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
     6
# met:
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
     7
#
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
     8
#     * Redistributions of source code must retain the above copyright
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
     9
# notice, this list of conditions and the following disclaimer.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    10
#     * Redistributions in binary form must reproduce the above
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    11
# copyright notice, this list of conditions and the following disclaimer
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    12
# in the documentation and/or other materials provided with the
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    13
# distribution.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    14
#     * Neither the name of Google Inc. nor the names of its
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    15
# contributors may be used to endorse or promote products derived from
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    16
# this software without specific prior written permission.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    17
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    18
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    19
# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    20
# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    21
# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    22
# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    23
# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    24
# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    25
# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    26
# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    27
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    28
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    29
"""Reader objects to abstract out different body response types.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    30
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    31
This module is package-private. It is not expected that these will
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    32
have any clients outside of httpplus.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    33
"""
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    34
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    35
import httplib
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    36
import itertools
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    37
import logging
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    38
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    39
logger = logging.getLogger(__name__)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    40
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    41
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    42
class ReadNotReady(Exception):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    43
    """Raised when read() is attempted but not enough data is loaded."""
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    44
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    45
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    46
class HTTPRemoteClosedError(httplib.HTTPException):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    47
    """The server closed the remote socket in the middle of a response."""
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    48
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    49
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    50
class AbstractReader(object):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    51
    """Abstract base class for response readers.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    52
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    53
    Subclasses must implement _load, and should implement _close if
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    54
    it's not an error for the server to close their socket without
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    55
    some termination condition being detected during _load.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    56
    """
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    57
    def __init__(self):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    58
        self._finished = False
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    59
        self._done_chunks = []
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    60
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    61
    @property
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    62
    def available_data(self):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    63
        return sum(map(len, self._done_chunks))
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    64
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    65
    def done(self):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    66
        return self._finished
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    67
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    68
    def read(self, amt):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    69
        if self.available_data < amt and not self._finished:
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    70
            raise ReadNotReady()
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    71
        need = [amt]
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    72
        def pred(s):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    73
            needed = need[0] > 0
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    74
            need[0] -= len(s)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    75
            return needed
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    76
        blocks = list(itertools.takewhile(pred, self._done_chunks))
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    77
        self._done_chunks = self._done_chunks[len(blocks):]
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    78
        over_read = sum(map(len, blocks)) - amt
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    79
        if over_read > 0 and blocks:
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    80
            logger.debug('need to reinsert %d data into done chunks', over_read)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    81
            last = blocks[-1]
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    82
            blocks[-1], reinsert = last[:-over_read], last[-over_read:]
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    83
            self._done_chunks.insert(0, reinsert)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    84
        result = ''.join(blocks)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    85
        assert len(result) == amt or (self._finished and len(result) < amt)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    86
        return result
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    87
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    88
    def _load(self, data): # pragma: no cover
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    89
        """Subclasses must implement this.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    90
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    91
        As data is available to be read out of this object, it should
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    92
        be placed into the _done_chunks list. Subclasses should not
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    93
        rely on data remaining in _done_chunks forever, as it may be
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    94
        reaped if the client is parsing data as it comes in.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    95
        """
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    96
        raise NotImplementedError
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    97
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    98
    def _close(self):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    99
        """Default implementation of close.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   100
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   101
        The default implementation assumes that the reader will mark
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   102
        the response as finished on the _finished attribute once the
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   103
        entire response body has been read. In the event that this is
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   104
        not true, the subclass should override the implementation of
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   105
        close (for example, close-is-end responses have to set
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   106
        self._finished in the close handler.)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   107
        """
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   108
        if not self._finished:
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   109
            raise HTTPRemoteClosedError(
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   110
                'server appears to have closed the socket mid-response')
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   111
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   112
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   113
class AbstractSimpleReader(AbstractReader):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   114
    """Abstract base class for simple readers that require no response decoding.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   115
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   116
    Examples of such responses are Connection: Close (close-is-end)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   117
    and responses that specify a content length.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   118
    """
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   119
    def _load(self, data):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   120
        if data:
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   121
            assert not self._finished, (
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   122
                'tried to add data (%r) to a closed reader!' % data)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   123
        logger.debug('%s read an addtional %d data', self.name, len(data))
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   124
        self._done_chunks.append(data)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   125
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   126
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   127
class CloseIsEndReader(AbstractSimpleReader):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   128
    """Reader for responses that specify Connection: Close for length."""
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   129
    name = 'close-is-end'
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   130
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   131
    def _close(self):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   132
        logger.info('Marking close-is-end reader as closed.')
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   133
        self._finished = True
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   134
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   135
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   136
class ContentLengthReader(AbstractSimpleReader):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   137
    """Reader for responses that specify an exact content length."""
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   138
    name = 'content-length'
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   139
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   140
    def __init__(self, amount):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   141
        AbstractReader.__init__(self)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   142
        self._amount = amount
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   143
        if amount == 0:
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   144
            self._finished = True
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   145
        self._amount_seen = 0
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   146
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   147
    def _load(self, data):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   148
        AbstractSimpleReader._load(self, data)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   149
        self._amount_seen += len(data)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   150
        if self._amount_seen >= self._amount:
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   151
            self._finished = True
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   152
            logger.debug('content-length read complete')
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   153
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   154
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   155
class ChunkedReader(AbstractReader):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   156
    """Reader for chunked transfer encoding responses."""
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   157
    def __init__(self, eol):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   158
        AbstractReader.__init__(self)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   159
        self._eol = eol
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   160
        self._leftover_skip_amt = 0
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   161
        self._leftover_data = ''
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   162
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   163
    def _load(self, data):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   164
        assert not self._finished, 'tried to add data to a closed reader!'
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   165
        logger.debug('chunked read an addtional %d data', len(data))
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   166
        position = 0
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   167
        if self._leftover_data:
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   168
            logger.debug('chunked reader trying to finish block from leftover data')
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   169
            # TODO: avoid this string concatenation if possible
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   170
            data = self._leftover_data + data
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   171
            position = self._leftover_skip_amt
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   172
            self._leftover_data = ''
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   173
            self._leftover_skip_amt = 0
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   174
        datalen = len(data)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   175
        while position < datalen:
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   176
            split = data.find(self._eol, position)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   177
            if split == -1:
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   178
                self._leftover_data = data
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   179
                self._leftover_skip_amt = position
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   180
                return
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   181
            amt = int(data[position:split], base=16)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   182
            block_start = split + len(self._eol)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   183
            # If the whole data chunk plus the eol trailer hasn't
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   184
            # loaded, we'll wait for the next load.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   185
            if block_start + amt + len(self._eol) > len(data):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   186
                self._leftover_data = data
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   187
                self._leftover_skip_amt = position
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   188
                return
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   189
            if amt == 0:
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   190
                self._finished = True
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   191
                logger.debug('closing chunked redaer due to chunk of length 0')
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   192
                return
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   193
            self._done_chunks.append(data[block_start:block_start + amt])
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   194
            position = block_start + amt + len(self._eol)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   195
# no-check-code